User:Rcampbell/3PM Testing Meeting notes
TODO: Ask if Alice is available to help out with performance testing on tinderboxes.
TODO: Connecting Tests to Tinderboxes
Talk to Rob Helmer, Paul Reid re: build and Tinderboxen.
Graydon has new page set for incorporation into performance tests.
Contents
Planning Meeting
Unit Test Framework
Proof of Concept - 1 wk
Roll out to Trunk +1 week
Integrate with test Tinderboxes
Database + Reporting Mechanism
Integrate with Bob Clary's JavaScript testing systems
make file targets for unit tests. Developers should run locally to be ensure unit tests pass before committing code.
Test Harnesses for Acceptance Test Frameworks
DOM
Layout
Graphical Comparisons
Eggplant incorporation
JSUnit for testing content scripting
Jssh for testing into the chrome
XPCShell might have overlap with UnitTest
Test cases go to Litmus or Tinderbox?
"Underneath the GUI"
Performance Testing / Memory Leaks
Alice to help setup/improve performance tests on Tinderbox.
The Path to Automation
Top level datapoints
Eggplant
JavaScript Test Integration
Security Tests - non public, exposure issue. Recovery issue.
Unit Tests
Acceptance Tests - Top Site testing
Performance / Memory Leak tests
L10N
Update Testing
Reporting
Eggplant is currently running on an old machine, needs updating. Has Perl::Litmus to dump data into Litmus.
JS Tests are already executable via command line tool. Need to massage data to unified format for storage/reporting. Coop thinks Litmus would be the place to store the results and needs to talk to Bob to make the storage decisions. Perl::Litmus can be used for this as well.
Issue with deciding how to generate or create tests in Litmus.
Security Tests could be automated with EggPlant. Litmus could have a "private" tag added to hide tests.
Unit Tests could be run through Tinderbox and pass some more detailed results to Litmus. Requires further investigation.
Acceptance Tests run through Tinderbox with results fed into Litmus.
Performance / Memory Leaks run through Tinderbox. Possibly need more well-formed results format from different tests for feeding into multiple systems. Needs further discussion with Timr, Alice and Schrep.
L10N talk with Rob Helmer about Tinderbox integration. Eggplant for verification. Dave, Tracy and Axel need to coordinate for Firefox 2 release.
Update Testing. Scary area. Require further analysis to break down failure points and study the different permutations.
