QA/Mozmill Test Automation/Fixing Broken Tests
Contents
Overview
| Lead: | Henrik Skupin |
| Co-workers: | Anthony Hughes (3.x), Geo Mealer (4.0) |
| Dates: | No end date set |
| Status: | Identifying and fixing tests |
| Repository Location: | http://hg.mozilla.org/qa/mozmill-tests/ |
| Tracking Bug / Bug List: | Open Broken Tests, All Broken Tests Branch-only Failures |
Excerpt
To ensure the quality of Mozmill tests and to get the tests into a state that results of test-runs have the same importance like other automated tests, broken tests have to be identified and fixed. This work will happen across all branches of Firefox we support with Mozmill tests.
Project Details
Working on broken tests the whole process can be separated into 4 steps. In the beginning you will have to identify broken tests. Afterward you will have to file a new bug on Bugzilla. That step makes sure that the brokenness is covered. If you decide to help in fixing broken tests, assign a bug to yourself and upload a fix. After it has been reviewed and checked into our test repository a verification should make sure the failure has been disappeared on all platforms.
Identification of Broken Tests
The identification of broken tests happens in different ways:
1. You can run all existing tests on your own machine to identify failing tests. Therefore you should run the normal tests and the restart tests. See our documentation on developer.mozilla.org how to do that.
2. Mozilla QA runs daily tests in the Mozilla Lab on all platforms. Test results for all those tests are send to our Brasstacks server. You can search through the results on the reports page or the top failures page.
3. Until a web dashboard will be available which will display all the correlations we track the results in the Mozmill Failures spreadsheet See links in #2 above.
If a test fails constantly make sure to run the same test again with the '--show-errors' option of the Mozmill CLI. The reported error message should be used in the bug report for a clear failure description.
Filing a Bug
If a failure has been discovered, a search on Bugzilla has to be performed before filing a new bug. It's necessary to make sure that no-one else already created a bug for that failing test. There is a query which lists all broken tests.
If none of the listed bugs covers the test or test module you are experiencing the failure in, please file a new bug. Make sure that the summary contains a copy of the error message and the test module it happens in.
After filing a bug, you should add it to the Pivotal tracker (email Anthony if you need access).
- Click MORE | Bugzilla
- Find your bug and drag it to ICEBOX
- Change the story to a "Bug" and add the "failure" tag
- Drag it to CURRENT to begin working on it
Disabling Test in Litmus
Unless you're going to be able to fix the error immediately (same day as discovered), please disable the test in Litmus. If you do not have the necessary access in Litmus, please contact one of the leads above to disable to test.
- Remove the Mozmill group from the Litmus test corresponding to the broken test.
- Note the Litmus test in the bug so that it can be re-enabled later.
Fixing a Broken Test
Each of the filed bugs is used to track the work of fixing that particular issue. If you decide to work on a bug please assign this bug to yourself or make a comment if you do not have these permissions. Someone else from the Mozmill team can assign the bug later.
Having a better understanding why the test fails is really helpful before starting to fix the issue. Therefore you should try to find out which tests have to be run before the failing test, just to make it fail. Given the result a simplified test could be created and attached to the bug. If that's not possible you can try to insert sleep statements, show up results in an alert box, or even use Venkman to debug the test.
Fixing a test means you should obey the test writing guidelines. Once you have updated the test, you should start at least one (better a couple in a row) complete test-run to ensure the test is really fixed.
Before the fix can be checked-in a review is necessary. Therefore create a patch and ask for review. Details about that process can be found on MDC.
Verifying a fix
Once the patch has been checked-in, the bug will be marked as fixed. We will wait for the next test-run on our machine in the QA lab. If the problem has really gone we are fine and you did a great job. Otherwise we have to back-out the patch and go back to the investigation phase. But finally we should be able to fix any of the problems.
Re-enabling Test in Litmus
If the bug notes that a Litmus test has been disabled due to the failure, please re-enable the Litmus test after the fix is successfully verified. This is done by re-adding the Mozmill group to the Litmus test.
If you do not have the necessary access to Litmus, please contact one of the leads above to re-enable the test.
