Auto-tools/Projects/Mochi Bisector

From MozillaWiki
Jump to: navigation, search

The Problem

Several times when developers attempt to check in a patch, they realize that their patch has an odd knock-on effect such that a test or a set of tests begin to consistently fail. Sometimes this set of tests appears to have nothing to do with their patch. In these cases, the developer is faced with attempting to bisect the mochitest test suite so that they can find the smallest set of tests that demonstrate the consistent failure and can then have a reasonable use case for debugging the problem.

You can see jlebar having to do this in bug 769254. Peterv has done this a number of times as well tracking down memory leaks that occurred after he landed a patch on try.

The Proposal

Create a second test runner for mochitest that re-uses all the logic from the normal runtests.py but that allows a developer to specify a condition (the condition could be that X tests fail with Y assertions). Then the new test runner would bisect over the set of mochitests (as per the given parameters - i.e. you would give mochitest the same chunk name etc that you see on try where you observed the failure) until it found the smallest number of tests to run that would reproduce the issue.

Note that this will take a considerable amount of JS work because the result collection and the stepping through the results are all done in JS. So, we will need to think about how to architect this since the JS side of mochitest isn't set up for easy re-implementation.

Deliverable

This can be a one off test runner. I'd like to re-use as much of the existing mochitest runner as possible so that this runner gets all the benefits of sharing that source, and so that the tests are being run the same way when they are being bisected. This will likely take some architecture and refactoring on the JavaScript side of the harness to achieve this.