Better DXR Testing

Here are some thoughts about improving the automated testing story for DXR.

Problems to solve

• It's hard to make a new test. You have to duplicate a lot of boilerplate, change dir names in said boilerplate, and the number of files goes way up. (Fixed in https://github.com/mozilla/dxr/commit/4cf2929.)
• Because new test fixtures are hard to make, we just jam everything into json-test, which couples all the tests together. We risk invalidating old tests every time we change something, and it's hard to keep track of what's significant. (Fixed in https://github.com/mozilla/dxr/commit/4cf2929.)
• We have to write our own lame testrunner: that "global failed" thing in search-test.py that keeps track of whether any tests have failed. And there's a bare except in there. We would rather not maintain any pieces of our own testrunner that we don't have to. We *certainly* don't want to duplicate them. (Fixed in https://github.com/mozilla/dxr/commit/20fddc60536b0bc10b2cd5daaea3aa0e685cfd1e.)
• There's another layer of custom test framework: run-tests.py. Right now, it's a spaghetti of shell scripts and sleeps. Surely at least we can share some common setup and teardown. (Fixed in https://github.com/mozilla/dxr/commit/20fddc60536b0bc10b2cd5daaea3aa0e685cfd1e.)
• The tested server runs in a different process than the testrunner, making dev clumsier. (We can't just set a breakpoint in a test and step from there into the search code, for instance.) (Fixed in https://github.com/mozilla/dxr/commit/20fddc60536b0bc10b2cd5daaea3aa0e685cfd1e.)

Things to keep

• It's easy to run a test case as a regular DXR deployment so you can explore.
• The top-level `make test` entrypoint is a nice, generic handle for Jenkins, no matter what language(s) or tool(s) the tests end up written with.

The Brave, New World

Have one class that supports running tests against a concrete DXR instance—files and all. That'll be good for complex tests with many files where the FS is the least confusing place to express them. (This now exists in DxrInstanceTestCase.)

Another class can support simple tests that need a single file or so. (Fortunately, everything ends up as a single file in C anyway.) You'll provide code as a string that gets laid down in a dynamically generated instance and compiled. There'll be a flag or well-marked place to put a breakpoint to leave the instance around in case you want to interact with it manually. We'll add a helper of some kind to make it easy to assert not only that certain files turn up in a search but that highlight ranges are as expected. This should make tests trivial to write and solve the coupling problem. The downside is that tests will run slower, since there will be more build runs. We can probably at least apply nose's multiprocessor stuff to it if we keep the temp files separate. (This now exists in SingleFileTestCase.)