L10n:Testing

This page is an outline of which tests we currently run, and should run, and takes a stab at putting some priorities to those.

tests run

The only real test currently run is

• compare locales

There are two different implementations of that test, which differ in detail and output, one in perl, one in python, the perl one being run as part of tinderbox, the python one run as part of Axel's status-1.8, on a cron job.

That 1.8 Status page also collects information on search engines, bookmarks, and RSS feed readers. Those are not strictly tests, but the page provides a good helper to assess 3rd party involvement in localized builds.

tests to be run

There are currently no runtime tests run at all on any localized builds. That'd be something worth fixing, as that would verify that the compare-locales test didn't overlook any errors (there are known loop holes). Some automatic UI exposure is needed for that, too. Start points for that are available in the l10n testing extensions.

Further areas of interest are:

Preferences 

firefox-l10n.js can contain any preference, and expose localized preferences builds usually don't expose. That file is currently not tested at all, though Axel usually does a review on that file at one point.

Search 

There are multiple artifacts to test about search. More below.

Localization coverage 

The python impl of compare locales has information on this, but it's brittle and needs to be fixed before exposing it as a measure. In particular, any localization statistic needs to exclude accesskeys and commandkeys, to be at least somewhat relevant.

Search tests

There are two basic angles to take at testing search, one is, testing that our code does what we expect it to, the second is, testing that the search provider does what we expect it to.

The first test requires to have a specification of what Search is expected to do, and should include things like

set of search plugins 

which search engines does a particular localization ship with

ordered search plugins 

which plugins should be sorted, and where

search urls 

Does the plugin use the right URL. Is there a conflict between spec and impl for a particular build. Note: This does depend on official vs. non-official, order, prefs, etc. That is, a single passed test may not necessarily transfer from a nightly to a release build or vice versa.

search suggest behaviour 

same as the above, just different URL. Note the 500ms timeout for suggest in addition.

The other angle would be to make sure that the resulting URL is good. This is somewhat of a nagios test that search engine providers keep their entry points up, suggest doesn't show timeouts etc. Note: Some search providers use redirects on their search, i.e. Amazon. Thus, the resulting URLs we see after a search is not the URL we're supposed to send to the search service provider. That's having a major impact on how much we can actually test in terms of search using the exact native code path.