Sheriffing/How To/Updating test expectations

Sometimes test expectations (if they pass, fail, time out etc.) need to be adjusted:

• web platform tests (wpt) are regularly imported from an external repository and the features tested might not have been implemented in Firefox yet. The import script sets the expectations based on Try pushes but the expected result can have changed because the feature got implemented between the Try push and the wpt test synchronization.
• Tests developed by Mozilla are expected to pass but can be disabled for some platform configurations if they fail, e.g. new versions of operating systems often fail some tests and get added with tier 2 priority. The tests are initially skipped and bugs filed to fix the issue and reenable the test.

mochitests

Mochitests test various parts of the browser, e.g. the UI functionality (browser-chrome, devtools), web browser interfaces (mochitest-plain). The most common annotation (below the test name in the .ini manifest file in the folder with the test file) is skip-if = followed by conditions under which the test shall not be executed, e.g.

skip-if = os == "win" && bits == 64

This skips the test for 64-bit builds on Windows.

If the conditions are spread across multiple lines, fulfilling the conditions in one line causes the test to get skipped - the lines are connected by an "OR" condition. Example:

skip-if =
  os == "win" && bits == 64
  os == "linux"

This causes the test to get skipped if the platform is either Windows with a 64-bit build or any Linux platform and Firefox build.

• The variables available and their values are listed in each mochitest log near the end of the setup. Search for "These variables are available in the mozinfo environment and can be used to skip tests conditionally" in the log.

reftests

Reftests display content ("paint it") with one technology and once again with others and compare if the results match.

A failure message could be:

REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/transform/partial-prerender-in-svg-1.html == layout/reftests/transform/partial-prerender-in-svg-1-ref.html | image comparison, max difference: 255, number of differing pixels: 400

The first number is the maximum difference for a pixel byte while the second one is the count of different pixels. These get used when a test gets set as intermittent:

fuzzy-if(Android,0-255,0-400)

The first parameter is the condition. The second one allows between 0 and 255 as difference for a byte, and the third parameter allows between 0 and 400 different pixels.

• The functions available for annotations can be found in the documentation in the repository.
• The variables available and their values are listed in each reftest log near the end of the setup. Search for "Dumping representation of sandbox which can be used for expectation annotations" in the log.

web-platform-tests

Expectation update from the command line

1. Open a console window.
2. Switch to the parent folder of the mozilla-unified directory with the cd command.
3. Create a folder named logs:
   mkdir logs
4. Install the following dependency:
   sudo apt-get install libssl-dev
5. Clone the script to download several logs for wpt tasks:
   git clone https://github.com/jgraham/fetchlogs.git
   You can also download it as .zip file but won't be able to update it from the command line in that case.
6. Switch to the folder with newly downloaded source code:
   cd fetchlogs
7. In Treeherder, identify the tasks whose unexpected test results should be used to update test expectations.
8. In the console, download the logs which a command similar to the following one:
   cargo run mozilla-central e40a136dc8760f3fdd8a29fad47581d16d646c80 --out-dir ../logs/ --filter-jobs test-linux1804-32-shippable-qr/opt-web-platform-tests-e10s-10
   Replace mozilla-central with the name of the tree and e40a136dc8760f3fdd8a29fad47581d16d646c80 with the revision of the push from which the logs shall be downloaded.
   --filter-jobs expects the name of the task as parameter (found at the bottom left in Treeherder) - or a regular expression to match multiple task names. If omitted, the log for every wpt task for that push will be downloaded. Don't copy the job symbol mentioned at the end.
   This depends on the Rust programming language which should be installed since ./mach bootstrap had been run in mozilla-unified
9. Switch to mozilla-unified
cd ../mozilla-unified/
10. Update the files with the test expectations:
    ./mach wpt-update ../logs/* --update-intermittent. This can take some time.
11. Check which files have been updated:
    hg status
    There are very likely more files modified (M) or added (A) than desired.
12. Revert the unintentional changes:
    hg revert -r . -X "path:testing/web-platform/meta/<continuation_of_path>"
    Multiple paths can be kept in the current state by using the -X ("exclude") pattern multiple times.
    <continuation_of_path> needs to be replaced with the path and file name of the failure message + ".ini". E.g. for TEST-UNEXPECTED-PASS | example/test.html | should implement interface use path:testing/web-platform/meta/example/test.html.ini.
13. Check hg status again.
14. Save the changes with hg commit -e

Manual update of expectations for web-platform-tests

See the official documentation.