Running the automated tests for site interventions

The easiest way to currently run the tests is to use a local build of Firefox on a desktop computer (artifact builds are recommended to save a lot of otherwise-wasted time).

Almost all of the automated tests run on their own in bulk. A desktop build of Firefox will test the desktop site interventions, while an Android build will start an emulator and test the Android interventions.

It is recommended to re-direct the output to a logfile.

# run desktop/laptop tests
./mach bootstrap # choose desktop artifact
./mach build
./mach test-interventions --headless | tee desktop.log

# run Android-specific tests (on an emulator)
./mach bootstrap # choose android artifact
./mach build
./mach test-interventions --headless | tee android.log

However, there are some tests which require a specific platform, VPN, or other conditions like logins, or they will be skipped (which should be mentioned in the resulting log):

Tests requiring Linux

# samsung_com.py
./mach test-interventions --headless --bug=1799968

Tests requiring Windows or MacOS

# saxoinvestor.fr
./mach test-interventions --headless --bug=1719859

Tests requiring Linux or Windows

# livescience.com
./mach test-interventions --headless --bug=1650317

Tests requiring Linux or MacOS

# bankofamerica.py.com
./mach test-interventions --headless --bug=1472075

Tests requiring a VPN connection to the USA

# directv.com.co
./mach test-interventions --headless --bug=1610344

# stream.directv.com
./mach test-interventions --headless --bug=1577519

Tests requiring VPN connections elsewhere

(to be determined/confirmed)

Tests requiring a login

# missingmail.usps.com
# note this test can take several minutes to run, as the site is very slow
./mach test-interventions --headless --bug=1644830 --config=logins.json

# Steam Community
# note that Steam requires two-factor authentication, so you will be asked
# to provide a code twice by the test as it runs.
./mach test-interventions --headless --bug=1570108 --do2fa --config=logins.json

Example logins.json file

The logins.json looks like this, only with actual passwords:

{
  "1570108": {"username":"steam_username", "password":"steam_password"},
  "1644830": {"username":"usps_username", "password":"usps_password"}
}

Note that the file must be valid JSON, so of course care should be taken with characters like quotes and brackets and commas. For example, if a password itself contains a double-quote, it must be escaped with a backslash:

"password": "password with a \" double quote"

Interpreting the results

The logfile will capture two summaries of which tests FAILED and were SKIPPED, once with the intervention active (to see if it is still working) and once with it disabled (to see if it is still even necessary, or if the site issues has been fixed).

For instance, here is one such summary:

=========================== short test summary info ============================
FAILED testing/webcompat/interventions/tests/test_1799994_vivobarefoot_com.py::test_enabled[with_interventions]
FAILED testing/webcompat/interventions/tests/test_1830752_afisha_ru.py::test_enabled[with_interventions]
FAILED testing/webcompat/interventions/tests/test_1836182_www_flatsatshadowglen_com.py::test_enabled[with_interventions]
SKIPPED [1] testing/webcompat/interventions/tests/test_1448747_discountcoffee_co_uk.py:12: Bug #1448747 skipped on platform (mac)
SKIPPED [1] testing/webcompat/interventions/tests/test_1448747_dylantalkstone_com.py:12: Bug #1448747 skipped on platform (mac)

The SKIPPED tests should be run on the appropriate platform, with logins, or checked manually if necessary.

FAILED tests require need to be checked to confirm whether the intervention is still working or necessary. Sometimes a test will fail because a site has changed, but it may fail simply due to a networking or other hiccup. As such it may be worth running each failing bug number's tests a second or third time, using the --bug= parameter as shown above (including the config file if the test requires a login).