Jetpack/Testing
This page explains how to run the SDK's unit tests, and how to interpret the results. It's aimed at people who don't have any familiarity with the SDK, but nevertheless have to run the tests to diagnose any test failures, for example because of regressions introduced by the underlying platform.
For any help with these tests come to the #jetpack channel and ask the team, Gozala is the module owner.
Contents
Running tests from compiled Firefox
You have two options to run tests from a built version of Firefox. Both do basically the same thing and are quite limited right now, if you want more functionality (like being able to select specific tests) you'll need to checkout the SDK repository separately and follow the later instructions.
Running with mach
From the source directory or object directory run:
mach mochitest addon-sdk/source
You can also run specific tests like so:
mach mochitest addon-sdk/source/test/test-sandbox.js
or possibly even:
mach mochitest test-sandbox.js
if the name doesn't collide with any other tests.
Running with make
Don't.
Disabling tests
Tests can be disabled using the regular mochitest ini manifest. See jetpack-package.ini and jetpack-addon.ini.
Running tests from the SDK repository
First, get the SDK from git:
git clone git://github.com/mozilla/addon-sdk.git cd addon-sdk
Or go into the SDK directory in mozilla-central:
cd addon-sdk
Activate the SDK (this isn't necessary but puts the cfx tool into your path, you can instead run it from addon-sdk/bin/cfx):
source bin/activate
To run all tests:
cfx testall
To run only the API tests (typically the ones that break when Firefox changes), invoke testpkgs instead of testall:
cfx testpkgs
To run only certain test suites, specify the --filter flag with a regular expression that matches the filenames of the test suites you want to run:
cfx testpkgs --filter panel
Specifying the Firefox Build
To run tests on a specific build of Firefox, specify the build via the --binary flag:
cfx testall --binary /Applications/Nightly.app/
cfx testall --binary ~/Mozilla/source/obj-dir/dist/bin/firefox
cfx testall --binary /c/Program\ Files\Nightly\firefox.exe
Attaching a Debugger
To attach gdb before running tests, specify the --no-run flag:
cfx testpkgs --no-run
--no-run doesn't actually run the command, it just prints out the shell command that would be invoked in order to run it. So the above command would produce output like:
To launch the application, enter the following command: /Applications/Firefox.app/Contents/MacOS/firefox-bin -profile /var/folder /me/me2B1lFDE0WZCgd33s2OTE+++TU/-Tmp-/tmp72nW_f.mozrunner -foreground -no-remote
You can then run the debugger using that command:
gdb --args /Applications/Firefox.app/Contents/MacOS/firefox-bin -profile /var/folder /me/me2B1lFDE0WZCgd33s2OTE+++TU/-Tmp-/tmp72nW_f.mozrunner -foreground -no-remote
Gory Details of Testing With cfx
There's cfx documentation here, and command line help is available by typing:
cfx --help
There are five different test commands:
test - run tests testcfx - test the cfx tool testex - test all example code testpkgs - test all installed packages testall - test whole environment
cfx test
cfx test runs the tests implemented for a single SDK "package". A package is a collection of JavaScript modules, and is defined as such by the presence of a file called "package.json" in its root directory. Among other things, "package.json" includes an string called "tests" which describes where the package's test suites live. If this is not defined, the default is the "tests" or "test" directory under the package root.
So if you navigate to a package's root directory (containing a package.json file) and execute cfx test, cfx will:
- initialize the tests directory using the "tests" value inside "package.json", or the top-level "tests/" or "test/" directory
- load every module it finds in the tests directory whose name starts with "test"
- call each function exported by that module, passing it a test object as an argument.
An add-on is a package, and the SDK's APIs are currently collected into two packages called "addon-kit" and "api-utils" (you can see them under the top-level "packages" directory).
Testing an Add-on
From the SDK root, navigate to "examples/reading-data" and explore the "tests" directory:
(addon-sdk)~/mozilla/addon-sdk/examples > cd reading-data/
(addon-sdk)~/mozilla/addon-sdk/examples/reading-data > ls tests
test-main.js
(addon-sdk)~/mozilla/addon-sdk/examples/reading-data > more tests/test-main.js
const m = require("main");
const self = require("self");
exports.testReplace = function(test) {
const input = "Hello World";
const output = m.replaceMom(input);
test.assertEqual(output, "Hello Mom");
var callbacks = { quit: function() {} };
// Make sure it doesn't crash...
m.main({ staticArgs: {} }, callbacks);
};
exports.testID = function(test) {
// The ID is randomly generated during tests, so we cannot compare it against
// anything in particular. Just assert that it is not empty.
test.assert(self.id.length > 0);
test.assertEqual(self.data.url("sample.html"),
"resource://reading-data-example-at-jetpack-dot-mozillalabs-dot-com-reading-data-data /sample.html");
};
So this add-on implements two unit tests. You can see that each test expects a test object as an argument, and that test defines functions like assert, assertEqual and timeout functionality like waitUntilDone. To learn more about what you can do with a test object check the api documentation.
If you type cfx test it will execute these two tests. We'll use the -v option to make it show us the test names:
(addon-sdk)~/mozilla/addon-sdk/examples/reading-data > cfx test -v
Using binary at '/Applications/Firefox.app/Contents/MacOS/firefox-bin'.
Using profile at '/var/folders/me/me2B1lFDE0WZCgd33s2OTE+++TU/-Tmp-/tmpASJfvx.mozrunner'.
Running tests on Firefox 6.0/Gecko 6.0 ({ec8030f7-c20a-464f-9b0e-13a3a9e97384}) under Darwin/x86_64-gcc3.
info: executing 'test-main.testReplace'
info: pass: a == b == "Hello Mom"
info: My ID is reading-data-example@jetpack.mozillalabs.com
info: executing 'test-main.testID'
info: pass: assertion successful
info: pass: a == b == "resource://reading-data-example-at-jetpack-dot-mozillalabs-dot-com-reading-data-data /sample.html"
3 of 3 tests passed.
OK
Total time: 6.058659 seconds
Program terminated successfully.
Testing an SDK Package
An SDK package like "addon-kit" or "api-utils" is a package too, so the procedure is the same. Executing cfx test from the root of "addon-kit" will execute all the following test suites:
(addon-sdk)~/mozilla/addon-sdk > (addon-sdk)~/mozilla/addon-sdk > cd packages/addon-kit/ (addon-sdk)~/mozilla/addon-sdk/packages/addon-kit > ls tests pagemod-test-helpers.js test-notifications.js test-request.js test-clipboard.js test-page-mod.js test-selection.js test-context-menu.html test-page-worker.js test-simple-storage.js test-context-menu.js test-panel.js test-tabs.js test-hotkeys.js test-passwords.js test-widget.js test-module.js test-private-browsing.js test-windows.js
This isn't ideal if you only want to test a single module, so you can use the -f option to cfx to filter the set of tests to run. This option takes a regex and runs only the test suites whose names match the regex. This example runs only the tests for the "clipboard" module:
(addon-sdk)~/mozilla/addon-sdk/packages/addon-kit > cfx test -f clipboard -v
Using binary at '/Applications/Firefox.app/Contents/MacOS/firefox-bin'.
Using profile at '/var/folders/me/me2B1lFDE0WZCgd33s2OTE+++TU/-Tmp-/tmpVLFIqe.mozrunner'.
Running tests on Firefox 6.0/Gecko 6.0 ({ec8030f7-c20a-464f-9b0e-13a3a9e97384}) under Darwin/x86_64-gcc3.
info: executing 'test-clipboard.testWithNoFlavor'
info: pass: assertion successful
info: pass: a == b == "text"
info: pass: a == b == "hello there"
info: pass: a == b == "hello there"
info: pass: a == b == "hello there"
info: executing 'test-clipboard.testWithFlavor'
info: pass: assertion successful
info: pass: a == b == "html"
info: pass: a == b == null
info: pass: a == b == "hello there"
info: pass: a == b == "hello there"
info: executing 'test-clipboard.testWithRedundantFlavor'
info: pass: assertion successful
info: pass: a == b == "text"
info: pass: a == b == "hello there"
info: pass: a == b == "hello there"
info: pass: a == b == "hello there"
info: executing 'test-clipboard.testNotInFlavor'
info: pass: assertion successful
info: pass: a == b == null
17 of 17 tests passed.
OK
Total time: 2.909257 seconds
Program terminated successfully.
cfx testcfx
cfx testcfx tests the cfx tool itself. cfx is written in Python, and testcfx uses Python's unittest framework.
cfx test code is implemented by a collection of Python modules under "python-lib/cuddlefish/tests". Each module defines a subclass of the unittest module's TestCase class, and implements one or more test functions, each of which begins with the string "test". For example, here's what the "test_rdf" module looks like:
(addon-sdk)~/mozilla/addon-sdk > cd python-lib/cuddlefish/tests
(addon-sdk)~/mozilla /addon-sdk/python-lib/cuddlefish/tests > more test_rdf.py
import unittest
import xml.dom.minidom
from cuddlefish import rdf
class RDFTests(unittest.TestCase):
def testBug567660(self):
obj = rdf.RDF()
data = u'\u2026'.encode('utf-8')
x = '<?xml version="1.0" encoding="utf-8"?><blah>%s</blah>' % data
obj.dom = xml.dom.minidom.parseString(x)
self.assertEqual(obj.dom.documentElement.firstChild.nodeValue,
u'\u2026')
self.assertEqual(str(obj).replace("\n",""), x.replace("\n",""))
cfx testcfx builds a TestSuite out of all the test cases it finds, then uses TestTestRunner to execute them.
It also looks for DocTests in the documentation files, and executes any that it finds. But there aren't any, at the moment.
cfx testex
cfx testex runs cfx test for all the add-ons which ship with the SDK under the examples/ directory.
cfx testpkgs
cfx testpkgs finds all packages installed under the packages/ directory, and executes cfx test for each package it finds.
cfx testall
cfx testall tests everything:
cfx testcfxcfx testexcfx testpkgs
