OBSOLETE

Please note that this test plan is now obsolete, and is no longer underway.

Boot2Gecko WebAPI Test Plan

Summary

Scope

This test plan covers qualification of WebAPIs developed for or used specifically in Boot2Gecko to interface with mobile devices. It does not cover the entirety of Gecko APIs available. The individual APIs covered are listed below and broken out into their own individual sub-plans.

Currently, only functional testing is being planned. Ultimately, performance and security should be addressed. However, there are too many unknowns at time of writing to adequately plan for them in advance.

Strategy

Functional test strategy will vary between APIs based on whether API effects can be effectively verified via automation, and whether that automation requires an emulator or not.

Broadly, testing will be divided between automation executed in continuous integration, and interactive exercises designed to manipulate APIs directly on-device. The expected amount of time given to each is roughly 75% to planning and CI implementation and 25% to implementing interactive exercises. However, as the exercises will be necessary for final qualification of the B2G stack on-device, those priorities may change closer to the shipping date.

CI-driven Automation

B2G continuous integration will offer two options: mochitest-plain and a new Marionette-based JS harness.

Tests written against the new harness will run only in B2G CI against QEMU, a device emulator. They can set things like orientation or battery status to a known value, which allows for rich virtual device manipulation for fixture setup.

Tests written against mochitest-plain can run in the standard mozilla-central Tinderbox test environment and anywhere else mochitests are typically run, as well as against the B2G codebase. Mochitest will give wider coverage, but will not be able to use the emulator to set up device-related fixtures and is thus significantly more limited.

Interactive Exercises

Some APIs have effects that cannot be verified through unattended automation. One example would be whether a microphone is open and receiving sound. Another example would be whether the phone is actually vibrating in response to an API call. These APIs must be verified interactively.

It would be tempting to use Gaia to exercise the API, but Gaia apps cannot be guaranteed to be using the APIs in a controlled fashion; that would require inappropriately locking in their implementation. So for these APIs, it is best to test using code specifically designed to exercise the APIs in a known fashion. In addition, as CI automation cannot be currently run on a final target device, it may be useful to have exercises even for APIs we address primarily through CI.

One decision is whether these exercises will be implemented as web pages or installable apps. Apps would be closer to how a developer will mostly use these APIs, and some APIs may only work for standalone applications. However, pages are more convenient as they don't require being preloaded onto the phone before testing can occur; they can just be navigated via onboard browser.

Tentatively, the plan is to implement exercises as pages where possible for expediency's sake, but to eventually move them into a standalone app that developers and the testing community can experiment with.

Also see the On Device Test Plan.

API Sub-Plan List

Template for new Sub-Plans can be found here.