B2G/QA/API Permissions Test Plan

From MozillaWiki
< B2G‎ | QA
Jump to: navigation, search

Boot2Gecko API Permissions Test Plan

Summary

Leads David Chan (irc: dchan)

Geo Mealer (irc: geo)
Paul Theriault (irc: ptheriault)

Contributors TBD
References Firefox OS Security Model Overview

Scope

This test plan covers Gecko permissions interaction from the point an app has installed into the permissions database downwards. It does not cover installation flow of the app itself, aside from verifying that the permissions manager properly reflects installed permissions.

Strategy

The original testing methodology for this was a simple round trip: we would install an app with a given set of permissions, then check that the app behaved consistently with those permissions. However, we hit a number of problems where testing dependencies were either unstable or not fully implemented and would interfere with one of more legs of this multi-step process.

This test plan takes a more isolated approach to testing permissions interaction, so that AUT instability doesn't block the entire test approach. As such, it's split into three major sections: what permissions look like by default, whether a (non-default) permission setting is properly honored by the API, and whether installed apps properly set permissions in the manager.

These test suites will likely be implemented in mochitest, with a possible dependency on mochitest-chrome for the third suite to allow completion of install flow.

Suites

Suite 1: Verify default permissions

Given an app context, and without modifying its permissions, are the default permissions as expected?

Tracking Bug

Methodology

  1. Install or host an app with a context of either content, hosted, installed, or certified.
  2. Don't include a custom manifest or otherwise modify the permissions table.
  3. Query permissions manager directly for app and check against expected values.

Dependencies

  • Install flow
  • Permissions manager

Suite 2: Verify API behavior

Given permissions in the database, does API allow/deny/prompt as expected?

Tracking Bug

Methodology

  1. Start from the context with lowest permissions (content, probably).
  2. Add permissions directly via SpecialPowers
  3. Test API allows/denies/prompts correctly.

Dependencies

  • Individual APIs
  • Permissions Manager

Suite 3: Verify permissions database population

Given an app context and manifest, does the permissions database populate correctly?

Tracking Bug

Methodology

  1. Install (or host) application with customized manifest
  2. Query permissions database directly for app and check against expected values.

Dependencies

  • Installation flow
  • Permissions Manager

Design Notes

In combination, these suites give similar assurance as round trip, but have distinct advantages on a dependency front. The biggest dependency becomes the permissions manager. Even with APIs broken we get verification of default permissions and population, and even with install flow broken we get verification of APIs.

The biggest downside of this approach is that APIs that check the app context directly (as opposed to honoring the permissions manager) will cause problems. If the API blocks the content context unconditionally, it can't be tested via Suite 2. If it passes some other context unconditionally, we wouldn't catch it here.

Augmenting Suite 2 by testing from each of the different contexts should catch each of those scenarios, but reintroduces a dependency on install flow in that suite. Once the code base has stabilized, this would be a worthy expansion to the tests.