Firefox OS/Performance/Memory acceptance criteria
This page intends to define the criteria by which we say "memory management is good enough to ship", and test plans we can use to check those criteria.
Terminology
Foreground. For an application X, part of X's user interface is visible onscreen.
Perceivable. For an application X, no part of X's user interface is visible onscreen, but a user can perceive effects of X. For example, a music player that runs while its user interface is not visible is perceivable.
Background. For an application X, no part of X's user interface is visible onscreen, and it is not perceivable.
System message. A notification generated by Gecko which is dispatched to a certified application X that has registered to listen to the message. If application X is not running when a system message is sent to it, X is launched. For example, an incoming telephone generates a system message that the Dialer application listens for.
Activity. A request generated by a user as part of an interaction with an application X. Activities target an application Y. If Y is not running when an activity targets it, Y is launched. Y may send back data to X as part of the activity. For example, choosing an avatar for a contact generates an activity which may be targeted at the Camera, Gallery, or many other applications.
Criteria
These criteria are only defined for "core Gaia applications" (TODO: enumerate these). Firefox OS developers have no direct control over third-party applications.
MANDATORY: Users' explicit launch of every application X (tap X's icon on homescreen) results in X being loaded and its user interface brought to the foreground.
MANDATORY: Every application X launched by a system message results in X being loaded and provides X the opportunity to display a user interface.
MANDATORY: For every activity initiated by a user in application X that
- Launches application Y
- Is "one-way", that is, application Y does not return data to X
application Y must be loaded and its user interface brought to the foreground.
MANDATORY: For every "activity" initiated by a user in application X that
- Launches application Y
- Is "two-way", that is, application Y must return data to X
application Y must be loaded and its user interface brought to the foreground. Application X must remain loaded while Y is in the foreground. When application Y returns data to X, X's user interface must be brought to the foreground.
MANDATORY: No foreground application is killed due to memory pressure while a perceivable or background application is alive.
MANDATORY: No perceivable application is killed due to memory pressure while a background application is alive.
QoS: The number of perceivable and background applications that stay alive over the workloads defined below does not regress, while meeting the mandatory criteria above. Informally, a best effort is made to maximize the number of background and perceivable applications that stay alive. Builds that regress the number of apps that stay alive under our workloads are rejected.
(TODO: define if or to what extent third-party code can affect these criteria
Constraints on user data
This section defines the extremes of user data that are considered within the scope of the conformance requirements in this document.
(TODO)
Gallery
- 1000 images? at what max size?
- max image size?
Workloads
This section defines the workloads by which the criteria above are evaluated. Along with each workload, the acceptance status is listed for the given commit IDs below.
MW0: [PASS] Every app is successfully launched into the foreground
MW1: [PASS] Music stays alive
MW2: [PASS] Active call stays active
MW3: [PASS] 112MB: (QoS) Maximum amount of memory that web apps can directly allocate (not including per-app memory)
MW4: [PASS] Apps are successfully launched into the foreground under memory pressure
MW0: Every app is successfully launched into the foreground
- Ensure wifi is enabled and a network has been successfully configured
- Long-tap POWER button, choose "Restart"
- Launch Dialer app, press HOME button
- Launch SMS app, press HOME button
- Launch Contacts app, press HOME button
- Launch Browser app, press HOME button
-
Launch Feedback app, press HOME button - Launch Camera app, press HOME button
- Launch Gallery app, press HOME button
- Launch FM Radio app, press HOME button
- Launch Settings app, press HOME button
- Launch Marketplace app, press HOME button
- Launch Maps app, press HOME button
- Launch Calendar app, press HOME button
- Launch Clock app, press HOME button
- Launch Cost Control app, press HOME button
- Launch E-Mail app, press HOME button
- Launch Music app, press HOME button
- Launch Video app, press HOME button
-
Launch Calculator app, press HOME button -
Launch Dev Marketplace app, press HOME button -
Launch Staging Marketplace app, press HOME button - Swipe to Everything.me screen, load the app UI
- Swipe down system tray UI. Tap Cost Control widget to launch Cost Control app.
MW1: Music stays alive
- Long-tap POWER button, choose "Restart"
- Launch Music app
- Start playing "Treasure Island"
- Press HOME button
- Run MW0 starting from step 3
MW2: Active call stays active
- Long-tap POWER button, choose "Restart"
- Pull down system tray, ensure Cost Control widget loads
- Press HOME button
- Launch
CalculatorFeedbackMusic app, press HOME button - Launch Clock app, press HOME button
- Launch FM Radio app, press HOME button
- Launch Camera app, press HOME button
- Launch SMS app, press HOME button
- Launch Browser app
- Load nytimes.com, wait for load to finish
- Press HOME button
- Launch Dialer app
- Initiate a call to any number
- Press HOME button
- Launch Contacts app
- Choose to add new contact
- Tap First Name field
- Enter name "John"
- Tap "Add Picture"
- Choose Gallery activity in popup
- Select image of woman at lecturn
- Do not crop
- Tap checkmark button in top-right
- Tap Done to add contact
- Press HOME button
- Launch Browser app
- Ensure that nytimes.com loads
- Tap green dialer UI strip at top of screen
- Tap red button to disconnect call
MW3: (QoS) Maximum amount of memory that web apps can directly allocate
- Long-tap POWER button, choose "Restart"
- Open settings app, go to "Display" and set "Screen timeout" to "Never"
- Go to "Device information", "More information", "Developer" and check "Remote debugging" and "Console enabled"
- Open browser app, navigate to http://people.mozilla.com/~cjones/membuster.html
- Tap the "Bust system memory" button
- Wait for the browser tab to crash
- Connect phone to USB debugging over adb and run the following command from the shell
adb logcat -d | grep membuster | tail -1 | awk '{ print $(NF) }'
MW4: Apps are successfully launched into the foreground under memory pressure
- Long-tap POWER button, choose "Restart"
- Open browser app, navigate to http://people.mozilla.com/~cjones/membuster.html
- Tap the "Bust processes" button
- Wait for example.com loads to finish in all tabs
- Place a call to phone from any number
In-flight improvements
See bug 797189 for the latest list.