QA/E10s Test Plan
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 03/02/2015 | 1.0 | Brindusa Tot | Created first draft |
Contents
Overview
Scope
This is the general QA Test Plan for Electrolysis ("e10s" for short). With e10s enabled, Firefox content runs in a different process than the browser itself. This will help to improve general performance and security. Our goals of testing are to identify areas of concerns and potential risk and to be able to give a recommendation when e10s will be able to be pushed to a wider audience (release).
The goal of the current wiki page is to gather the testing efforts done in support of the Electrolysis project. It defines the overall testing requirements and provides an integrated view of the test activities.
Enabling and Disabling Electrolysis
On current Nightly and Aurora versions, e10s is enabled by default. To disable e10s, open Preferences->General and un-check the "Enable multi-process" check-box and then restart the browser.
Ownership
Manager, Firefox Quality Engineering - Ryan VanderMeulen
PM, QA Engineeting Team - Rares Bologa
QA Engineering Team - Brindusa Tot
PM, Mozilla Release QA Team - Florin Mezei
Mozilla Release QA Team - Andrei Vaida
Testing summary
Scope of Testing
In Scope
Full testing of Firefox browser will be performed. Testing will be performed against Nightly, Aurora and Beta builds. Areas that need to be included in testing are:
- User Engagement
- Stability
- Scrolling and Zooming
- UI Smoothness
- Page Load
- Plugins
- Graphics
- Startup/Shutdown Time
- Memory Usage
- Performance
Not In Scope
Add-on Compatibility and Accessibility are not in the scope of current release. New test plans will be created for these areas of functionality once ready for testing. Link available for e10s Add-ons compatibility is http://arewee10syet.com.
Test types
Manual testing is performed by both Mozilla QA Engineering and Mozilla Release QA teams.
We will focus our testing on the at-risk features identified by the engineering teams, validation of experiment code functionality prior to deployment, assistance with bug triage and verification, general functional and exploratory testing
Automated Testing
A large number of automated tests are written and run against different OSes. There is a dedicated team led by Blake Kaplan that is responsible for ensuring that automated tests are run where possible with e10s enabled in continuous integration.
Note: Because of constraints on Windows hardware test pools, automated tests on e10s are not enabled on Windows XP and Windows 8 on production branches. Instead, Windows 7 is the focus of all e10s test coverage on production branches for Gecko 47+. The Ash project branch which tracks mozilla-central was also created to provide test coverage across all desktop platforms where resource constraints are an issue.
Details about test coverage can be found at Test Coverage link
Requirements for testing
Environments
It is desirable for testing to be performed on a wide range of environments, but focus will be on:
- Windows 7
- Windows 10
- OSX 10.10 (OSX 10.6-10.8 are EOL)
- Ubuntu 14/15
For testing, we will alternate 32 and 64 FF builds and we will do the same with the above mentioned OSes.
Notes:
Testing will also be performed on lower-end hardware, like single core machines with Windows 7 or Windows Vista. The testing focus on these machines will focus on graphics and performance/responsiveness.
Builds
This section should contain links for builds with the feature -
- Nightly builds are available at link
- Aurora builds are available at link
- Beta builds are available at link
Bug Management
Any issue identified during testing will be logged in Bugzilla. When filing a new e10s bug, the template from the following link: http://is.gd/aTza8A, will be used.
The tracking-e10s:? flag will be set so that it shows up in the e10s team's weekly triage bug list. Each e10s bug will be triage by engineering teams and prioritized based on severity.
Features covered
e10s specific testing
Manual testing with e10s enabled has been performed for the below features and all issues found have also been verified with e10s disabled prior to logging them.
Full functional testing
General testing was performed on Aurora 46 and Beta 45 on the following browser areas: Audio Compatibility, Video Compatibility, Web Compatibility, Geolocation, Safe Browsing, Plug-in compatibility, WEBGL Compatibility, PDF Compatibility, Image support, Crash Reporter, Location bar, Search Suggestions, Sync, Downloads, Bookmarks and History, READER VIEW & READABILITY.JS, Browser customization, Scroll and Zoom
| QA Manager contact | Ryan VanderMeulen |
|---|---|
| QA Contact Engineering Team | Michelle Funches |
| QA Contact Desktop Team | Andrei Vaida |
| Team Responsible | FF Version tested | Test type | Test Cases | Automation | e10s bugs | Signoff | Signoff Date |
|---|---|---|---|---|---|---|---|
| Moz QA Eng | Aurora 46 | Manual | Aurora_test | - | n/a | 03.10.2016 | |
| Desktop QA Team | Beta 45 | Manual | Beta_tests | - | n/a | ||
| Desktop QA Team | Beta 46 | Manual | Beta_tests | - |
n/a |
n/a |
Password Manager
Tested Password Manager against Nightly 47.
| QA Manager contact | Ryan VanderMeulen |
|---|---|
| QA Contact | Vlad Bacia |
| Team Responsible | FF Version tested | Test Plan | Test type | Test Cases | Automation | e10s bugs | Signoff | Signoff Date |
|---|---|---|---|---|---|---|---|---|
| Moz QA Eng | Nightly 47 | - | Manual | tc | TBD | - | - |
- Meta bug on Password Manager - 1118955
APZ bugs verification
| Engineering Manager, Pan/Zoom | Kartikaya Gupta |
|---|---|
| QA Manager contact | Ryan VanderMeulen |
| QA Contact | Brindusa Tot |
On this area, Kats' team was assisted with bug triage on APZ bugs and verification of major impact bugs.
Hello(loop)
| Engineering Manager, Hello | Ian Bicking |
|---|---|
| Firefox Hello Desktop Technical Lead | Mark Banner |
| QA Manager contact | Ryan VanderMeulen |
| QA Contact | Brindusa Tot |
| Team Responsible | FF Version tested | Test Plan | Test type | Test Cases | Automation | e10s bugs | Signoff | Signoff Date |
|---|---|---|---|---|---|---|---|---|
| Moz QA Eng | Nightly 48 | Hello TP | Manual | Hello TC | - | - |
- Meta bug on Hello - 1226706
E10s Experiment Testing
Tested Experiments on Beta 45 and Nightly 47.
| QA Manager contact | Ryan VanderMeulen |
|---|---|
| QA Contact | Michelle Funches |
| Team Responsible | FF Version tested | Test Plan | Test type | Test Cases | Automation | e10s bugs | Signoff | Signoff Date |
|---|---|---|---|---|---|---|---|---|
| Moz QA Eng | Multi-process Firefox A/B test Beta 45 | TP | Manual | TC | TBD | - | Yes | 01/28/2016 |
| Moz QA Eng | Multi-process Firefox A/B Experiment Beta 2nd Experiment | TP | Manual | TC | TBD | - | Yes | 02/11/2016 |
| Moz QA Eng | DisplayPort Size Tuning aka Checkerboard Experiment | TP | Manual | TC | TBD | - | Yes | 03/02/2016 |
- Meta bug on Multi-process Firefox A/B test Beta 45 - 1238802
- Meta bug on Multi-process Firefox A/B Experiment Beta 2nd Experiment - 1244187
- Meta bug on DisplayPort Size Tuning aka Checkerboard Experiment - 1251052
Other Features tested generally
The following features have been tested on Nightly channel: functional validation has been performed for each feature with e10s enabled and disabled.
Push Notifications
| Development contact | Edwin Wong |
|---|---|
| QA Manager contact | Ryan VanderMeulen |
| QA Contact | Michelle Funches |
| Team Responsible | FF Version tested | Test Plan | Test type | Test Cases | Automation | e10s bugs | Signoff | Signoff Date |
|---|---|---|---|---|---|---|---|---|
| Moz QA Eng | Nightly 45 | Test Plan | Manual | tc | TBD | [DONE] | 01/06/2016 |
- Meta bug: bug 1201571
Synced Tabs Sidebar
| Development contact | Edwin Wong |
|---|---|
| QA Manager contact | Ryan VanderMeulen |
| QA Contact | Brindusa Tot |
| Team Responsible | FF Version tested | Test Plan | Test type | Test Cases | Automation | e10s bugs | Signoff | Signoff Date |
|---|---|---|---|---|---|---|---|---|
| Moz QA Eng | Nightly 47 | Push Notification TP | Manual | tc | TBD | Conditional Signoff | 03.08.2016 |
- Meta bug: bug 1214379
Activity Stream
| Development contact | Edwin Wong |
|---|---|
| QA Manager contact | Ryan VanderMeulen |
| QA Contact | Paul Oiegas Vlad Bacia |
| Team Responsible | FF Version tested | Test Plan | Test type | Test Cases | Automation | e10s bugs | Signoff | Signoff Date |
|---|---|---|---|---|---|---|---|---|
| Moz QA Eng | Nightly 48 | Activity Stream TP | Manual | tc | TBD | - | - |
- Meta bug: bug [-]
Risk analysis
Bug Handling/Triage
- Bug triage has been mostly focusing on bugs specifically flagged and nominated as e10s issues. We may be missing incoming issues that are filed as generic bugs and don't get the right e10s nominations. This can be possibly mitigated by more focused review of all incoming bugs.
- Bug prioritization is based on subjective opinions about the relative severity of a reported issue. This is mitigated to some extent by these decisions being made in a group setting with representatives from different stakeholders present and able to make their case.
Test Coverage
- The Engineering QA team focus has been on validating e10s functionality primarily on the Nightly channel, and not on channels which are closer to potential release. Only when specific requests are received from engineering teams to validate e10 functionality on other channels (like Aurora) has focused testing been performed.
- The Softvision QA teams are small relative to the overall number of users, even on pre-release channels, and we have a relatively homogenous set of hardware we're testing on. We acknowledge the risk of missing bugs due to this lack of diversity. This risk is at least somewhat mitigated by shipping e10s enabled by default to our Nightly and DevEdition populations and by the recent experiments on Beta.
- Manual testing will never be an exhaustive way of testing changes that affect the entire codebase. By choosing to focus on specific features for targeted testing, we are intentionally *not* exhaustively testing other functionality in a deliberate manner. This is hopefully mitigated by the ongoing efforts to expand our automated test coverage across a larger percentage of the test suite and across a wider variety of platforms in CI.
- Our automated test infrastructure is severely resource constrained, limiting the number of platforms we can test on and the frequency with which those tests can be run. We've partially mitigated some of these issues by using the Ash branch to run the test suite against a larger number of platforms, but this only tracks mozilla-central, leaving us potentially exposed to uncaught problems or missed backports on release branches.
- Intermittent test failures occur at high frequency in our automated test infrastructure, making it difficult to find and track down new issues that get lost in the noise. Intermittent test failures are generally prioritized below other work, adding further difficulty to this. We don't have a great mitigation for this at this point in time, other than disabling flaky tests and accepting the loss in test coverage that results from it.
At-Risk Features/Functionality
- The initial rollout plan is for only users with add-ons disabled. Code has been added to detect that at runtime and may possibly have edge cases where users end up with e10s enabled that shouldn't. This has been mostly-mitigated by ongoing exploratory testing and crash/Telemetry data analysis and verification.
- Significant changes have been made to the graphics pipeline to support accelerated graphics over IPC. Prior experience has shown that any major changes to graphics can cause regressions that are difficult to find until the changes are shipping to release populations. We need to ensure that we have effective blocklisting mechanisms in place and as thorough testing as possible to minimize unanticipated regressions and point releases once e10s ships.
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | Yes | |
| Theme (high contrast) | Yes | |
| UI | ||
| Mouse-only operation | Yes | |
| Keyboard-only operation | Yes | |
| Display (HiDPI) | Yes | |
| Interraction (scroll, zoom) | Yes | |
| Usable with a screen reader | No | e.g. with NVDA |
| Help/Support | ||
| Help/support interface required | No | Make sure links to support/help pages exists and are easy reachable. |
| Support documents planned(written) | No | Make sure support documents are written and are correct. |
| Install/Upgrade | ||
| Feature upgrades/downgrades data as expected | Yes | |
| Does sync work across upgrades | Yes | |
| Requires install testing | No | separate feature/application installation needed (not only Firefox) |
| Affects first-run or onboarding | ??? | Florin/Lawrence are investigating if there is a dedicated QA for this. If this must be tested for current feature, then we should add in detail column the team/person assigned. |
| Does this affect partner builds? Partner build testing | ??? | yes-no options, add comment with details about who will lead testing |
| Enterprise | Raise up the topic to developers to see if they are expecting to work differently on ESR builds | |
| Enterprise administration | N/A | |
| Network proxies/autoconfig | N/A | |
| ESR behavior changes | N/A | |
| Locked preferences | N/A | |
| Data Monitoring | ||
| Temporary or permanent telemetry monitoring | Yes | List of error conditions to monitor |
| Telemetry correctness testing | ||
| Server integration testing | ? | |
| Offline and server failure testing | ? | |
| Load testing | ? | |
| Add-ons | No | Separate test plan, will be on a different release, as a separate ship criteria |
| Add-on API required | - | |
| Comprehensive API testing | - | |
| Permissions | - | |
| Testing with existing/popular add-ons | - | |
| Security | Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | |
| 3rd-party security review | ||
| Privilege escalation testing | ||
| Fuzzing | ||
| Web Compatibility | depends on the feature | |
| Testing against target sites | Yes | |
| Survey of many sites for compatibility | Yes | |
| Interoperability | depends on the feature | |
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | - | |
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | - | |
| Interaction of this feature with other browser features | Yes |
Sign off
Criteria
Check list
- All blocking bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA).
- There is a separate wiki page in regards to Electrolysis release criteria, at the following wiki page.
Results
For each tested feature, the links to the executed manual test-cases can be found at section "Features covered", see Features covered
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | [DONE] | |
| Testing Infrastructure setup | [DONE] | Make sure e10s is enabled |
| Test Plan Creation | [IN PROGRESS] | |
| Test Cases Creation | [IN PROGRESS] | |
| Full Functional Tests Execution | [IN PROGRESS] | |
| Automation Coverage | Blake Kaplan (mrbkap) | |
| Performance Testing | Need to see who is in charge | |
| All Defects Logged | [IN PROGRESS] | |
| Critical/Blockers Fixed and Verified | [IN PROGRESS] | Should verify that for each tested feature criticals and blockers are fixed and verified |
| QA Aurora - Full Testing | [DONE] | Email sent on 03/11/2016 |
| QA Push Notifications | [DONE] | Email sent on 01/06/2016 |
| QA Synced Tabs Sidebar | [DONE] | Email sent on 03/08/2016 |
| QA Next feature to be tested | Email to be sent |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | Google docs |
| Test case execution | Google docs |
| Bugs management | Bugzilla |
For Bugzilla, when filing a new e10s bug , use the template from the following link: http://is.gd/aTza8A
Please add the tracking-e10s:? flag or place 'e10s' in the bug title so that it shows up in the team's weekly triage bug list.
References
List and links related to E10s effort
Electrolysis wiki pages
Summary: Goal, Enabling and Disabling Electrolysis, What to Expect, Schedule, Contributing, Accessibility Support, Add-ons Compatibility, Past Milestones, Communication, People, Reference, Bug Lists, Meeting Notes, Weekly Status
Electrolysis Test Coverage wiki page
Summary: Table with Test coverage on diff OSes, Related Bugs, Individual Test Issues, Mochitest a11y, Mochitest chrome, Windows Coverage
Electrolysis Release Criteria
Summary: APZ Regressions, Release Criteria, User Engagement, Stability, Jank, Scrolling, Slow Scripts, UI Smoothness, Page Load, Plugin Jank, Graphics Performance, Startup/Shutdown Time, Memory Usage, Release Blocking Bugs, M8/M9 bugs, Release Criteria meta bugs' blockers, APZ Bugs, Tests, Accessibility Add-ons
Release Criteria Document: contains information related to criteria for releasing E10s specific to different browser functionalities, like Stability, Performance, Automation testing
E10s performance tasks meeting minutes (Feb-19)
E10s meetings agenda and notes
