B2G/QA/Process
Contents
QA Process
This page lists the relevant information to B2G QA processes.
Mozilla QA Process Overview
Please visit MDN QA page for an overview of quality assurance activities at Mozilla.
B2G Roadmap
Here is the overview of the current Firefox OS product roadmap. Check this page for new information regarding future releases.
B2G Test Planning
UX Specification for Each Release
B2G Test Plans
Note: Some information may be out of date, in which case you can notify the QA owner of the affected component(s).
New Device Test Plan
Email Test Plan
Browser Test Plan
Crash Reporting Test Plan
Graphics Test Plan
List of Firefox OS Modules/Dev Owners (link) in case you need to ask questions regarding the new features of the component.
Firefox OS functional teams breakdown (link)
QA owners for Firefox OS components (link)
B2G Test Process
Please review the following beforehand:
Before the B2G release, following QA activities should be completed:
- Go through the list of new features (check for feature-b2g flag in Bugzilla), and assess for the creation of new test cases and update/deprecate existing test cases as necessary
- Create and update test cases in Moztrap
- Execute regression test suite, incorporated with the new test cases, and verify all release blocker bugs are fixed
- Ensure the builds are passing daily smoketests
- Execute performance/stability test suite
Smoke Test / Dogfood Processes
Status Tracking
- Features on QA Dashboard
- Bugs on QA Dashboard
- How to use: put [COM=NFC] in 'QA whiteboard', the bug shows up nicely to the count of NFC box, feature page filters for feature-b2g flag, bug page filters for blocking-b2g flag.
- Mozilla Charts
- Meeting minutes
Bugzilla Howtos (link)
Moztrap Howtos
- Test case creation primer
- Moztrap General
- Moztrap Guideline for Firefox OS
- MozTrapHelper by Shing - recommend to add this to your Firefox browser.
Triage
Refer to this page for the detailed information on how we use Bugzilla for triage process.
Refer to this page for B2G triage process.
When to Block a Bug for Release
Should Block
- Features our product team has committed us to
- Major issue in new feature - especially those in which a large number of users will be impacted, or a smaller number of users will be significantly impacted
- Major identifiable regression (perf or otherwise)
- Non-localizable strings
- Top Crashes
- sec-high, sec-critical Security bugs
- Smoke-test regression (special-case insta-blocker)
- Data loss
- Issues that block partner certification (basically legal issues)
- Issues getting a lot of support calls with partners or on SUMO (exceptions need to be made after FC though)
- Issues critical around updates (especially if there's been a repro)
- Anything critical around the first time experience
- Major Dialer, SMS, and VM communication issues (lower bar than other blockers - it's a phone)
- Issues that prevent automated tests in established testsuites (visible test suites on b2g integration branches on TBPL) from running green at least 90% of the time.
- Certification Waivers
Should Not Block - any exceptions to these rules must be discussed on b2g-release-drivers@mozilla.org or with Release Management
- Enhancements
- New Features (Not committed for the release)
- New perf requirements (see enhancements)
- Non-critical string changes (think of the l10n, people)
- Polish and other minor issues
- Unfinished localization (except perhaps in the last 3 weeks)
- Issues requiring the user to "zoom in and out and in and out and in and out..."
- Issues in languages not being shipped in the version of B2G
- Bugs without clear STR or that are not reproducible
- Bugs that do not impact production phones or the simulator
Needs Decision
- Non-regression bugs that we chose not to block on in the last release
- (though, sometimes this is because partner said "we won't block this time if you fix in the next release)
Qanalysts Tasks
Following documents describe the tasks assigned to QAnalyst:
(If you cannot view these documents, ask ktucker for permission in #fxosqa. You must have a valid mozilla LDAP account)
- QAnalyst Daily Tasks
- Jenkins Reporting
- Automation Reporting
- Exploratory Tests
- Exploratory Tests with Resolved Fixes
- Getting Regression Window
- Executing Automation
- Application Coverage Spreadsheet
Marigold
Overview
Marigold (Cheng-Du, China) is Mozilla's QA outsource team. They were in charge of executing FxOS tests, and they covers following tasks requested by Mozilla. (You may need a valid Mozilla LDAP account to access following documents)
- Manual Test
- Daily smoke test
- Regression test
- Full test
- RAT test
- Bug work (qawanted,regressionwindow-wanted,verifyme)
- Power consumption measurement
- Test automation - Gaia-UI-test
- Patch verification