QA/Desktop Firefox/Walkthroughs/Rapid Betas
From MozillaWiki
< QA | Desktop Firefox | Walkthroughs
Contents
Rapid Betas
The following documents the changes made to the above processes for adopting more frequent Beta builds. As of 2013-06-25 (Firefox 23) we are adopting twice-weekly Betas.
Timeline
Week | Monday | Tuesday | Wednesday | Thursday | Friday |
1 | Beta migration | Beta 1 built | Beta 1 QA sign-off | ||
2 | Beta 2 built | Beta 2 QA sign-off | Beta 3 built | Beta 3 QA sign-off | |
3 | Beta 4 built | Beta 4 QA sign-off | Beta 5 built | Beta 5 QA sign-off | |
4 | Beta 6 built | Beta 6 QA sign-off | Beta 7 built | Beta 7 QA sign-off | |
5 | Beta 8 built | Beta 8 QA sign-off | Beta 9 built | Beta 9 QA sign-off | |
6 | Beta 10 built | Beta 10 QA sign-off Release migration RC go-to-build |
|||
7 | RC QA pre-sign-off Mirror push |
RC QA sign-off Live push Live QA sign-off |
Sign-off Criteria
Beta 1
- Features signed off by feature owners as "beta ready"
- Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
- >80% Critical bug fixes verified (as identified by QA lead)
- Manual regression tests in Moztrap uncovers 0 blocking regressions
- Tests: Breakpad, Mac OSX 10.8 code-signing, NVidia 3D Vision, and Rel-noted features smoketests
- Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
- Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
- betatest and releasetest channels tested before sign-off
- beta channel tested after Release Engineering pushes the snippets
- Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters
Beta 2-9
- Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
- Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
- betatest and releasetest channels tested before sign-off
- beta channel tested after Release Engineering pushes the snippets
- Manual testing to occur daily outside of the sign-off process:
- Week 2
- Tests: Breakpad, MD5 hash algorithm, and Plug-in compatibility exploratory smoketests
- Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.7, Ubuntu Linux 64-bit
- Week 3
- Tests: Breakpad, Mac OSX 32-bit mode, and Web compatibility exploratory smoketests
- Platforms: Windows Vista, Windows 7 32-bit, Mac OSX 10.6, Ubuntu Linux 32-bit
- Week 4
- Tests: Breakpad, Software update for root-owned files, and Add-on compatibility exploratory smoketests
- Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
- Week 5
- Tests: Breakpad, What's New page for software update, and Plug-in compatibility exploratory smoketests
- Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.7, Ubuntu Linux 32-bit
- Week 2
- Bug fix verifications to occur daily outside of the sign-off process, supported by triage on Wednesdays & Fridays
- Testdays should be organized during Week 2 and Week 5 incorporating bug triage, regression testing, and feature testing
- Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) consulted weekly to uncover any red flags from early adopters
Beta 10
- Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
- >90% critical bug fixes verified (as identified by QA lead)
- Manual regression tests in Moztrap uncovers 0 blocking regressions
- Tests: Breakpad, NVidia 3D Vision, Mac OSX 10.5 & Windows 2000 de-support, Web compatibility exploratory smoketests
- Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.6, Ubuntu Linux 64-bit
- Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
- betatest and releasetest channels tested before sign-off
- beta channel tested after Release Engineering pushes the snippets
- Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters
Release Candidate
- Features signed off by feature owners as "release ready"
- Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
- All critical bug fixes verified (as identified by QA lead)
- Manual regression tests in Moztrap uncovers 0 blocking regressions
- Tests: Breakpad, Partner repack update, and Windows share compatibility smoketests
- Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
- Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
- betatest tested before conditional sign-off
- releasetest tested aftet Release Engineering pushes snippets to mirrors
- release channel tested after Release Engineering pushes snippets to release
- Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters
- Important: Due to an intermittent build-time stability issue related to Radeon drivers we need to do twin builds
- Once build1 is available run functional and betatest update automation
- Once automation is complete, send an email to Release Engineering to generate build2
- Once build2 is available run functional and betatest update automation on it
- build1 will be pushed to releasetest on Monday if it's not found to have the stability issue
- build2 will be pushed to releasetest on Monday if build1 is is found to have the stability
- All manual testing and bug fix verifications should still be tested against build1