QA/Desktop Firefox/Walkthroughs/Rapid Betas

From MozillaWiki
Jump to: navigation, search

Rapid Betas

The following documents the changes made to the above processes for adopting more frequent Beta builds. As of 2013-06-25 (Firefox 23) we are adopting twice-weekly Betas.

Timeline

Week Monday Tuesday Wednesday Thursday Friday
1 Beta migration Beta 1 built Beta 1 QA sign-off
2 Beta 2 built Beta 2 QA sign-off Beta 3 built Beta 3 QA sign-off
3 Beta 4 built Beta 4 QA sign-off Beta 5 built Beta 5 QA sign-off
4 Beta 6 built Beta 6 QA sign-off Beta 7 built Beta 7 QA sign-off
5 Beta 8 built Beta 8 QA sign-off Beta 9 built Beta 9 QA sign-off
6 Beta 10 built Beta 10 QA sign-off
Release migration
RC go-to-build
7 RC QA pre-sign-off
Mirror push
RC QA sign-off
Live push
Live QA sign-off

Sign-off Criteria

Beta 1

  • Features signed off by feature owners as "beta ready"
  • Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
  • >80% Critical bug fixes verified (as identified by QA lead)
  • Manual regression tests in Moztrap uncovers 0 blocking regressions
    • Tests: Breakpad, Mac OSX 10.8 code-signing, NVidia 3D Vision, and Rel-noted features smoketests
    • Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
  • Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
    • betatest and releasetest channels tested before sign-off
    • beta channel tested after Release Engineering pushes the snippets
  • Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters

Beta 2-9

  • Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
  • Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
    • betatest and releasetest channels tested before sign-off
    • beta channel tested after Release Engineering pushes the snippets
  • Manual testing to occur daily outside of the sign-off process:
    • Week 2
      • Tests: Breakpad, MD5 hash algorithm, and Plug-in compatibility exploratory smoketests
      • Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.7, Ubuntu Linux 64-bit
    • Week 3
      • Tests: Breakpad, Mac OSX 32-bit mode, and Web compatibility exploratory smoketests
      • Platforms: Windows Vista, Windows 7 32-bit, Mac OSX 10.6, Ubuntu Linux 32-bit
    • Week 4
      • Tests: Breakpad, Software update for root-owned files, and Add-on compatibility exploratory smoketests
      • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
    • Week 5
      • Tests: Breakpad, What's New page for software update, and Plug-in compatibility exploratory smoketests
      • Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.7, Ubuntu Linux 32-bit
  • Bug fix verifications to occur daily outside of the sign-off process, supported by triage on Wednesdays & Fridays
  • Testdays should be organized during Week 2 and Week 5 incorporating bug triage, regression testing, and feature testing
  • Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) consulted weekly to uncover any red flags from early adopters

Beta 10

  • Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
  • >90% critical bug fixes verified (as identified by QA lead)
  • Manual regression tests in Moztrap uncovers 0 blocking regressions
    • Tests: Breakpad, NVidia 3D Vision, Mac OSX 10.5 & Windows 2000 de-support, Web compatibility exploratory smoketests
    • Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.6, Ubuntu Linux 64-bit
  • Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
    • betatest and releasetest channels tested before sign-off
    • beta channel tested after Release Engineering pushes the snippets
  • Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters

Release Candidate

  • Features signed off by feature owners as "release ready"
  • Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
  • All critical bug fixes verified (as identified by QA lead)
  • Manual regression tests in Moztrap uncovers 0 blocking regressions
    • Tests: Breakpad, Partner repack update, and Windows share compatibility smoketests
    • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
  • Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
    • betatest tested before conditional sign-off
    • releasetest tested aftet Release Engineering pushes snippets to mirrors
    • release channel tested after Release Engineering pushes snippets to release
  • Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters
  • Important: Due to an intermittent build-time stability issue related to Radeon drivers we need to do twin builds
    • Once build1 is available run functional and betatest update automation
    • Once automation is complete, send an email to Release Engineering to generate build2
    • Once build2 is available run functional and betatest update automation on it
    • build1 will be pushed to releasetest on Monday if it's not found to have the stability issue
    • build2 will be pushed to releasetest on Monday if build1 is is found to have the stability
    • All manual testing and bug fix verifications should still be tested against build1