Releases/Firefox 30/Test Plan

From MozillaWiki
Jump to: navigation, search

« Firefox 29 | Firefox 30 | Firefox 31 »

Firefox 30 Test Plan

Summary

Milestone Firefox 30
Status Currently in Beta
QA-Lead Tracy Walker (irc: tracy)
QA-2nd Robert Kaiser (irc: KaiRo)
Rel-Eng contact Ben Hearsum (irc: bhearsum)
Rel-Man contact Lukas Blakk (irc: lsblakk)

The following is the test plan for Firefox 30 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 30. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

  • 2014.02.03: [DONE] Nightly
  • 2014.03.17: [DONE] Aurora
  • 2014.04.25: [DONE] Pre-Beta feature sign-offs
  • 2014.04.29: [DONE] Beta 1
  • 2014.05.06: [DONE] Beta 2
  • 2014.05.09: [DONE] Beta 3
  • 2014.05.14: [DONE] Beta 4
  • 2014.05.16: [DONE] Beta 5
  • 2014.05.21: [DONE] Beta 6
  • 2014.05.23: [DONE] Beta 7
  • 2014.05.28: [DONE] Beta 8
  • 2014.05.30: [DONE] Beta 9
  • 2014.06.04: [DONE] Beta 10 (RC build 1)
  • 2014.06.06: [DONE] RC (build 2)
  • 2014.06.10: [DONE] Final

Features

Disabled

The following features have been deemed not ready for release and will be delayed to Firefox 31 or later.

On Track

Signed Off

The following features have no blockers and have been deemed ready for release.

Nightly

Fixed Bugs

How to Help
  • Download and install the latest Firefox Aurora build
  • Load this list of bug reports
  • Read each bug report until you find one you understand and want to test
  • Download and install the version of Firefox used to report the bug
  • Using the older build, test the claim made in the bug report to see if you can reproduce it
    • If you cannot reproduce the bug move on to another bug or ask the reporter to verify the fix
  • Using the latest Nightly, test to see if you can reproduce the bug
    • If you cannot reproduce the bug set the STATUS to VERIFIED FIXED and add a comment describing your testing
    • If you can still reproduce the bug set the STATUS to REOPENED and add a comment describing your testing
  • Add any bug reopened to the list below using the bug number template (verified fixes will be tracked elsewhere)
  • If you need help, send email to Tracy Walker with the bug summary in the Subject field
Reopened Bugs

Reported Bugs

Triage unconfirmed bugs.

How to Help
  • Download and install the latest Firefox Nightly
  • Load this list of bug reports
  • Read each bug report until you find one you understand and want to test
  • Test the claim made in the bug report and see if you can reproduce it
  • If you can reproduce the bug, set the STATUS to NEW, add a comment describing your testing, and try to find a regression window
  • Any confirmed bugs should be added to the list below using the bug number template
  • If you need help, send email to Tracy Walker with the bug summary in the Subject field
Confirmed Bugs

Stability

How to Help
  • Review the Firefox 30 Explosiveness Report
    • Any signatures rising significantly (>= 2.0 explosiveness rating) without a bug number should be reported to Bugzilla and nominated for tracking
    • If there is already a bug report, update the report with the latest numbers
    • Ensure the bug report has the topcrash-platform keyword added (where platform is the platform where crashes occur -- ie. topcrash-win for Windows)
    • Ensure the bug is listed below using the bug number template
  • Review the Firefox 30 Topcrash Report
    • Any signature in the top 10 without a bug number should be reported to Bugzilla and nominated for tracking
    • If there is already a bug report, update the report with the latest ranking information
    • Ensure the bug report has the topcrash-platform keyword added (where platform is the platform where crashes occur -- ie. topcrash-win for Windows)
    • Ensure the bug is listed below using the bug number template
  • Review the Firefox 30 tracked bugs that aren't tracked in 29 or 28.
    • Each bug listed needs to have its rank updated once per week
    • Use the topcrash reports from crash-stats to get the rank for each branch
    • Add this information to the bug report
    • Ensure the bug report has the topcrash-platform keyword added (where platform is the platform where crashes occur -- ie. topcrash-win for Windows)
    • Conversely, remove the topcrash keyword if the crash is no longer in the top-10
  • For each of these tasks, ensure the etherpad is updated with your name and the date checked
Topcrash Bugs

QA Wanted

How to Help
  • Download and install the latest Firefox Nightly
  • Review this list of bug reports to find a bug you both understand and want to test (each should contain a comment detailing what's needed from QA)
    • If the bug needs steps to reproduce, try to find a reproducible set of steps
    • If the bug needs a regression window, try to find a regression window
    • If the bug is fixed and needs verification, test to confirm you can reproduce the bug on an older build but not the latest build
  • Any tested bugs should be added to the list below using the bug number template
  • Remove the qawanted keyword from any fulfilled request
  • If you need help, send email to Tracy Walker with the bug summary in the Subject field
Tested Bugs

Testdays

Aurora

Branch Migration

Pre Sign-off Tasks

  • [DONE] Firefox branch migrated (Release Management)
  • [DONE] Mozmill Tests branch migrated (Automation Development)
  • [DONE] Updates throttled on aurora channel (Release Engineering)
  • [DONE] Updates verified throttled on aurora channel (QA)

Sign-off Tasks

  • [DONE] Mozmill Daily functional testruns pass with no failures ([report])
  • [DONE] Mozmill Daily auroratest update testruns pass with no failures ([report])
  • [DONE] Scope features tested and signed off for Aurora uplift

Post Sign-off Tasks

  • [DONE] Aurora sign-off email sent (QA)
  • [DONE] Updates unthrottled on aurora channel (Release Engineering)
  • [DONE] Updates verified unthrottled on aurora channel (QA)

Beta

Regression Testing

  • Run these tests for whatever platform(s) you have available.
  • Report a bug if you encounter one.
  • If you report a bug check to see if it's a regression by testing a previous Firefox version (if it doesn't reproduce in earlier versions it's a regression).
  • Add any reported bugs to the Bugs Found section below.
  • All results will appear in this dashboard

Beta 1

Build Info

Check List

  • [DONE] Regression testing (report) - FAIL
    • regression found: bug 1003839 - Live bookmark is still created despite cancelling the subscription
      • shipped to beta1 with this regression. it is tracked by Rel-Man to ensure it is fixed prior to final 30 release.
  • [DONE] Automation Functional tests (report) - PASS
  • [DONE] Updates on betatest channel (report) - PASS
  • [DONE] Updates on releasetest channel (report) - PASS
  • [DONE] Updates on beta channel (report)
    • bug 1004575 - ondemand_update test runs stalling out during recording results.

Regression Testing

  • Please run a brief smoketest for the features signed off on prior to merge to ensure they continue to be ready for our beta audience.
    • GStreamer 1.0 Support (Alexandra Lucinet)
    • Preference UI for proxy autologin (Bogdan Maris)
    • Places (Andrei Vaida)
    • Background-blend-mode (Catalin Varga)

Beta 2

Build Info

Check List

Regression Testing

Beta 3

  • shipped with tested failures note in regression testing reports. Further investigation is needed, but we'll ensure bugs are filed and tracked.

Build Info

Check List

Regression Testing

Beta 4

Build Info

Check List

Regression Testing

Beta 5

Build Info

Check List

  • [DONE] Regression testing (moztrap report) - PASS
  • [DONE] Automation Functional tests (report) - PASS
  • [DONE] Updates on betatest channel (report) - PASS
  • [DONE] Updates on releasetest channel (report) - PASS
  • [DONE] Updates on beta channel (report) - PASS

Regression Testing

Beta 6

Build Info

Check List

  • [DONE] Regression testing (Fix Verifications) - PASS
  • [DONE] Automation Functional tests (report) - PASS
  • [DONE] Updates on betatest channel (report) - PASS
  • [DONE] Updates on releasetest channel (report) - PASS
  • [DONE] Updates on beta channel (report) - PASS

Regression Testing

Beta 7

Build Info

Check List

  • [DONE] Regression testing (Fix Verifications) - PASS
  • [DONE] Automation Functional tests (report) - PASS
  • [DONE] Updates on betatest channel (report) - PASS
  • [DONE] Updates on releasetest channel (report) - PASS
  • [DONE] Updates on beta channel (report) - PASS

Regression Testing

  • Fix Verifications, focusing on those most recently fixed during this cycle.
    • bug 980714 - Blinking caret in panel text - VERIFIED
    • bug 1003839 - Live bookmark is still created despite cancelling the subscription. - VERIFIED

Beta 8

Build Info

Check List

Regression Testing

Beta 9

Build Info

Check List

Regression Testing

Release

Release Candidate Build 1

Build Info

Check List

Regression Testing

Release Candidate Build 2

Build Info

Check List

  • [DONE] Sanity Check - PASS
  • [DONE] Automation Functional tests (report) - PASS
  • [DONE] Updates on releasetest channel beta builds (report) - PASS
  • [DONE] Updates on beta channel (report) - PASS

Regression Testing

Final Release

  • [DONE] Updates on releasetest channel release builds (report) - PASS
    • [DONE] Spot checks for bug 1009893 - Show /whatsnew tour URL to users updating to 30.0 and 31.0 (from all past versions of Firefox) - PASS
  • [DONE] Updates on release channel release builds (report) - PASS
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_30/Test_Plan&oldid=987622"