Releases/Firefox 35/Test Plan
From MozillaWiki
< Releases
Contents
- 1 Summary
- 2 Features
- 3 Iterations
- 4 Stability
- 5 Bug Triage
- 6 Milestones
- 7 Statistics
Summary
This page is to track testing of Firefox 35 from mozilla-central (Nightly) through to mozilla-release.
Bug queries and graphs for this release and Firefox iterations: Firefox QA Dashboard
How you can help
Would you like to help test Firefox 35? Join us! Here's some things you can do:
- Do some basic setup tasks to get all the tools you need in place.
- Come to the next Bugzilla Tour.
- Set up multiple profiles in Firefox.
- Download and install Nightly (Firefox 35)
- Try some tasks in One and Done: Triage an unconfirmed bug; Verify a fixed bug
- Join our mailing list on dev-quality
- Hang out in our irc channels: irc.mozilla.org #qa and #testday
- If you have questions you can contact Robert Kaiser (:KaiRo)
Schedule
Please consult the Rapid Release Calendar for more information.
| Milestone | Date | Checks |
| Nightly | 2014-09-02 | Conduct daily stability and bug triage, ensure features are owned |
| Aurora Migration | 2014-10-13 | Conduct testing to ensure Firefox 35 Aurora builds are okay, sign off for updates on Friday |
| Aurora Updates enabled | 2014-10-17 | Continue daily stability and bug triage, ensure features are tested |
| Beta Uplift | 2014-11-25 | Review testing and sign off every beta release |
| Release | 2015-01-05 (?) | Review tests and sign off release |
Meetings
| Meeting | Purpose | When | Vidyo Room | IRC Backchannel |
| CrashKill | Raise stability concerns with the CrashKill team | 10:00 Pacific on Monday | Stability | #crashkill |
| Channel Status | Raise quality concerns with the Release Management team | 10:00 Pacific on Tuesday & Thursday | ReleaseCoordination | #planning |
| Firefox QA | Meet the Desktop QA team and hear what they're working on | 8:30am Pacific on Wednesdays | DesktopQA | #qa |
| QA Team | Meet the entire QA team and hear what they're working on! | 1:30am Pacific on Wednesdays | QA | #qa |
Checks
| Priority | Check | Description | Frequency | Last Checked |
| 1 | Stability | Review the stability reports to identify new/rising crash signatures | Daily | 2014-09-10 |
| 2 | Pushlog | Review the pushlog to identify fixed bugs needing verification | Daily | 2014-09-XX (not keeping up tho) |
| 2.5 | Iteration | Triage the iteration bugs, add QA contacts | Twice weekly | 2014-09-XX |
| 3 | QA Needed Bugs | Review and assign qawanted bugs for investigation | At least twice a week | 2014-09-XX |
| 4 | Unconfirmed Bugs | Triage and test unconfirmed bug reports | At least once a week | 2014-09-XX |
| 5 | Fixed Bugs | Flag fixed bugs for verification | At least once a week | 2014-09-XX |
| 6 | Verification Needed Bugs | Test verifyme bugs to confirm fixed bugs are fixed | At least once a week | 2014-09-XX |
| 7 | Reported Bugs | Document and escalated reported bugs as necessary | Every Monday | 2014-09-XX |
| 8 | Features | Ensure features are owned and on track for release | Once a week | 2014-09-XX |
| 9 | Automation | Ensure automation bugs are owned and on track for resolution | Once a week | 2014-09-XX |
Features
The following tracks features targeted for Firefox 35. For more information consult this wiki.
- Ensure all features which do not require testing are flagged qe-verify-
- Ensure all features which do require testing are flagged qe-verify+ and have an assigned QA Contact.
- Ensure all features are either signed-off or disabled before release
Web RTC - bundle in SDP handling
- Feature owner: Nils Olhmeier
- Status: Unknown (Backed out from 33)
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 784491 | SDP handling should support Bundle - parsing & generation work | RESOLVED | DUPLICATE |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
HTTPS for in-product URLs
- Feature owner: Camelia Badau
- Status: N/A - Tracking as individual sub features (implementation for specific URLs)
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 771788 | Use HTTPS instead of HTTP for in-product URLs | RESOLVED | WORKSFORME |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
In-content preferences
- Feature owner: Camelia Badau
- Status: Currently in Aurora 36
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 738796 | Visual polish for in-content preferences | VERIFIED | FIXED |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Places - Unified Autocomplete
- Feature owner: Andrei Vaida
- Status: Currently in Aurora 36
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 995091 | Enable Unified Autocomplete | RESOLVED | FIXED |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
Modal/non-Modal Dialogs
- Feature owner: Camelia Badau
- Status: Currently in Aurora 36
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 100180 | UI freezes if alert/dialog comes up while dragging (Modal dialog during drag causes hang) | VERIFIED | FIXED |
| 613785 | Switch to using tab-modal prompt dialogs for HTTP authentication | RESOLVED | FIXED |
| 616853 | The onbeforeunload dialog should be tab-modal | VERIFIED | FIXED |
| 951627 | Implementation: Use something other than a modal dialog to ask for setting the default browser | VERIFIED | FIXED |
4 Total; 0 Open (0%); 1 Resolved (25%); 3 Verified (75%);
Tracking protection
- Feature owner: Catalin Varga
- Status: Currently in Aurora 36
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1029886 | tracking bug for tracking protection | RESOLVED | FIXED |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
Hello button Tour
- Feature owner: Catalin Varga
- Status: Dev Work In Progress
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1080942 | UITour modifications for Firefox Hello | VERIFIED | FIXED |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
DevEdition Beta Promo
- Feature owner: Bogdan Maris
- Status: On Track
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1078539 | Add a doorhanger notification for beta users using devtools to promote dev-browser | VERIFIED | FIXED |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Send Video to Device
- Feature owner: ???
- Status: Currently in Aurora 36 (Activation late-landed in Aurora)
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1054959 | Add 'Send Video To Device' to the context menu for sending videos from desktop to a second screen | VERIFIED | FIXED |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Iterations
Firefox development now works on two-week iterations. This section tracks the bugs selected for development in these iterations. Please consult the Iteration Dev Walkthrough and the Firefox/IterativeDevelopment page for more information on this process.
- Ensure all bugs have qe-verify status indicated.
- Ensure all qe-verify+ bugs have a QA Contact (The Feature Owner page can help here)
- Ensure all bugs important to be verified are VERIFIED before the end of the next iteration
- See the Firefox QA dashboard (link at the start of this page) for bug queries and status.
Stability
Reports
- Daily explosiveness report: Firefox 35
- 7-day top crash report: 35.0a1
- Consult this walkthrough for more information
Bug Triage
Pushlog
The following tracks triage of the Mozilla pushlogs.
- Review each bug in the pushlog and determine if the bug should be
- A) To be tested by the QA community --> add the qe-verify+ flag to the bug and set the status-firefox34 flag to fixed
- B) To be tested by a new volunteer --> add the [good first verify] whiteboard tag
- C) To be tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
- D) Not to be tested at all --> add the qe-verify-flag
- If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing
- Be sure to set the status-firefox34:fixed flag
- Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures
QA Needed Bugs
The following tracks bugs which have requests for urgent QA assistance.
- Choose a bug to test from the list below
- Read it and make sure you understand what information is needed (this may be steps or URLs to reproduce, a regression range, or something else altogether)
- Use the latest build of this Firefox version and test until you've discovered the information necessary
- Report this information to the bug and remove the qawanted keyword
- Document the bug in the Completed list
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Unconfirmed Bugs
The following tracks triage and testing of incoming, unconfirmed bug reports.
- Choose a bug to test from the list below
- Use the latest Nightly build to confirm the bug reproduces
- Flag with reporter with needinfo if you cannot reproduce and need more information
- Set the status to NEW and ensure the bug is in the appropriate component if you can reproduce the bug (provide as much detail about your testing as possible)
- If it's a Firefox bug you may want to add the firefox-backlog? flag to it.
All Unconfirmed Bugs
66 Total; 66 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Unchanged Since One Week
65 Total; 65 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Unchanged Since One Month
65 Total; 65 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Orphaned Fixes
Recent fixes with no target milestone set
- See "Ongoing work" section of the Firefox QA dashboard for a Bugzilla query.
Recent fixes with status-firefox35 missing.
Verification Needed Bugs
- Review the bugs below in the Pending Triage section and select a bug you want to test
- Make sure you understand the bug before you begin testing it
- Test on a previously known broken build to make sure you can reproduce the bug
- Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
- Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
- Reopen the bug if it still reproduces
- See the Firefox QA dashboard for bug queries and numbers.
Fixed Bugs
The following tracks triage and testing of bugs fixed for this Firefox version.
- Select a bug and review it to determine if it should be
- A) To be tested by the QA community --> add the qe-verify+ flag to the bug and set the status-firefox35 flag to fixed
- B) To be tested by a new volunteer --> add the [good first verify] whiteboard tag
- C) To be tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
- D) Not to be tested at all --> add the qe-verify- flag
- If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing. If it looks like it may be or should be in the testsuite, add the testsuite? flag.
- Be sure to set the status-firefox35:fixed flag
- See the Firefox QA dashboard for bug queries and numbers.
Reported Bugs
The following tracks bugs reported through day-to-day testing.
- Once per week, Softvision will send an email identifying their found issues
- Make sure the issues are documented below
- Review the NEW issues to make sure they are actionable
- Actionable bugs can be flagged "firefox-backlog?" according to the backlog triage criteria
- Any regressions should be nominated to track and be escalated to Release Engineering
- Review the FIXED issues to make sure they are verified
(To be updated 7-25)
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1003142 | A translucent shadow remains on the page/desktop if alert comes up while dragging | RESOLVED | WORKSFORME |
| 1003172 | Small artifact on Ubuntu when closing the search engine list from the new tab page | RESOLVED | WORKSFORME |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
Milestones
Aurora Migration
Post-merge Work (as soon as updates are disabled):
- [DONE] MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the auroratest channel (before first Aurora builds are generated)
Pre-sign-off Checks:
- [DONE] Automation - document all Mozmill test failures (on functional tests) and memory usage regressions (on endurance tests - note that those are only run on nightly, so you look for regressions that happened there). Bugs for those are usually filed by the automation team, but check that they are on top of that.
- [DONE] Features - all scoped features are signed off as ready for Aurora by owners. (If you haven't heard anything to the contrary, that is probably true, but make sure we don't know of blocking issues.)
- [DONE] Bugs - document and review any reported bugs and issues, nominate them for tracking if serious (i.e. make sure all bugs needed to be tracked have at least a request for that up).
- [DONE] Stability - review crash stats before sign-off to raise any red flags.
- [DONE] L10n - contact L10n drivers (usually :Pike) to see if there are any known localization issues.
Post-sign-off Checks (as soon as updates are enabled again):
- [DONE] MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel
- [DONE] Updates - previous versions of Aurora update to the new version (check a few old builds manually once next round of new builds is available)
35-specific steps around Dev Edition landing:
- [DONE] 11/07 - MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the auroratest channel (before first Aurora builds are generated)
- [DONE] 11/10 - MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel
Automation
- Testrun Reports
| Report | Test Failures |
| Add-ons | |
| Functional | |
| Remote | |
| L10n | |
| Updates | |
| Endurance |
- Memory Usage Report
| Platform | Min Memory Usage | Max Memory Usage | Avg Memory Usage | Net Compared to Firefox 32 |
| Windows 32-bit | ||||
| Mac OS X | ||||
| Linux 32-bit | ||||
| Linux 64-bit |
Memory regression being tracked/investigated in: <bug number>
Features
| Feature | Owner | Aurora Sign-off Status | Potential Blockers |
| name | someone | ||
| name |
Updates
| Platform | Source Build | Target Build | Result |
| Windows | |||
| Mac OS X | |||
| Linux | |||
Reported Bugs
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Betas
Sign-off emails
- For Beta 1 and RC, sign off on beta-cdntest, RelMan needs to give the "go" for pushing out
- For Betas in between, directly send "go" for pushing when beta-cdntest is OK.
Known Mozmill Failures
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Beta 1
Build Information
- build1: dcd372476827 (ID: 20141201162954)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Smoke testing covering: Web compatibility, DevEdition Beta Promo, No DevEdition carried from Aurora
- [DONE] Bug verification
- Automated (configs for ondemand)
Bugs Reported
11 Total; 11 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 2
Build Information
- build1: bbf0eddf518c (ID: 20141208150535)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing.
- [DONE] Hello Exploratory testing.
- [ON TRACK] Bug verification
- Automated (configs for ondemand)
Bugs Reported
8 Total; 8 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 3
Build Information
- build1: f7f8ccd5032b (ID: 20141211142524)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing.
- [DONE] Bug verification
- Automated (configs for ondemand)
Bugs Reported
4 Total; 4 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 4
Build Information
- build1: 46b84b0fb7a9 (ID: 20141216011218)
- build2: 216d85d75098 (ID: 20141216120925)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Sanity testing (build1)
- [DONE] Regression testing (build2)
- [DONE] Bug verification
- Automated (configs for ondemand, build2)
- [DONE] Automation Functional tests (report)
- [DONE] Updates on beta-cdntest channel (report)
- [DONE] Updates on beta channel (report)
- expected failures for en-US builds on trains before 35 due to bug 1107621.
Bugs Reported
8 Total; 8 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 5
Build Information
- build1: 5475c6ac8a26 (ID: 20141218174327)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing
- [DONE] Bug verification
- Automated (configs for ondemand)
- [DONE] Automation Functional tests (report)
- [DONE] Updates on beta-cdntest channel (report)
- [DONE] Updates on beta channel (report)
- expected failures for en-US builds on trains before 35 due to bug 1107621.
Bugs Reported
| ID | Summary | Resolution |
|---|---|---|
| 1113580 | On Mac Crash Reporter is out of focus | DUPLICATE |
| 1113633 | WebGL game "Barfight" no longer displays some textures (errors in shader.js) | WONTFIX |
| 1113637 | Blank display for some WebGL games (Could not initialise shader error) | INVALID |
| 1113647 | crash in @0x0 | r600_dri.so@0x147f7b | WORKSFORME |
4 Total; 4 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 6
Build Information
- build1: 687f19c2c8ee (ID: 20141222200458)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing
- [DONE] Bug verification
- Automated (configs for ondemand)
- [DONE] Automation Functional tests (report)
- [DONE] Updates on beta-cdntest channel (report)
- [DONE] Updates on beta channel (report)
- expected failures for en-US builds on trains before 35 due to bug 1107621.
Bugs Reported
5 Total; 5 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 8
Build Information
- build1: 6147cf28a38e (ID: 20141229214612)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Sanity testing
- [DONE] Loop sanity testing for bug 1104279
- [DONE] Bug verification
- Automated (configs for ondemand)
- [DONE] Automation Functional tests (report)
- [DONE] Updates on beta-cdntest channel (report)
- [DONE] Updates on beta channel (report)
- expected failures for en-US builds on trains before 35 due to bug 1107621.
Bugs Reported
3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Release Candidate
Build Information
- build1: 345cd3b9c445 (ID: 20150105205925)
- build2: 12df76fc5ac4 (ID: 20150106233618)
- build3: dfd83036d713 (ID: 20150108202552)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Sanity testing (build1) - General Web Compatibility, WebGL
- [DONE] Exploratory testing (build2) for Hello feature
- [DONE] Verification of bug 1076975 (build2)
- [DONE] Exploratory testing (build3) covering: Hello, Web Compatibility
- [DONE] Verification of bug 1119189 (build3)
- [DONE] Verification of bug 1110760 (build3)
- [DONE] Bug verification
- Automated (configs for ondemand, build2)
- This is the 35.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on releasetest.
- [DONE] Automation Functional tests (report)
- [DONE] Updates on beta-cdntest channel (report)
- [DONE] Updates on beta channel (report)
- expected failures for en-US builds on trains before 35 due to bug 1107621.
Build 3 (done for bug 1119189, bug 1117140, bug 1110760):
- Automated (configs for ondemand)
- This is the 35.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on releasetest.
- [DONE] Automation Functional tests (report)
- [DONE] Updates on beta-cdntest channel (report)
- [DONE] Updates on beta channel (report)
- expected failures for en-US builds on trains before 35 due to bug 1107621.
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on release channel (report)
Bugs Reported
13 Total; 13 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Release 35.0
Build Information
- See Release Candidate Build 3
Testing
- Update Testing - see Release Candidate Build 3
- Spot-testing on showing What's New pages (updates from 34.x should not show it, for older versions should show it), found that it's never shown on Mac (Linux/Windows are fine), reported as bug 1120976
- Throttle Testing (see Walkthrough for details)
- Used commands:
./throttle.sh https://aus3.mozilla.org/update/1/Firefox/34.0.5/20141126041045/WINNT_x86-msvc/en-US/release/update.xml ./throttle.sh https://aus3.mozilla.org/update/1/Firefox/34.0.5/20141126041045/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml ./throttle.sh https://aus3.mozilla.org/update/1/Firefox/34.0.5/20141126041045/Linux_x86-gcc3/en-US/release/update.xml ./throttle.sh https://aus3.mozilla.org/update/1/Firefox/34.0.5/20141126041045/Linux_x86_64-gcc3/en-US/release/update.xml
- Results
- After release push (expected to be @ 25%), 2015-01-13: Win32: 22% - Mac: 25% - Linux32: 24% - Linux64: 24%
- After disabling updates (expected to be @ 0%), 2015-01-15: Win32: 0% - Mac: 0% - Linux32: 0% - Linux64: 0%
- After fully enabling updates (expected to be @ 100%), 2015-01-16: Win32: 100% - Mac: 100% - Linux32: 100% - Linux64: 100%
- Results
35.0.1
Build Information
- build1: 5ea9473a99ff (ID: 20150122214805)
Testing
- Manual
- [DONE] Exploratory testing covering: Web compatibility, Secure browsing, Webmail clients, Crash reporter
- [DONE] Verification of drivers
Bugs Reported
4 Total; 4 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Statistics
Iteration Development
- 35.1: x bugs verified, y bugs qa- (Z bugs total)
Stability
- X / Y topcrash bugs verified fixed as of 2014-0x-xx (Z%)
Reported Bugs
- X / Y reported bugs verified fixed as of 2014-0x-xx (Z%)
