Releases/Firefox 14/Test Plan
« Firefox 13 | Firefox 14 | Firefox 15 »
Contents
- 1 Firefox 14 Test Plan
- 1.1 Summary
- 1.2 Details
- 1.3 Features
- 1.4 Nightly Merge
- 1.5 Endurance Automation
- 1.6 Functional Automation
- 1.7 Unconfirmed Bug Triage
- 1.8 Aurora Merge
- 1.9 Beta 6
- 1.10 Beta 7
- 1.11 Beta 8
- 1.12 Beta 9
- 1.13 Beta 10
- 1.14 Beta 11
- 1.15 Beta 12
- 1.16 Firefox 14.0.1 Release Candidates
Firefox 14 Test Plan
Summary
Milestone | Firefox 14 |
Status | Currently in Beta -- Shipping July 17, 2012. Tentative. |
Lead | Juan Becerra (irc: juanb) |
2nd | Anthony Hughes (irc: ashughes) |
Builds | [ftp] [http] [web] |
The following is the test plan for Firefox 14 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 14. After the release this document will be used as an archive of what was done to validate this release.
If you have some free time, please pick a task below and get in touch with one of the leads.
Details
The following is a description of the tasks which need to be completed throughout this release.
- 2012-03-13: [DONE] Nightly Merge
- 2012-04-24: [DONE] Aurora Merge
- 2012-06-05: [DONE] Pre-Beta feature sign-offs
- 2012-06-07: [DONE] Beta 6 (Starting at version 6 to sync with Firefox Mobile)
- 2012-06-14: [DONE] Beta 7
- 2012-06-21: [DONE] Beta 8
- 2012-06-28: [DONE] Beta 9
- 2012-07-05: [DONE] Beta 10
- 2012-07-11: [DONE] Beta 11
- 2012-07-13: [ON TRACK] Release Candidate
- 2012-07-17: {{|}} Final
Features
See the Feature Release Tracking wiki for a general overview
- [DONE]
- Cycle Collector Perf improvements (Virgil Dicu); landed in Firefox 13
- [ON TRACK]
- Windows8 Metro Firefox (Vlad Ghetiu)
- Hang Detector and Reporter (Simona Badau)
- URL Autocomplete (Vlad Ghetiu)
- Firefox Migrators (Paul Silaghi)
- Pseudo-Class Lock (Ioana Budnar)
- Support for Mouse Capture / Mouse Lock (Virgil Dicu) - Signed off
- Opt-in activation for plugins (Paul Silaghi); Initial minimal implementation landed for 14 (opt-in), further work in 15 and 16.
- [AT RISK]
- [MISSED]
- Incremental garbage collection (Ioana Budnar); delayed to Firefox 15
- Panel-based Download Manager (Simona Badau) opt-in; delayed to Firefox 15
- Silent Update: Background updates (Vlad Ghetiu); delayed to Firefox 15
- Windows Start-up Performance Improvements (Jason Smith); delayed to Firefox 15
- Speedy Session Restore (Paul Silaghi); delayed to Firefox 15
- Silent Update: Updated workflow for users with incompatible add-ons (Ioana Budnar); delayed to Firefox 17
- Show PDF inline (Mihaela Velimiroviciu); delayed to Firefox 16
- Web Apps Integration to Desktop (Jason Smith); delayed to Firefox 15
Nightly Merge
- Functional Automation - result
- Update Spotchecks - result
Endurance Automation
The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.
- What to Do
- Check the Endurance Charts
- If you see an anomalous event (ie. spike or dip), get the report for the specific event
- If the event happens twice in a row, file a bug under Mozmill Tests
- Notify Juan Becerra of the issue
- Record the event in this wiki below
- Current Status (Mihaela Velimiroviciu) - Last Updated
- 2012-06-25
- bug 752887 - Increased memory usage for endurance tests on Linux Aurora branch
- bug 768026 - Higher memory consumption on Win 6.0 on Beta branch since the merge
Functional Automation
The purpose of this section is to track any potential functional regressions via results reported in the dashboard.
- What to Do
- Check the testrun reports
- If you see any failing tests, check that there are not any bugs on file for it already
- If not, file a bug under Mozmill Tests
- Notify Juan Becerra of the issue
- Record the event in this wiki below
- Current Status (Vlad Maniac / Alex Lakatos) - Last Updated
- No issues found
Unconfirmed Bug Triage
The purpose of this section is to triage UNCONFIRMED incoming bugs. The below charts provides a summary of the unconfirmed bugs currently in the Firefox product:
- P1 Task - Get UNCONFIRMED Bugs Out of Firefox General and Untriaged If Possible
- Check this <insert link> for UNCONFIRMED bugs found for this Firefox Release's Beta Timeframe
- Read each bug and determine what product and component they belong under in bugzilla
- Move the bug to the right product and component
- CC any applicable team members that may be interested in the bug
- P2 Task - Confirm UNCONFIRMED Bugs in this 6-Week Cycle
- Check the <insert link> bugs
- Test the bugs in the latest Firefox Nightly
- If you can't reproduce the bug, ask for the reporter to provide more information
- If you reproduce the bug, add your testing information and mark the bug NEW
- Notify Juan Becerra of the NEW issue
- P3 Task - Confirm UNCONFIRMED Bugs from Other Releases
- Check the UNCONFIRMED bugs
- Test the bugs in the latest Firefox Nightly
- If you can't reproduce the bug, ask for the reporter to provide more information
- If you reproduce the bug, add your testing information and mark the bug NEW
- Notify Juan Becerra of the NEW issue
- Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated
Aurora Merge
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | PASS - 2012-04-26 |
Windows Vista | PASS - 2012-04-26 |
Windows 7 32-bit | PASS - 2012-04-26 |
Windows 7 64-bit | PASS - 2012-04-26 |
Mac OS X 10.6 | PASS - 2012-04-26 |
Mac OS X 10.7 | PASS - 2012-04-26 |
Ubuntu 11.10 32-bit | PASS - 2012-04-26 |
Ubuntu 11.10 64-bit | PASS - 2012-04-26 |
- Test Failures
- /testPasswordManager/testPasswordNotificationBar.js (bug 749156)
- /testPasswordManager/testPasswordSavedAndDeleted.js (bug 749206)
- /testSecurity/testBlueLarry.js (bug 748686)
- /testSecurity/testGreenLarry.js (bug 748686)
- /testSecurity/testGreyLarry.js (bug 748686)
Updates
The following are details of update verifications for Firefox 12 merge to Aurora
- Auroratest
- [ON TRACK] [ashughes] Windows XP:
- [ON TRACK] [ashughes] Windows 7:
- [ON TRACK] [ashughes] Mac OS X 10.6:
- [ON TRACK] [ashughes] Ubuntu 11.04:
- Aurora
- [ON TRACK] [ashughes] Windows XP:
- [ON TRACK] [ashughes] Windows 7:
- [ON TRACK] [ashughes] Mac OS X 10.6:
- [ON TRACK] [ashughes] Ubuntu 11.04:
- Issues Found
Manual Testing
Smoketests
Run the Aurora Merge Smoketests in Litmus and report results below.
Platform | Assignee | Result |
Windows XP | Ioana | PASS - 2012-04-27 |
Windows Vista | Simona | PASS - 2012-04-27 |
Windows 7 32-bit | Paul | PASS - 2012-04-27 |
Windows 7 64-bit | Simona | PASS - 2012-04-27 |
Mac OS X 10.6.8 | Paul | PASS - 2012-04-27 |
Mac OS X 10.7.3 | Paul | PASS - 2012-04-27 |
Ubuntu 11.10 32-bit | Virgil | PASS - 2012-04-27 |
Ubuntu 11.10 64-bit | Mihaela | PASS - 2012-04-27 |
Results in Litmus
Feature Sign-offs
Run the Aurora 14.0 sign-offs tests in Litmus and report results below.
Feature | Owner | Win32 | Mac | Linux |
Panel-based Download Manager | Simona Badau | PASS - 2012-04-27 | PASS - 2012-04-27 | PASS - 2012-04-27 |
Windows 8 Metro Firefox | Vlad Ghetiu | N/A | N/A | N/A |
Cycle Collector Perf improvements | Virgil Dicu | PASS - 2012-04-27 | PASS - 2012-04-30 | PASS - 2012-04-30 |
Incremental Garbage Collection | Ioana Budnar | PASS - 2012-04-27 | PASS - 2012-04-27 | PASS - 2012-04-27 |
Hang Detector and Reporter | Simona Badau | N/A | N/A | N/A |
URL Autocomplete | Vlad Ghetiu | PASS - 2012-04-27 | PASS - 2012-04-27 | PASS - 2012-04-27 |
Opt-in activation for plugins | Paul Silaghi | PASS - 2012-04-27 | PASS - 2012-04-30 | PASS - 2012-04-30 |
Show PDF Inline | Mihaela Velimiroviciu | N/A | N/A | N/A |
Firefox migrators | Paul Silaghi | PASS - 2012-04-30 | PASS - 2012-04-30 | N/A |
Silent Update: Background updates | Vlad Ghetiu | N/A | N/A | N/A |
Web Apps integration | Jason Smith | FAIL - 2012-05-01 | FAIL - 2012-05-01 | N/A - 2012-05-01 |
Pseudo-Class Lock | Ioana Budnar | PASS - 2012-04-30 | PASS - 2012-04-30 | PASS - 2012-04-30 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | FAIL - 2012-04-30 | FAIL - 2012-04-30 | FAIL - 2012-04-30 |
Results in Litmus
Notes:
- Windows 8 Metro Firefox - still in the definition phase.
- Hang Detector and Reporter - only available on the Nightly Profiling branch.
- Silent Update: Background updates - didn't land in Aurora yet.
- Show PDF Inline - feature is not available in latest Aurora builds (since Apr 26)
Issues Found
ID | Summary | Priority | Status |
---|---|---|---|
749579 | Visual glitch when playing chrome angrybirds in SD | -- | RESOLVED |
750245 | Mouse lock pointer not working for quake demo | -- | RESOLVED |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
Known Issues
18 Total; 3 Open (16.67%); 9 Resolved (50%); 6 Verified (33.33%);
Pre-beta features sign offs
Run the Firefox 14.0 pre-beta sign off tests in Litmus and report results below.
Feature | Owner | Win32 | Mac | Linux |
URL Autocomplete | Vlad Ghetiu | PASS - 2012-05-31 | PASS - 2012-05-31 | PASS - 2012-05-31 |
Opt-in activation for plugins | Paul Silaghi | PASS - 2012-05-31 | PASS - 2012-05-31 | PASS - 2012-05-31 |
Firefox migrators - Safari Migrator | Paul Silaghi | PASS - 2012-05-31 | PASS - 2012-05-31 | N/A - 2012-05-31 |
Pseudo-Class Lock | Ioana Budnar | FAIL - 2012-05-31 | PASS - 2012-05-31 | PASS - 2012-05-31 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | PASS - 2012-05-31 | PASS - 2012-05-31 | PASS - 2012-05-31 |
Results in Litmus
Issues Found
ID | Summary | Priority | Status |
---|---|---|---|
760171 | The inspector node menu feature allows the user to remove tabbed browsing | P1 | RESOLVED |
760175 | The :focus pseudoClass is not kept for the parent node when deleting a node | -- | RESOLVED |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
Known Issues
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Beta 6
Builds | FTP, HTTP |
ID | 20120605113340 |
Changeset | 89ec8943347a |
- Checklist
- [DONE] Functional automated tests, synced to Litmus
- [DONE] Manual tests
- [ON TRACK] Bug fix verifications (target 50%)
- [ON TRACK] Security fix verifications (target 25%)
- [DONE] Update tests on releasetest
- [DONE] Get final sign-off from Bob Moss
- [DONE] Update tests on beta
Automated Tests
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | PASS - 2012-06-06 |
Windows Vista | PASS - 2012-06-06 |
Windows 7 32-bit | PASS - 2012-06-06 |
Windows 7 64-bit | PASS - 2012-06-06 |
Mac OS X 10.6 | PASS - 2012-06-06 |
Mac OS X 10.7 | PASS - 2012-06-06 |
Ubuntu 11.10 32-bit | PASS - 2012-06-06 |
Ubuntu 11.10 64-bit | PASS - 2012-06-06 |
- Test Failures (known bugs)
Manual Tests
The following are the detailed results of the manual testing.
Smoketests
Run the Firefox 14.0 Smoketests in Litmus and report results below.
- Working on cloning the branch as of 06-04 @11:50pm Pacific
Platform | Assignee | Result |
Windows XP | Simona | PASS - 2012-06-06 |
Windows Vista | tester | RESULT - 2012-06-06 |
Windows 7 32-bit | tester | RESULT - 2012-06-06 |
Windows 7 64-bit | tester | RESULT - 2012-06-06 |
Mac OS X 10.6.8 | tester | RESULT - 2012-06-06 |
Mac OS X 10.7.4 | tester | RESULT - 2012-06-06 |
Ubuntu 12.04 32-bit | tester | RESULT - 2012-06-06 |
Ubuntu 11.10 64-bit | tester | RESULT - 2012-06-06 |
Windows 7 32-bit (es-AR) | Gabriela | PASS - 2012-06-06 |
Results in Litmus
Regression Testing
Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.
Platform | Assignee | Result: report |
Windows 7 32-bit | tester | RESULT - 2012-06-06 |
Mac OS X 10.6.8 | tester | RESULT - 2012-06-06 |
Ubuntu 11.10 64-bit | tester | RESULT - 2012-06-06 |
Windows XP | tester | RESULT - 2012-06-06 |
Windows 7 32-bit (es-AR) | Gabriela | PASS - 2012-06-07 |
Exploratory Testing
- [DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad
- [DONE] Exploratory testing on the new features Etherpad
Feature Testing
Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
Feature | Owner | Win32 | Mac | Linux |
URL Autocomplete | Vlad Ghetiu | PASS - 2012-06-06 | PASS - 2012-06-06 | PASS - 2012-06-06 |
Opt-in activation for plugins | Paul Silaghi | PASS - 2012-06-06 | PASS - 2012-06-06 | PASS - 2012-06-06 |
Firefox migrators - Safari Migrator | Paul Silaghi | PASS - 2012-06-06 | PASS - 2012-06-06 | N/A |
Pseudo-Class Lock | Ioana Budnar | PASS - 2012-06-06 | PASS - 2012-06-06 | PASS - 2012-06-06 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | PASS - 2012-06-06 | PASS - 2012-06-06 | PASS - 2012-06-06 |
Mac 10.8 Signing
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).
- Download the .dmg file and drag the Firefox.app file to a location on your computer
- Open a terminal window and cd to the folder Firefox.app is located
- Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- Click "open" and browse to any website, Firefox should start and you should be able to browse
- Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
Spot check build after this test.
Update Tests
The following are details of update verifications
Platform | Releasetest: report | Beta: report |
Windows XP | PASS - 2012-06-06 | RESULT - 2012-06-06 |
Windows Vista | PASS - 2012-06-06 | RESULT - 2012-06-06 |
Windows 7 32-bit | PASS - 2012-06-06 | RESULT - 2012-06-06 |
Windows 7 64-bit | PASS - 2012-06-06 | RESULT - 2012-06-06 |
Mac OS X 10.6 | PASS - 2012-06-06 | RESULT - 2012-06-06 |
Mac OS 10.7 | PASS - 2012-06-06 | RESULT - 2012-06-06 |
Ubuntu 11.10 32-bit | PASS - 2012-06-06 | RESULT - 2012-06-06 |
Ubuntu 11.10 64-bit | PASS - 2012-06-06 | RESULT - 2012-06-06 |
- Issues
Bug Fix Verifications
The following are the detailed results of verifications of bug fixes landed in Beta.
- Any fix you very should have the status-firefox14 flag set to verified
- If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query | Triaged | Remaining | % Complete |
QA+: for testers to verify functional fixes | 14 | 25 | 36% |
QA+ SG: for the Security team to verify security fixes | 7 | 23 | 23% |
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs | 297 | 0 | 100% |
Softvision verifications etherpad
QAWANTED
- Need monitoring of crash stats for bug 760190
- Need steps to reproduce bug 709860
- bug 755728 only happens on mobile
- bug 748047 needs verification (rest profile)
- bug 748569 needs verification (migration code)
Issues Found
4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);
Known Issues
6 Total; 0 Open (0%); 6 Resolved (100%); 0 Verified (0%);
Beta 6 Feedback
If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.
Builds can be found here: FTP, HTTP
You can post your feedback below or in this etherpad.
Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.
Beta 7
Builds | FTP, HTTP |
ID | TBD |
Changeset | 89ec8943347a |
- Checklist
- [DONE] Functional automated tests, synced to Litmus
- [DONE] Manual tests
- [DONE] Bug fix verifications (target 50%)
- [ON TRACK] Security fix verifications (target 25%)
- [DONE] Update tests on releasetest
- [ON TRACK] Get final sign-off from Bob Moss
- [ON TRACK] Update tests on beta
Automated Tests
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | PASS - 2012-06-12 |
Windows Vista | PASS - 2012-06-12 |
Windows 7 32-bit | PASS - 2012-06-12 |
Windows 7 64-bit | PASS - 2012-06-12 |
Mac OS X 10.6 | PASS - 2012-06-12 |
Mac OS X 10.7 | PASS - 2012-06-12 |
Ubuntu 11.10 32-bit | PASS - 2012-06-12 |
Ubuntu 11.10 64-bit | PASS - 2012-06-12 |
- Test Failures (known bugs)
Manual Tests
The following are the detailed results of the manual testing.
Smoketests
Run the Firefox 14.0 Smoketests in Litmus and report results below.
Platform | Assignee | Result |
Windows XP | tester | RESULT - 2012-06-13 |
Windows Vista | tester | RESULT - 2012-06-13 |
Windows 7 32-bit | Simona | In progress - 2012-06-13 |
Windows 7 64-bit | tester | RESULT - 2012-06-13 |
Mac OS X 10.6.8 | VladG | PASS - 2012-06-13 |
Mac OS X 10.7.4 | tester | RESULT - 2012-06-13 |
Ubuntu 12.04 32-bit | tester | RESULT - 2012-06-13 |
Ubuntu 11.10 64-bit | tester | RESULT - 2012-06-13 |
Windows 7 32-bit (es-AR) | Gabriela | RESULT - 2012-06-13 |
Results in Litmus
Regression Testing
Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.
Platform | Assignee | Result: report |
Windows 7 32-bit | tester | RESULT - 2012-06-12 |
Mac OS X 10.6.8 | tester | RESULT - 2012-06-12 |
Ubuntu 11.10 64-bit | tester | RESULT - 2012-06-12 |
Windows XP | tester | RESULT - 2012-06-12 |
Windows 7 32-bit (es-AR) | Gabriela | PASS - 2012-06-07 |
Exploratory Testing
- [DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad
- [DONE] Exploratory testing on the new features Etherpad
Feature Testing
Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
Feature | Owner | Win32 | Mac | Linux |
URL Autocomplete | Vlad Ghetiu | RESULT - 2012-06-12 | RESULT - 2012-06-12 | RESULT - 2012-06-12 |
Opt-in activation for plugins | Paul Silaghi | RESULT - 2012-06-12 | RESULT - 2012-06-12 | RESULT - 2012-06-12 |
Firefox migrators - Safari Migrator | Paul Silaghi | RESULT - 2012-06-12 | RESULT - 2012-06-12 | N/A |
Pseudo-Class Lock | Ioana Budnar | RESULT - 2012-06-12 | RESULT - 2012-06-12 | RESULT - 2012-06-12 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | RESULT - 2012-06-12 | RESULT - 2012-06-12 | RESULT - 2012-06-12 |
Mac OS 10.8 Signing | Required | n/a | RESULT - 2012-06-12 | n/a |
Mac 10.8 Signing
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).
- Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - VladG
- Open a terminal window and cd to the folder Firefox.app is located
- PASS - VladG
- Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - VladG
- Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - VladG - the output is containing "com.apple.FinderInfo" and "com.apple.quarantine"
- Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - VladG
- Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - VladG
- Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS - VladG
Spot check build after this test.
Update Tests
The following are details of update verifications
Platform | Releasetest: report | Beta: report |
Windows XP | PASS - 2012-06-12 | RESULT - 2012-06-12 |
Windows Vista | PASS - 2012-06-12 | RESULT - 2012-06-12 |
Windows 7 32-bit | PASS - 2012-06-12 | RESULT - 2012-06-12 |
Windows 7 64-bit | PASS - 2012-06-12 | RESULT - 2012-06-12 |
Mac OS X 10.6 | PASS - 2012-06-12 | RESULT - 2012-06-12 |
Mac OS 10.7 | PASS - 2012-06-12 | RESULT - 2012-06-12 |
Ubuntu 11.10 32-bit | PASS - 2012-06-12 | RESULT - 2012-06-12 |
Ubuntu 11.10 64-bit | PASS - 2012-06-12 | RESULT - 2012-06-12 |
- Issues
Bug Fix Verifications
The following are the detailed results of verifications of bug fixes landed in Beta.
- Any fix you very should have the status-firefox14 flag set to verified
- If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query | Triaged | Remaining | % Complete |
QA+: for testers to verify functional fixes | 20 | 19 | 51% |
QA+ SG: for the Security team to verify security fixes | 7 | 23 | 23% |
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs | 297 | 0 | 100% |
Softvision verifications etherpad
QAWANTED
- Need monitoring of crash stats for bug 760190
- bug 746036#c13 needs exploratory testing with Yahoo Toolbar / Trusteer Rapport - investigated and updated (Paul).
- bug 761990 Feedback button regression - investigated and updated (Simona).
- bug 748047 needs verification (reset profile) - verified as fixed (Ioana).
- bug 748569 needs verification (migration code)
- Need steps to reproduce bug 709860 - investigated and commented, no STR yet (Virgil)
- bug 755728 low priority investigation - investigated and updated (Paul).
Issues Found
3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);
Known Issues
12 Total; 0 Open (0%); 12 Resolved (100%); 0 Verified (0%);
Beta 7 Feedback
If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.
Builds can be found here: FTP, HTTP
You can post your feedback below or in this etherpad.
Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.
Beta 8
Builds | FTP, HTTP |
ID | 20120619191901 |
Changeset | Changeset |
- Checklist
- [DONE] Functional automated tests, synced to Litmus
- [DONE] Manual tests
- [ON TRACK] Bug fix verifications (target 50%)
- [ON TRACK] Security fix verifications (target 25%)
- [DONE] Update tests on releasetest
- [ON TRACK] Get final sign-off from Bob Moss
- [ON TRACK] Update tests on beta
Automated Tests
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | PASS - 2012-06-20 |
Windows Vista | PASS - 2012-06-20 |
Windows 7 32-bit | PASS - 2012-06-20 |
Windows 7 64-bit | PASS - 2012-06-20 |
Mac OS X 10.6 | PASS - 2012-06-20 |
Mac OS X 10.7 | PASS - 2012-06-20 |
Ubuntu 11.10 32-bit | PASS - 2012-06-20 |
Ubuntu 11.10 64-bit | PASS - 2012-06-20 |
- Test Failures (known bugs)
Manual Tests
The following are the detailed results of the manual testing.
Smoketests
Run the Firefox 14.0 Smoketests in Litmus and report results below.
Platform | Assignee | Result |
Windows XP | Simona | PASS - 2012-06-19 |
Windows Vista | tester | RESULT - 2012-06-19 |
Windows 7 32-bit | tester | RESULT - 2012-06-19 |
Windows 7 64-bit | tester | RESULT - 2012-06-19 |
Mac OS X 10.6.8 | tester | RESULT - 2012-06-19 |
Mac OS X 10.7.4 | tester | RESULT - 2012-06-19 |
Ubuntu 12.04 32-bit | tester | RESULT - 2012-06-19 |
Ubuntu 11.10 64-bit | tester | RESULT - 2012-06-19 |
Windows 8 32-bit build (en-US) | juanb | PASS - 2012-06-20 |
Results in Litmus
Regression Testing
Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.
Platform | Assignee | Result: report |
Windows 7 32-bit | tester | RESULT - 2012-06-19 |
Mac OS X 10.6.8 | tester | RESULT - 2012-06-19 |
Ubuntu 11.10 64-bit | tester | RESULT - 2012-06-19 |
Windows XP | tester | RESULT - 2012-06-19 |
Exploratory Testing
- [DONE] Exploratory testing on plugins, addons, bookmarks, new features etc Etherpad
Feature Testing
Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
Feature | Owner | Win32 | Mac | Linux |
URL Autocomplete | Vlad Ghetiu | RESULT - 2012-06-19 | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Opt-in activation for plugins | Paul Silaghi | RESULT - 2012-06-19 | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Firefox migrators - Safari Migrator | Paul Silaghi | RESULT - 2012-06-19 | RESULT - 2012-06-19 | N/A |
Pseudo-Class Lock | Ioana Budnar | RESULT - 2012-06-19 | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | RESULT - 2012-06-19 | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Mac OS 10.8 Signing | Required | n/a | RESULT - 2012-06-19 | n/a |
Mac 10.8 Signing
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).
- Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS (Ioana)
- Open a terminal window and cd to the folder Firefox.app is located
- PASS (Ioana)
- Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS (Ioana)
- Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS (Ioana)
- Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS (Ioana)
- Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS (Ioana)
- Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS (Ioana)
Spot check build after this test. Results
Update Tests
The following are details of update verifications
Platform | Releasetest: report | Beta: report |
Windows XP | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Windows Vista | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Windows 7 32-bit | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Windows 7 64-bit | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Mac OS X 10.6 | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Mac OS 10.7 | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Ubuntu 11.10 32-bit | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
Ubuntu 11.10 64-bit | RESULT - 2012-06-19 | RESULT - 2012-06-19 |
- Issues
Bug Fix Verifications
The following are the detailed results of verifications of bug fixes landed in Beta.
- Any fix you very should have the status-firefox14 flag set to verified
- If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query | Triaged | Remaining | % Complete |
QA+: for testers to verify functional fixes | 7 | 32 | 89% |
SG with testcase: for the Security team to verify security fixes | 7 | 23 | 23% |
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs | 297 | 0 | 100% |
Softvision verifications etherpad
QAWANTED
- Need monitoring of crash stats for bug 760190
- bug 746036#c13 needs exploratory testing with Yahoo Toolbar / Trusteer Rapport - investigated and updated (Paul).
- bug 761990 Feedback button regression - investigated and updated (Simona).
- bug 748047 needs verification (reset profile) - verified as fixed (Ioana).
- bug 748569 needs verification (migration code)
- Need steps to reproduce bug 709860 - investigated and commented, no STR yet (Virgil)
- bug 755728 low priority investigation - investigated and updated (Paul).
- bug 764157 regression range wanted - found regression range/updated (Virgil)
- bug 754000 performance regression - investigated and updated (Paul).
Issues Found
ID | Summary | Priority | Status |
---|---|---|---|
766548 | [adbe 3226428] Taking a speed test on YouTube hangs Firefox | -- | RESOLVED |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
Known Issues
10 Total; 0 Open (0%); 10 Resolved (100%); 0 Verified (0%);
Feedback
If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.
Builds can be found here: FTP, HTTP
You can post your feedback below or in this etherpad.
Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.
Beta 9
- NOTE this is a chemspill Beta to mitigate requirement to restart OS on Flash update
Builds | FTP, HTTP |
ID | 20120624012213 |
Changeset | d050090e578c |
- Checklist
- [DONE] Functional automated tests
- [ON TRACK] Manual tests
- [ON TRACK] Bug fix verifications
- [DONE] Update tests on releasetest
- {{ |}} Peer sign-off
- {{ |}} Update tests on beta
- Currently blocked on update testing due to bug 767854 (server issue)
Automated Tests
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | PASS - 2012-06-24 |
Windows Vista | PASS - 2012-06-24 |
Windows 7 32-bit | PASS - 2012-06-24 |
Windows 7 64-bit | PASS - 2012-06-24 |
Mac OS X 10.6 | PASS - 2012-06-24 |
Mac OS X 10.7 | PASS - 2012-06-24 |
Ubuntu 11.10 32-bit | PASS - 2012-06-24 |
Ubuntu 11.10 64-bit | PASS - 2012-06-24 |
- Test Failures (known bugs)
- /restartTests/testDefaultBookmarks/test1.js: bug 767819
- /testPrivateBrowsing/testAboutPrivateBrowsing.js: bug 767820
- /testCookies/testRemoveAllCookies.js: bug 767821
- /testSecurity/testGreyLarry.js: bug 767822
Manual Testing
Run manual smoke tests in Litmus.
- Win 7 x86 - PASS - 2012-06-25 (Mihaela)
Focus Area | Win7-32 | WinXP | Mac | Linux |
Plug-ins | PASS - 2012-06-25 | PASS - 2012-06-25 | PASS - 2012-06-25 | PASS - 2012-06-25 |
Top-sites | PASS - 2012-06-25 | PASS - 2012-06-25 | PASS - 2012-06-25 | PASS - 2012-06-25 |
Mac OSX 10.8 | N/A | N/A | PASS - 2012-06-25 | N/A |
Exploratory testing etherpad: https://mozqa.etherpad.mozilla.org/14b9Exploratory
Plug-ins
- Tests:
- Flash 11.3
- Java 6, 7
- Silverlight
- QuickTime
- Try videos, games, chat, document viewing, or other rich interfaces, closing tabs/windows with loaded plug-in content, updating from older plugin versions (particularly with content already loaded and running), try reloading/new tab/new window with plugin content after update
- Results:
- Windows XP: PASS - 2012-06-25 (Simona)
- Windows 7: PASS - 2012-06-25 (Paul)
- Ubuntu 12.04: PASS - 2012-06-25 (Virgil)
- Mac OSX 10.6.8: PASS - 2012-06-25 (Vladg)
- Details & Bugs:
Top Site Compatibility
- Tests:
- YouTube: video playback, 3D, uploading large files (>100MB), commenting
- Facebook: chat, status, pictures, video, games, timeline, groups
- Twitter: tweet, follow, unfollow, media sharing, trends
- Banking website authentication and transactions
- E-Store authentication, shopping cart, and checkout
- Webmail: Hotmail, Gmail, Yahoo
- keep an eye out for crashes
- Results:
- Windows XP: PASS - 2012-06-25 (Simona)
- Windows 7: PASS - 2012-06-25 (Paul)
- Ubuntu 12.04: PASS - 2012-06-25 (Virgil)
- Mac OSX 10.6.8: PASS - 2012-06-25 (Vladg)
- Details & Bugs:
Mac OS X 10.8 Code-signing
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).
- Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - 2012-06-25 - VladG
- Open a terminal window and cd to the folder Firefox.app is located
- PASS - 2012-06-25 - VladG
- Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - 2012-06-25 - VladG
- Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - 2012-06-25 - VladG
- Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - 2012-06-25 - VladG
- Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - 2012-06-25 - VladG
- Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, Firefox should start up and browse without any issues
- PASS - 2012-06-25 - VladG
Do the following testing to make sure the build works as expected...
- Tests:
- Try all the menu item functions
- Gestures
- Plug-in compatibility
- When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
- Top-site compatibility
- Add-on compatibility
- Software update, especially with rich profiles and fallback updates (use betatest channel)
- Results: PASS - 2012-06-25 - VladG
- Details & Bugs:
Update Tests
The following are details of update verifications
Platform | Releasetest: report | Beta: report |
Windows XP | PASS - 2012-06-24 | PASS - 2012-06-25 |
Windows Vista | PASS - 2012-06-24 | PASS - 2012-06-25 |
Windows 7 32-bit | PASS - 2012-06-24 | PASS - 2012-06-25 |
Windows 7 64-bit | PASS - 2012-06-24 | PASS - 2012-06-25 |
Mac OS X 10.6 | PASS - 2012-06-24 | PASS - 2012-06-25 |
Mac OS 10.7 | PASS - 2012-06-24 | PASS - 2012-06-25 |
Ubuntu 11.10 32-bit | PASS - 2012-06-24 | PASS - 2012-06-25 |
Ubuntu 11.10 64-bit | PASS - 2012-06-24 | PASS - 2012-06-25 |
- Issues
Bug Verifications
Add your name beside whatever you are testing so people don't duplicate your results.
- bug 686335 - Bad plugin search order prevents upgrading Flash without an OS restart
- bug 758363 - Implement NPN_ReloadPlugins for out-of-process Plugins
- NOTE that testing any of these in a VM requires the VM to be run in Protected Mode
My advice would be to take a snapshot of the VM before starting and rolling back before each test they can be run from a clean snapshot.
- NOTE that not all of these plugin versions are available for Mac and Linux.
Just test with whatever old versions you can find and update to the latest. Report the version numbers in your results.
Tests
Flash 10.2 (blocklisted)
- Download and install Flash 10.2
- Blocklist Flash by forcing a ping
- Load some Flash content in tabs before updating
- Pave-over install Flash 11.3.300.262
- Reload tabs with Flash
Flash 10.3 (enabled & disabled)
- Download and install Flash 10.3
- Load some Flash content in tabs before updating
- Pave-over install Flash 11.3.300.262
- Reload tabs with Flash
- Uninstall Flash and retest with Flash disabled before updating
Flash 11.2 (enabled & disabled)
- Download and install Flash 11.2
- Load some Flash content in tabs before updating
- Pave-over install Flash 11.3.300.262
- Reload tabs with Flash
- Uninstall Flash and retest with Flash disabled before updating
Flash 11.3 (enabled & disabled)
- Download and install Flash 11.3.300.257
- Load some Flash content in tabs before updating
- Pave-over Flash 11.3.300.262
- Uninstall Flash and retest with Flash disabled before updating
Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=true)
- Download and install Flash 11.2
- Start Firefox and open a page with flash content
- Download but do not install Flash 11.3 Auto-update Installer
- Open a command line to the download folder and run:
- install_flash_player.exe -install -iv 9
- Reload the page with Flash content once the installer finishes
- Uninstall Flash
- Repeat the test, loading Flash content in a new tab after update
- Repeat the test, loading Flash content in a new window after update
Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=false)
- Download and install Flash 11.2
- Start Firefox and open a page with flash content
- Download but do not install Flash 11.3 Auto-update Installer
- Open a command line to the download folder and run:
- install_flash_player.exe -install -iv 9
- Reload the page with Flash content once the installer finishes
- Uninstall Flash
- Repeat the test, loading Flash content in a new tab after update
- Repeat the test, loading Flash content in a new window after update
Java 6u31 (enabled & disabled)
- Download and install Java 6u31
- Load some Java content in tabs before updating
- Pave-over install Java 7u5
- Reload tabs with Java content after updating
- Uninstall Java and retest with Java disabled before updating
Java 7u2 (enabled & disabled)
- Download and install Java 7u2
- Load some Java content in tabs before updating
- Pave-over install Java 7u5
- Reload tabs with Java content after updating
- Uninstall Java and retest with Java disabled before updating
Results
Expected
- Plugin updates without any problems
- Latest version of the plugin is listed in about:plugins
- Content loads (or reloads) after update without crashing
- NOTE: Multiple versions may be listed in about:plugins
- OKAY as long as plugin content is using the latest version
- Multiple entries can be cleared by restarting the OS
Windows 7 (ashughes)
- PASS Flash 10.2 -> 11.3.300.262 (blocklisted) via pave-over
- PASS Flash 10.3 -> 11.3.300.262 (enabled) via pave-over
- PASS Flash 11.2 -> 11.3.300.262 (enabled) via pave-over
- PASS Flash 11.3 -> 11.3.300.262 (enabled) via pave-over
- PASS Flash 11.3 -> 11.3.300.262 (disabled) via auto-update
- PASS Flash 11.3 -> 11.3.300.262 (dom.ipc.plugins.enabled=false) via auto-update
- PASS Java 6u31 (disabled) -- requires Firefox restart
- PASS Java 7u2 (enabled)
Windows XP (ashughes)
- PASS Update Flash 10.2 -> 11.3.300.262 (disabled) via pave-over
- PASS Update Flash 11.2 -> 11.3.300.262 (enabled) via pave-over
- PASS Update Flash 11.3 -> 11.3.300.262 (enabled) via pave-over
- PASS Update Flash 11.3 -> 11.3.300.262 (dom.ipc.plugins.enabled=false) via auto-update
- PASS Update Java 6u31 -> 7u5 (disabled)
- PASS Update Java 7u2 -> 7u5 (enabled)
Mac OSX 10.6 (ashughes)
- PASS Update Flash 10.2 -> 11.3.300.257 (disabled) via pave-over
- PASS Update Flash 11.2 -> 11.3.300.257 (enabled) via pave-over
- PASS Update Flash 11.2 -> 11.3.300.257 (dom.ipc.plugins.enabled=false) via pave-over
- result Update Java 6u* -> 7u5 (disabled) via Software Update
- result Update Java 7u* -> 7u5 (enabled) via Software Update
- Flash:
- Flash installer forces Firefox to quit before updating
- Disabling before update does not automatically enable after update
Ubuntu 12.04 (ashughes)
- PASS Flash 11.2.205.233 -> 11.2.205.236 (enabled) via pave-over
- PASS Flash 11.2.205.233 -> 11.2.205.236 (disabled) via pave-over
- PASS Flash 11.2.205.233 -> 11.2.205.236 (dom.ipc.plugins.enabled=false) via pave-over
- result Update Oracle Java 6u* -> 7u5 (disabled) via pave-over
- result Update IcedTea Java Plugin (enabled) via Software Update
- Flash Plugin:
- Overwriting when Firefox is running will result in an expected recoverable plugin crash
- Disabling a plugin before update does not automatically re-enable after pave-over update
- Java: As of yet unable to find a reliable way to install/update an old version (Ubuntu only provides the latest)
Known Issues
- Windows: Disabled Java requires restart of Firefox to re-enable after update
- Side-by-Side Flash versions in about:config when paving over Flash 11.3.300.257 64-bit with 11.3.300.262 32-bit; resolved only by OS restart; does not seem to affect functionality/stability
- NOTE: This is expected as long as newly loaded or reloaded Flash content uses the latest version
- Side-by-Side Flash versions in about:config when simulating 11.2->11.3 auto-update with install_flash_player.exe -install -iv 9
- NOTE: This is expected as long as newly loaded or reloaded Flash content uses the latest version
- Hang/crash when watching Youtube after pave-over update (bug 686335#c63) -- not reproducible using auto-update
Issues Found
ID | Summary | Priority | Status |
---|---|---|---|
768002 | crash in npjp2.dll@0x796b with Java SE 7 Update 5 | -- | RESOLVED |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
Known Issues
10 Total; 0 Open (0%); 10 Resolved (100%); 0 Verified (0%);
Beta 10
Builds | FTP, HTTP |
ID | 20120628060610 |
Changeset | e4445f905091 |
- Checklist
- [DONE] Functional automated tests, synced to Litmus
- [DONE] Manual tests
- [ON TRACK] Bug fix verifications (target 50%)
- [ON TRACK] Security fix verifications (target 25%)
- [DONE] Update tests on releasetest
- [ON TRACK] Get final sign-off from Bob Moss
- [ON TRACK] Update tests on beta
Automated Tests
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | PASS - 2012-06-28 |
Windows Vista | PASS - 2012-06-28 |
Windows 7 32-bit | PASS - 2012-06-28 |
Windows 7 64-bit | PASS - 2012-06-28 |
Mac OS X 10.6 | PASS - 2012-06-28 |
Mac OS X 10.7 | PASS - 2012-06-28 |
Ubuntu 11.10 32-bit | PASS - 2012-06-28 |
Ubuntu 11.10 64-bit | PASS - 2012-06-28 |
- Test Failures (known bugs)
Manual Tests
The following are the detailed results of the manual testing.
Smoketests
Run the Firefox 14.0 Smoketests in Litmus and report results below.
Platform | Assignee | Result |
Windows XP | Simona | PASS - 2012-06-28 |
Results in Litmus
Regression Testing
Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below. Results of manual testing (details in exploratory etherpad):
Platform | Assignee | Result: report |
Windows 7 32-bit | Simona | PASS - 2012-06-29 |
Mac OS X 10.6.8 | Vlad | PASS - 2012-06-29 |
Ubuntu 11.10 64-bit | Virgil | PASS - 2012-06-29 |
Windows 7 64-bit | Paul | PASS - 2012-06-29 |
Exploratory Testing
[DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad
Feature Testing
Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
Feature | Owner | Win32 | Mac | Linux |
URL Autocomplete | Vlad Ghetiu | RESULT - 2012-06-28 | RESULT - 2012-06-28 | RESULT - 2012-06-28 |
Opt-in activation for plugins | Paul Silaghi | RESULT - 2012-06-28 | RESULT - 2012-06-28 | RESULT - 2012-06-28 |
Firefox migrators - Safari Migrator | Paul Silaghi | RESULT - 2012-06-28 | RESULT - 2012-06-28 | N/A |
Pseudo-Class Lock | Ioana Budnar | RESULT - 2012-06-28 | RESULT - 2012-06-28 | RESULT - 2012-06-28 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | RESULT - 2012-06-28 | RESULT - 2012-06-28 | RESULT - 2012-06-28 |
Mac OS 10.8 Signing | VladG | n/a | PASS - 2012-06-29 | n/a |
Mac 10.8 Signing
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).
- Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - VladG - 2012-06-29
- Open a terminal window and cd to the folder Firefox.app is located
- PASS - VladG - 2012-06-29
- Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - VladG - 2012-06-29
- Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - VladG - 2012-06-29
- Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - VladG - 2012-06-29
- Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - VladG - 2012-06-29
- Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS - VladG - 2012-06-29
Spot check build after this test.
- Try all the menu item functions
- Gestures
- Plug-in compatibility
- When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
- Top-site compatibility
- Add-on compatibility
- Software update, especially with rich profiles and fallback updates (use betatest channel)
Results: PASS - 2012-06-29 - VladG
Update Tests
The following are details of update verifications
Platform | Releasetest: report | Beta: report |
Windows XP | PASS - 2012-06-28 | RESULT - 2012-06-28 |
Windows Vista | PASS - 2012-06-28 | RESULT - 2012-06-28 |
Windows 7 32-bit | PASS - 2012-06-28 | RESULT - 2012-06-28 |
Windows 7 64-bit | PASS - 2012-06-28 | RESULT - 2012-06-28 |
Mac OS X 10.6 | PASS - 2012-06-28 | RESULT - 2012-06-28 |
Mac OS 10.7 | PASS - 2012-06-28 | RESULT - 2012-06-28 |
Ubuntu 11.10 32-bit | PASS - 2012-06-28 | RESULT - 2012-06-28 |
Ubuntu 11.10 64-bit | PASS - 2012-06-28 | RESULT - 2012-06-28 |
- Issues
Bug Fix Verifications
The following are the detailed results of verifications of bug fixes landed in Beta.
- Any fix you very should have the status-firefox14 flag set to verified
- If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query | Triaged | Remaining | % Complete |
QA+: for testers to verify functional fixes | 31 | 8 | 79% |
QA+ SG: for the Security team to verify security fixes | |||
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs |
Softvision verifications etherpad
QAWANTED
- Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.
Issues Found
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Beta 11
Builds | FTP, HTTP |
ID | 20120704090211 |
Changeset | 48842c93fcb2 |
- Checklist
- [DONE] Functional automated tests, synced to Litmus
- [DONE] Manual tests
- [DONE] Bug fix verifications (target 50%)
- [ON TRACK] Security fix verifications (target 25%)
- [ON TRACK] Update tests on releasetest
- [ON TRACK] Get final sign-off from Bob Moss
- [ON TRACK] Update tests on beta
Automated Tests
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | PASS - 2012-07-04 |
Windows Vista | PASS - 2012-07-04 |
Windows 7 32-bit | PASS - 2012-07-04 |
Windows 7 64-bit | PASS - 2012-07-04 |
Mac OS X 10.6 | PASS - 2012-07-04 |
Mac OS X 10.7 | PASS - 2012-07-04 |
Ubuntu 11.10 32-bit | PASS - 2012-07-04 |
Ubuntu 11.10 64-bit | PASS - 2012-07-04 |
- Test Failures (known bugs)
Manual Tests
The following are the detailed results of the manual testing.
Smoketests
Run the Firefox 14.0 Smoketests in Litmus and report results below.
Platform | Assignee | Result |
Windows XP | Ioana | PASS - 2012-07-05 |
Results in Litmus
Exploratory Testing
- [DONE] Exploratory testing on plugins, new features etc Etherpad
Feature Testing
Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
Feature | Owner | Win32 | Mac | Linux |
URL Autocomplete | Vlad Ghetiu | RESULT - 2012-07-04 | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Opt-in activation for plugins | Paul Silaghi | RESULT - 2012-07-04 | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Firefox migrators - Safari Migrator | Paul Silaghi | RESULT - 2012-07-04 | RESULT - 2012-07-04 | N/A |
Pseudo-Class Lock | Ioana Budnar | RESULT - 2012-07-04 | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | RESULT - 2012-07-04 | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Mac OS 10.8 Signing | Required | n/a | RESULT - 2012-07-04 | n/a |
Mac 10.8 Signing
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).
- Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - VladG - 2012-07-05
- Open a terminal window and cd to the folder Firefox.app is located
- PASS - VladG - 2012-07-05
- Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - VladG - 2012-07-05
- Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - VladG - 2012-07-05
- Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - VladG - 2012-07-05
- Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - VladG - 2012-07-05
- Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS - VladG - 2012-04-05
Spot check build after this test: PASS - VladG - 2012-04-05
- Try all the menu item functions
- Gestures
- Plug-in compatibility
- Top-site compatibility
- Add-on compatibility
Update Tests
The following are details of update verifications
Platform | Releasetest: report | Beta: report |
Windows XP | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Windows Vista | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Windows 7 32-bit | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Windows 7 64-bit | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Mac OS X 10.6 | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Mac OS 10.7 | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Ubuntu 11.10 32-bit | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
Ubuntu 11.10 64-bit | RESULT - 2012-07-04 | RESULT - 2012-07-04 |
- Issues
Bug Fix Verifications
The following are the detailed results of verifications of bug fixes landed in Beta.
- Any fix you very should have the status-firefox14 flag set to verified
- If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query | Triaged | Remaining | % Complete |
QA+: for testers to verify functional fixes | 31 | 8 | 79% |
QA+ SG: for the Security team to verify security fixes | |||
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs |
Softvision verifications etherpad
QAWANTED
- Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.
Issues Found
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Known Issues
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Beta 12
Builds | FTP, HTTP |
ID | 20120710123126 |
Changeset | TBD |
- Checklist
- [DONE] Functional automated tests, synced to Litmus
- [DONE] Manual tests
- [DONE] Bug fix verifications (target 50%)
- [ON TRACK] Security fix verifications (target 25%)
- [DONE] Update tests on releasetest
- [ON TRACK] Get final sign-off from Bob Moss
- [ON TRACK] Update tests on beta
Automated Tests
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | PASS - 2012-07-10 |
Windows Vista | PASS - 2012-07-10 |
Windows 7 32-bit | PASS - 2012-07-10 |
Windows 7 64-bit | PASS - 2012-07-10 |
Mac OS X 10.6 | PASS - 2012-07-10 |
Mac OS X 10.7 | PASS - 2012-07-10 |
Ubuntu 11.10 32-bit | PASS - 2012-07-10 |
Ubuntu 11.10 64-bit | PASS - 2012-07-10 |
- Test Failures (known bugs)
Manual Tests
The following are the detailed results of the manual testing.
Smoketests
Run the Firefox 14.0 Smoketests in Litmus and report results below.
Platform | Assignee | Result |
Windows 7 32-bit | Simona | PASS - 2012-07-11 |
Results in Litmus
Exploratory Testing
- [DONE] Exploratory testing on plugins, new features etc Etherpad
Feature Testing
Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
Feature | Owner | Win32 | Mac | Linux |
URL Autocomplete | Vlad Ghetiu | RESULT - 2012-07-10 | RESULT - 2012-07-10 | RESULT - 2012-07-10 |
Opt-in activation for plugins | Paul Silaghi | RESULT - 2012-07-10 | RESULT - 2012-07-10 | RESULT - 2012-07-10 |
Firefox migrators - Safari Migrator | Paul Silaghi | RESULT - 2012-07-10 | RESULT - 2012-07-10 | N/A |
Pseudo-Class Lock | Ioana Budnar | RESULT - 2012-07-10 | RESULT - 2012-07-10 | RESULT - 2012-07-10 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | RESULT - 2012-07-10 | RESULT - 2012-07-10 | RESULT - 2012-07-10 |
Mac OS 10.8 Signing | Required | n/a | PASS - 2012-07-11 | n/a |
Mac 10.8 Signing
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).
- Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - VladG - 2012-07-11
- Open a terminal window and cd to the folder Firefox.app is located
- PASS - VladG - 2012-07-11
- Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - VladG - 2012-07-11
- Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - VladG - 2012-07-11
- Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - VladG - 2012-07-11
- Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - VladG - 2012-07-11
- Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS - VladG - 2012-07-11
Spot check build after this test - PASS - VladG - 2012-07-11
- Try all the menu item functions
- Gestures
- Plug-in compatibility
- When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
- Top-site compatibility
- Add-on compatibility
- Software update
Update Tests
The following are details of update verifications
Platform | Releasetest: report | Beta: report |
Windows XP | PASS - 2012-07-11 | PASS - 2012-07-11 |
Windows Vista | PASS - 2012-07-11 | PASS - 2012-07-11 |
Windows 7 32-bit | PASS - 2012-07-11 | PASS - 2012-07-11 |
Windows 7 64-bit | PASS - 2012-07-11 | PASS - 2012-07-11 |
Mac OS X 10.6 | PASS - 2012-07-11 | PASS - 2012-07-11 |
Mac OS 10.7 | PASS - 2012-07-11 | PASS - 2012-07-11 |
Ubuntu 11.10 32-bit | PASS - 2012-07-11 | PASS - 2012-07-11 |
Ubuntu 11.10 64-bit | PASS - 2012-07-11 | PASS - 2012-07-11 |
- Issues
Bug Fix Verifications
The following are the detailed results of verifications of bug fixes landed in Beta.
- Any fix you very should have the status-firefox14 flag set to verified
- If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query | Triaged | Remaining | % Complete |
QA+: for testers to verify functional fixes | 36 | 3 | 92% |
QA+ SG: for the Security team to verify security fixes | |||
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs |
Softvision verifications etherpad
QAWANTED
- Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.
Issues Found
ID | Summary | Priority | Status |
---|---|---|---|
772786 | Wrong favicons displayed in the Recently Bookmarked folder | -- | RESOLVED |
772842 | The newly opened tab is not always focused when taking a speed test from a youtube fullscreen video | -- | RESOLVED |
773236 | On Windows Vista Flash content is white after exiting Full Screen | -- | RESOLVED |
3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);
Known Issues
ID | Summary | Priority | Status |
---|---|---|---|
675645 | Firefox freezes when using Pop out while in Fullscreen mode on Youtube | P3 | RESOLVED |
711545 | Crash reports from last evening still not found | -- | VERIFIED |
763237 | [adbe 3221904] Firefox crash in F1398665248_____________________________ | -- | RESOLVED |
770207 | On Windows the "Feedback" button is not highlighted when hovering the mouse over it | -- | RESOLVED |
4 Total; 0 Open (0%); 3 Resolved (75%); 1 Verified (25%);
Firefox 14.0.1 Release Candidates
Builds | FTP, HTTP |
ID | 20120713134347 |
Changeset | e5728a4e106c |
- Checklist
- [DONE] Functional automated tests
- [DONE] Manual tests
- [DONE] Bug fix verifications
- [ON TRACK] Security fix verifications
- [DONE] Update tests on releasetest
- [ON TRACK] Get final sign-off
- [ON TRACK] Update tests on release
Automated Tests
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | PASS - 2012-07-15 |
Windows Vista | PASS - 2012-07-15 |
Windows 7 32-bit | PASS - 2012-07-15 |
Windows 7 64-bit | PASS - 2012-07-15 |
Mac OS X 10.6 | PASS - 2012-07-15 |
Mac OS X 10.7 | PASS - 2012-07-15 |
Ubuntu 11.10 32-bit | PASS - 2012-07-15 |
Ubuntu 11.10 64-bit | PASS - 2012-07-15 |
- Test Failures (known bugs)
Manual Tests
The following are the detailed results of the manual testing.
Smoketests
Run the Firefox 14.0 Smoketests in Litmus and report results below.
Platform | Assignee | Result |
Windows 7 32-bit | Mihaela | PASS - 2012-07-16 |
Mac OS X 10.7.4 | Ioana | PASS - 2012-07-16 |
Ubuntu 12.04 32-bit | Virgil | PASS - 2012-07-16 |
Results in Litmus
Exploratory Testing
- [DONE] Exploratory testing on plugins, new features, web compatibility etc Etherpad
Feature Testing
By now we should have all features signed off.
Mac 10.8 Signing
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).
- Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - VladG - 2012-07-16
- Open a terminal window and cd to the folder Firefox.app is located
- PASS - VladG - 2012-07-16
- Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - VladG - 2012-07-16
- Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - VladG - 2012-07-16
- Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - VladG - 2012-07-16
- Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - VladG - 2012-07-16
- Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS - VladG - 2012-07-16
Spot check build after this test.
Try all the menu item functions - PASS - VladG - 2012-07-16
- Gestures
- Plug-in compatibility
- Top-site compatibility
- Add-on compatibility
- Software update
Update Tests
The following are details of update verifications
Platform | Releasetest: report | Beta: report |
Windows XP | PASS - 2012-07-15 | RESULT - 2012-07-15 |
Windows Vista | PASS - 2012-07-15 | RESULT - 2012-07-15 |
Windows 7 32-bit | PASS - 2012-07-15 | RESULT - 2012-07-15 |
Windows 7 64-bit | PASS - 2012-07-15 | RESULT - 2012-07-15 |
Mac OS X 10.6 | PASS - 2012-07-15 | RESULT - 2012-07-15 |
Mac OS 10.7 | PASS - 2012-07-15 | RESULT - 2012-07-15 |
Ubuntu 11.10 32-bit | PASS - 2012-07-15 | RESULT - 2012-07-15 |
Ubuntu 11.10 64-bit | PASS - 2012-07-15 | RESULT - 2012-07-15 |
- Issues
Bug Fix Verifications
The following are the detailed results of verifications of bug fixes landed in Beta.
- Any fix you very should have the status-firefox14 flag set to verified
- If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query | Triaged | Remaining | % Complete |
QA+: for testers to verify functional fixes | 36 | 3 | 92% |
QA+ SG: for the Security team to verify security fixes | |||
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs |
Softvision verifications etherpad
QAWANTED
- Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.
Issues Found
ID | Summary | Priority | Status |
---|---|---|---|
774212 | Improve live bookmark loading indications | P5 | RESOLVED |
774244 | When HTML5 videos end - the suggestions are no longer displayed | -- | RESOLVED |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
Known Issues
ID | Summary | Priority | Status |
---|---|---|---|
667399 | RealPlayer G2 LiveConnect plugin crash | -- | RESOLVED |
692446 | Firefox prompts to install windows media player plugin even though it is already installed | -- | RESOLVED |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);