Releases/Firefox 26/Test Plan
« Firefox 25 | Firefox 26 | Firefox 27 »
Contents
Firefox 26 Test Plan
Summary
Milestone | Firefox 26 |
Status | Currently in Nightly -- merging to Aurora on 2013-09-16 |
Lead | Juan Becerra (irc: juanb) |
2nd | Anthony Hughes (irc: ashughes) |
Builds | [ftp] [http] [web] |
The following is the test plan for Firefox 26 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 26. After the release this document will be used as an archive of what was done to validate this release.
If you have some free time, please pick a task below and get in touch with one of the leads.
Schedule
- 2013.08.05: Nightly
- 2013.09.20: Aurora
- 2013.10.25: Pre-Beta feature sign-offs
- 2013.10.31: Beta 1
- 2013.11.05: Beta 2
- 2013.11.07: Beta 3
- 2013.11.12: Beta 4
- 2013.11.14: Beta 5
- 2013.11.19: Beta 6
- 2013.11.21: Beta 7
- 2013.11.26: Beta 8
- 2013.11.28: Beta 9
- 2013.12.03: Beta 10
- 2013.12.09: Release Candidate
- 2013.12.10: Final
Features
- Sign-off Criteria
- All user stories covered by manual or automated tests
- Manual testing complete with no blocking regressions
- Automated testing complete with no blocking regressions
- 0 untriaged unconfirmed bugs
- 0 untriaged unverified fixes
- Status
Feature | Owner | Documentation | Tracked Bugs | Aurora Merge Sign-off | Pre-beta Sign-off |
Firefox Metro | Juan Becerra | Bug Testing | [DONE] | Disabled | |
Updates for limited accounts | Alexandra Lucinet | Test Plan | 1 bug | [DONE] | [DONE] |
H264/AAC/MP3 support - MP3 decoding on XP | Cornel Ionce | Test Plan | 1 bug | [DONE] | [DONE] |
Web App Runtime | Bogdan Maris | Test Plan | 11 bugs | [DONE] | [DONE] |
Download Manager (backend work) | Mihaela Velimiroviciu, Mihai Morar | 16 bugs | [DONE] | [DONE] | |
Session Restore (backend work) | Cornel Ionce | 19 bugs | [DONE] | [DONE] | |
JSON Add-ons DB | Tracy Walker, Mihai Morar | 1 bug | [DONE] | NA | |
Make plugins default to click-to-play | Paul Silaghi | Test Plan | 1 bug | [DONE] | [DONE] |
App Manager | Petruta Rasa | MDN doc | NA | [ON TRACK] |
Nightly
Aurora
Branch Migration
Pre Sign-off Tasks
- [DONE] Firefox branch migrated (Release Management)
- [DONE] Mozmill Tests branch migrated (Automation Development)
- {{|}} Updates throttled on aurora channel (Release Engineering)
- {{|}} Updates verified throttled on aurora channel (QA)
Sign-off Tasks
- [DONE] Mozmill Daily functional testruns pass with no failures (report)
- [DONE] Mozmill Daily auroratest update testruns pass with no failures (report)
- [DONE] Scope features tested and signed off for Aurora uplift
- Metro Firefox will be enabled in Aurora for the first time. Two of the issues blocking it have been resolved and verified. See {{bug|903866} and bug 918107 which deal with software keyboard functionality and using a "whatsnew" page that points to a support article.
Post Sign-off Tasks
- Aurora sign-off email sent (QA)
- Updates unthrottled on aurora channel (Release Engineering)
- Updates verified unthrottled on aurora channel (QA)
WebRTC
Using the Moztrap test as a guideline, verify that Firefox 26.0a2 is no worse than Firefox 22.0 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:
- Have only one browser open on each machine at any given time
- Have only one call running at any given time on a particular machine
- Make sure the caller and callee are always on *different* machines.
- Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
- If you find regressions, report a bug and CC Randell Jesup and Maire Reavy; they can help track down if the regressions are real or not
- Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
Caller | Callee | Result |
Firefox 26.0a2 on Windows 7 | Firefox 26.0a2 on Windows 7 | PASS |
Firefox 22.0 on Windows 7 | Firefox 26.0a2 on Windows XP | FAIL (6*) |
Chrome Stable on Windows 7 | Firefox 26.0a2 on Windows 8 | PASS |
Firefox 26.0a2 on Windows 7 | Firefox 22.0 on Mac OSX 10.8 | FAIL (2*) |
Firefox 26.0a2 on Windows 7 | Chrome Stable on Ubuntu 12.04 | FAIL (4*) |
Firefox 22.0 on Windows XP | Firefox 26.0a2 on Windows 7 | FAIL (6*) |
Firefox 26.0a2 on Windows XP | Firefox 22.0 on Windows XP | FAIL (2*) |
Firefox 26.0a2 on Windows XP | Chrome Stable on Windows 8 | PASS (5*) |
Chrome Stable on Windows XP | Firefox 26.0a2 on Mac OSX 10.8 | PASS |
Firefox 22.0 on Windows XP | Firefox 26.0a2 on Ubuntu 13.04 | FAIL (6*) |
Chrome Stable on Windows 8 | Firefox 26.0a2 on Windows 7 | PASS |
Firefox 26.0a2 on Windows 8 | Chrome Stable on Windows XP | PASS |
Firefox 22.0 on Windows 8 | Firefox 26.0a2 on Windows 8 | PASS (1*) |
Firefox 26.0a2 on Windows 8 | Firefox 22.0 on Windows 8 | PASS (5*) |
Firefox 22.0 on Windows 8 | Firefox 26.0a2 on Mac OSX 10.8 | PASS(5*) |
Chrome Stable on Windows 8 | Firefox 26.0a2 on Ubuntu 13.04 | PASS |
Firefox 26.0a2 on Mac OSX 10.8 | Firefox 22.0 on Windows 7 | PASS (5*) |
Chrome Stable on Mac OSX 10.8 | Firefox 26.0a2 on Windows XP | PASS |
Firefox 22.0 on Mac OSX 10.8 | Firefox 26.0a2 on Windows 8 | PASS |
Firefox 26.0a2 on Mac OSX 10.8 | Chrome Stable on Mac OSX 10.8 | PASS |
Firefox 26.0a2 on Mac OSX 10.8 | Firefox 22.0 on Ubuntu 13.04 | PASS |
Firefox 26.0a2 on Ubuntu 12.04 | Chrome Stable on Windows 7 | PASS |
Firefox 26.0 on Ubuntu 12.04 | Firefox 26.0a2 on Windows XP | PASS (1*) |
Chrome Stable on Ubuntu 12.04 | Firefox 26.0a2 on Windows 8 | FAIL (3*) |
Firefox 26.0a2 on Ubuntu 12.04 | Firefox 22.0 on Mac OSX 10.8 | FAIL (6*) |
Firefox 26.0a2 on Ubuntu 12.04 | Chrome Stable on Ubuntu 13.04 | PASS |
Detailed Results:
- (1*) Callee sometimes hears the caller with a delay (~1sec)
- reproducible also with 25.0a2
- (2*) Image freezes on caller side after ~15 seconds
- not reproducible with Firefox 22
- bug 922051 Image freezes on caller side after a short period of time
- (3*) Caller and callee see only themselves in both video windows (large and small); there is no sound connection between the 2 parts
- reproducible with Firefox 22
- (4*) Caller and callee remain in "Connecting..." state
- reproducible with Firefox 22 and 25.0a2
- (5*) Caller hears delayed sound
- reproducible with Firefox 22 and 25beta1
- (6*) Image freezes on callee side
- not reproducible with Firefox 22 and 25.0a2
- bug 922132 Image freezes on callee side after a while
Beta
- Click to Play by default - check plugin heavy websites for failures and provide feedback on UX
Beta 1
Build Info
- Builds: build1
- Build ID: 20131028225529
- Changes: build1
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Breakpad, Click-to-Play Plugins, Session Restore, MP3 playback
- Platforms: Windows XP, Windows 7 64-bit, Windows Vista, Windows 8.1 32-bit, Mac OSX 10.9, Ubuntu 13.10 64-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- When you get to the WebRTC smoketest refer to the Special Coverage for WebRTC section below for coverage details
- If you encounter a bug, report it and add it to the list below using the bug number template
If you find a site where click-to-play is broken please add it to this document
- Notes
- Websites tested for CTP: https://etherpad.mozilla.org/firefox26b1-plugins
- Tests were not available for run for Windows Vista, so they were run without entering results in Moztrap.
Issues Found
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
- bug 932786 - CTP doorhanger is not showing the right corresponding pop-up
- bug 932813 - Time progress jumps to the end for mp3s via network
- bug 932777 - "Firefox is already running... " error is displayed when trying to open a link .
- bug 932824 - The overlay to Activate Adobe Flash is briefly displayed
- bug 932832 - Click to play overlay not showing for Google Talk plugin.
Beta 2
Build Info
- Builds: build1
- Build ID: 20131104182142
- Changes: build1
- Mozmill Configs
Check List
- [DONE] Automation (report)
- Investigating a couple of cookie test failures in pl and ru. ... It turned out to be bug 856084
- [DONE] Regression Testing (report)
- No new issues reported.
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Session Restore (bug 931891), WebAudio, Downloads.jsm, Updates for limited accounts
- Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.7, Ubuntu 12.04 32-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- If you encounter a bug, report it and add it to the list below using the bug number template
- Notes
Issues Found
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
Beta 3
Build Info
- Builds: build1
- Build ID: 20131107161719
- Changes: build1
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: CTP (bug 934503,bug 932786,bug 932446,bug 745187), WebAudio
(bug 933156,bug 898291,bug 924286), Mixed Content Blocker, WebApp Runtime
- Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu 12.04 64-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- If you encounter a bug, report it and add it to the list below using the bug number template
- Notes
- Websites tested for CTP: https://etherpad.mozilla.org/firefox26b3-plugins
Issues Found
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
Beta 4
Build Info
- Builds: build1
- Build ID: 20131111154639
- Changes: build1
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- Mozmill testing infrastructure is experiencing hiccups, but from the partial results, it looks like updates are working fine.
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Graphics (CSS gradients, SVG), layout, TLS, geolocation, PDF Viewer
- Platforms: Windows XP, Windows 8.1 64bit, Windows 7 32bit, Mac OSX 10.6, Ubuntu 12.04 32-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- If you encounter a bug, report it and add it to the list below using the bug number template
- Notes
- Sites tested for Graphics, Layout (besides those in the Web Compatibility test) and TLS: https://etherpad.mozilla.org/firefox26b4-exploratory
Issues Found
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
- bug 937663 - The pdf embedded using object tag is blocked by the mixed content blocker
Beta 5
Build Info
- Builds: build1
- Build ID: 20131114085019
- Changes: build1
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: History
- Platforms: Windows XP, Windows 7 64bit, Mac OSX 10.7, Ubuntu 12.04 64-bit
- Instructions
- Perform exploratory testing on the History component and add the details of your testing here.
- If you encounter a bug, report it and add it to the list below using the bug number template.
- Notes
Issues Found
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
Beta 6
Build Info
- Builds: build1
- Build ID: 20131118212339
- Changes: build1
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: SVG, DOM:Core and HTML, Core:Layout, SocialAPI, WebGL
- Platforms: Windows XP, Windows 7 64bit, Mac OSX 10.8, Ubuntu 13.04 64-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- Make sure to also do some exploratory testing where permitted (including testing more sites for the web compatibility test)
- If you encounter a bug, report it and add it to the list below using the bug number template
- Notes
Issues Found
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
- bug 940378 - Multiple issues with Social API sidebar
Beta 7
Build Info
- Builds: build1
- Build ID: 20131122094025
- Changes: build1
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing
- Results: A couple of minor issues were found, and an existing SVG performance issue was identified.
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via exploratory testing.
- Coverage
- Tests: CTP (bug 941137, bug 932854), SVG bug 933057
- Platforms: Windows XP, Windows 7 64bit, Mac OSX 10.7, Ubuntu 12.04 32-bit
- Instructions
- Perform exploratory testing on CTP and SVG starting from the bugs above.
- Add the details of your testing here.
- If you encounter a bug, report it and add it to the list below using the bug number template
- Notes
Issues Found
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
- bug 942834 - Loading message is stuck on SVG animation
- bug 942860 - Notification bar for hidden plugins shows for a few seconds and then hides for www.solomid.net page.
Beta 8
Build Info
- Builds: build1
- Build ID: 20131125215016
- Changes: build1
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [ON TRACK] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Installer (bug 941397), Downloads Panel, WebRTC
- Platforms: Windows 8.1 64bit, Windows 7 32bit, Mac OSX 10.9, Ubuntu 13.10 32-bit & Ubuntu 12.04 32-bit (for WebTRC)
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- Make sure to also do some exploratory testing where permitted
- If you encounter a bug, report it and add it to the list below using the bug number template
- Installer coverage
- Windows 7 64bit, 8.1 64bit, Windows XP 32bit, Windows Vista 32bit
- ja, zh-TW, zh-CN, ko, th, he, ru, hi-IN, ar
- WebRTC coverage
Caller | Callee | Notes |
Firefox 26.0b8 Windows 8.1 | Firefox 26.0b8 Windows 7 | |
Firefox 26.0b8 Windows 8.1 | Firefox 26.0b8 Ubuntu 12.04 | |
Firefox 26.0b8 Mac OS X 10.9 | Firefox 26.0b8 Windows 8.1 | |
Firefox 26.0b8 Ubuntu 12.04 | Firefox 26.0b8 Windows 7 | |
Firefox 26.0b8 Windows 7 | Firefox 26.0b8 Mac OS X 10.9 | |
Firefox 26.0b8 Mac OS X 10.9 | Firefox 26.0b8 Ubuntu 12.04 | |
Firefox 26.0b8 Windows 7 | Chrome stable Windows 8.1 | |
Chrome stable Ubuntu 12.04 | Firefox 26.0b8 Windows 8.1 | |
Firefox 26.0b8 Windows 8.1 | Chrome stable Mac OS X 10.9 | |
Chrome stable Windows 7 | Firefox 26.0b8 Ubuntu 12.04 | |
Chrome stable Mac OS X 10.9 | Firefox 26.0b8 Windows 7 | |
Chrome stable Ubuntu 12.04 | Firefox 26.0b8 Mac OS X 10.9 | |
Firefox 26.0b8 Windows 8.1 | Chrome stable Windows 8.1 | |
Chrome stable Windows 7 | Firefox 26.0b8 Windows 7 | |
Firefox 26.0b8 Mac OS X 10.9 | Chrome stable Mac OS X 10.8.5 | |
Chrome stable Ubuntu 12.04 | Firefox 26.0b8 Ubuntu 12.04 |
Issues Found
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
- bug 943634 - stub installer for he needs to be localized
- bug 943824 - hi-IN stub installer is only partially localized
- bug 943832 - hi-IN installer isn't localized
- bug 943836 - Installers and the browser itself display glyphs instead of actual font type for specific languages
- bug 943383 - 'Allow now' becomes 'allow and remember' if looking again in the CTP doorhanger
- bug 943885 - Inaccurate number of downloads is displayed on the download panel.
- bug 943843 - The download size reported by Firefox is much smaller than the actual size on the disk when download size is not known initially.
Beta 9 (skipped)
Beta 10
Build Info
- Builds: build1
- Build ID: 20131202182626
- Changes: build1
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Downloads Panel bug 941099, bug 934155, EV certificates bug 917380, TLS (disabled) bug 733647
- Platforms: Windows XP, Windows 7 64bit, Mac OSX 10.8, Ubuntu 13.10 64-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- Make sure to also do some exploratory testing where permitted
- If you encounter a bug, report it and add it to the list below using the bug number template
- Notes
- The sites tested for the Browsing secure sites and Remove trust for extended validation root certificates tests can be found here.
Issues Found
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
- bug 945703 - Pasting a download link into the download manager sometimes fails.
- bug 945707 - Pausing a download fails when done after a retry
Release Candidate
Build Info
- Builds: build2
- Build ID: 20131205075310
- Changes: build2
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [ON TRACK] Updates on releasetest channel (report)
- [ON TRACK] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Breakpad, Partner repack update, Windows share compatibility, Mac OSX signing, Browsing secure sites
- Platforms: Windows 8.1 64-bit, Windows XP 32-bit, Mac OSX 10.9 (32-bit mode), Ubuntu 13.10 32-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- Make sure to also do some exploratory testing where permitted
- If you encounter a bug, report it and add it to the list below using the bug number template
- Notes
- The list of secure sites tested is available here.
Issues Found
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
- bug 947196 - Cannot play video uploaded in google drive
- bug 947207 - "Trash" button's label overlaps the menu items on Yahoo Mail
Bug Triage
Query | Description | Remaining |
QAWANTED | Bugs requiring immediate investigative assistance | 0 |
VERIFYME | Bugs with fixes requiring immediate verification | |
FIXED | Bugs with fixes which should be verified as soon as possible | |
REGRESSION | Bugs requiring identification of the first affected build | |
UNCONFIRMED | Bugs requiring confirmation if they are reproducible in a current build |
Known Issues
Bug | QA Contact | Status |
799315 | RESOLVED FIXED | |
745187 | NEW | |
870779 | NEW | |
804580 | RESOLVED DUPLICATE | |
922639 | NEW | |
929067 | NEW | |
931790 | NEW | |
832271 | RESOLVED DUPLICATE | |
493124 | NEW | |
872413 | NEW | |
856900 | NEW | |
900463 | NEW | |
928379 | NEW | |
799663 | Cornel Ionce | NEW |
865208 | Cornel Ionce | NEW |