Releases/Firefox 18/Test Plan

From MozillaWiki
Jump to: navigation, search

« Firefox 17 | Firefox 18 | Firefox 19 »

Firefox 18 Test Plan

Summary

Milestone Firefox 18
Status Currently in Aurora -- merging to Beta on November 19, 2012
Lead Juan Becerra (irc: juanb)
2nd Anthony Hughes (irc: ashughes)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 18 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 18. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

  • 2012-08-27: [DONE] Nightly Merge
  • 2012-10-08: [DONE] Aurora Merge
  • 2012-11-16: [DONE] Pre-Beta feature sign-offs
  • 2012-11-22: {{|}} Beta 1
  • 2012-11-29: {{|}} Beta 2
  • 2012-12-06: {{|}} Beta 3
  • 2012-12-11: {{|}} Beta 4
  • 2012-12-18: {{|}} Beta 5
  • TBA {{|}} Beta 6
  • TBA {{|}} Release Candidate
  • TBA {{|}} Final

Nightly

Merge

Functional Automation - report - PASS (2012-08-28)
Update Spotchecks - report - PASS (2012-08-28)

Features

See the Feature Release Tracking wiki for a general overview

Features

See the Feature Release Tracking wiki for a general overview

Feature Owner Status Tests
Panel-based Download Manager Simona Badau On Track
Show PDF inline Mihaela Velimiroviciu On Track
Modern JIT Compiler for JavaScript (IonMonkey) No QA needed here.

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Mihaela Velimiroviciu) - Last Updated

Bugzilla query error


Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Vlad Maniac / Alex Lakatos) - Last Updated


Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs.

What to Do
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Anthony Hughes of the NEW issue
Current Status (SoftVision) - Last Updated

Result Spreadsheet

Aurora Merge Sign-off

Automated Tests

Results of the Mozmill automation testruns.

Functional Tests
Platform Result: report Failures: known bugs
Windows XP result
Windows Vista result
Windows 7 32-bit result
Windows 7 64-bit result
Mac OS X 10.6 result
Mac OS X 10.7 result
Ubuntu 11.10 32-bit result
Ubuntu 11.10 -bit result
Bugs
Endurance Tests
Platform Result: report Max Memory
Windows XP result
Windows Vista result
Windows 7 32-bit result
Windows 7 64-bit result
Mac OS X 10.6 result
Mac OS X 10.7 result
Ubuntu 11.10 32-bit result
Ubuntu 11.10 -bit result
Bugs

Feature Sign-offs

  • Log in to moztrap.mozilla.org
  • Select Run Tests
  • Select Product: Desktop Firefox
  • Select Versions: 18
  • Select Runs: Firefox 18 Aurora Merge Feature sign off
Feature Owner Win32 Mac Linux
Panel-based Download Manager Simona Badau disabled-2012-10-12 disabled-2012-10-12 disabled-2012-10-12
Show PDF inline Mihaela Velimiroviciu PASS-2012-10-12 PASS-2012-10-12 PASS-2012-10-12
Modern JIT Compiler for JavaScript (IonMonkey) No need for QA owner NA NA NA
Details

Pre-Beta Feature Sign-offs

  • Log in to moztrap.mozilla.org
  • Select Run Tests
  • Select Product: Desktop Firefox
  • Select Versions: 18
  • Select Runs: Firefox 18 Aurora Merge Feature sign off
Feature Owner Win32 Mac Linux
Panel-based Download Manager Simona Badau disabled disabled disabled
Show PDF inline Mihaela Velimiroviciu PASS PASS PASS
Per-Site Third-Party Cookie Setting Ioana Budnar PASS PASS PASS
Modern JIT Compiler for JavaScript (IonMonkey) No need for QA owner NA NA NA
Details

Beta 1

Builds Build 1: FTP, HTTP
ID 20121121075611
Changeset 1f1083953e62 (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Build #1 Results (report)2012-11-22
Windows XP PASS - 2012-11-22
Windows Vista PASS - 2012-11-22
Windows 7 32-bit PASS - 2012-11-22
Windows 7 64-bit PASS - 2012-11-22
Windows 8 32-bit PASS - 2012-11-22
Windows 8 64-bit PASS - 2012-11-22
Mac OS X 10.6 PASS - 2012-11-22
Mac OS X 10.7 PASS - 2012-11-22
Mac OS X 10.8 PASS - 2012-11-22
Ubuntu 12.04 32-bit PASS - 2012-11-22
Ubuntu 12.04 64-bit PASS - 2012-11-22
Test Failures (known bugs)

Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows 7 Virgil PASS - 2012-11-23
Windows 8 Ioana PASS - 2012-11-23
Mac OSX 10.7 Mihaela PASS - 2012-11-23
Mac OSX 10.8 Simona PASS - 2012-11-23
Ubuntu 12.04 Paul PASS - 2012-11-23

Update Testing

  • Partial: 17.0b6
  • Complete: <= 17.0b5
Platform Releasetest (report) Beta (report)
Windows XP PASS - 2012-11-22 PASS - 2012-11-22
Windows Vista PASS - 2012-11-22 PASS - 2012-11-22
Windows 7 32-bit PASS - 2012-11-22 PASS - 2012-11-22
Windows 7 64-bit PASS - 2012-11-22 PASS - 2012-11-22
Windows 8 32-bit PASS - 2012-11-22 PASS - 2012-11-22
Windows 8 64-bit PASS - 2012-11-22 PASS - 2012-11-22
Mac OS X 10.6 PASS - 2012-11-22 PASS - 2012-11-22
Mac OS X 10.7 PASS - 2012-11-22 PASS - 2012-11-22
Mac OS X 10.8 PASS - 2012-11-22 PASS - 2012-11-22
Ubuntu 12.04 32-bit PASS - 2012-11-22 PASS - 2012-11-22
Ubuntu 12.04 64-bit PASS - 2012-11-22 PASS - 2012-11-22
Issues

Feature Sign-off

Feature owners to test and sign-off overall quality of their features following uplift to Beta.

Feature Owner Win32 Mac Linux
Show PDF inline Mihaela Velimiroviciu PASS - 2012-11-23 PASS - 2012-11-23 PASS - 2012-11-23
Per-Site Third-Party Cookie Setting Ioana Budnar PASS - 2012-11-22 PASS - 2012-11-23 PASS - 2012-11-22

Beta 2

Builds Build 1: FTP, HTTP
ID TBD
Changeset e8b696e40f36 (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Build #1 Results (report)2012-11-28
Windows XP PASS - 2012-11-28
Windows Vista PASS - 2012-11-28
Windows 7 32-bit PASS - 2012-11-28
Windows 7 64-bit PASS - 2012-11-28
Windows 8 32-bit PASS - 2012-11-28
Windows 8 64-bit PASS - 2012-11-28
Mac OS X 10.6 PASS - 2012-11-28
Mac OS X 10.7 PASS - 2012-11-28
Mac OS X 10.8 PASS - 2012-11-28
Ubuntu 12.04 32-bit PASS - 2012-11-28
Ubuntu 12.04 64-bit PASS - 2012-11-28
Test Failures (known bugs)

Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows 7 Mihaela PASS - 2012-11-29
Windows 8 Ioana PASS - 2012-11-29
Mac OSX 10.7 Manuela PASS - 2012-11-29
Mac OSX 10.8 Virgil PASS - 2012-11-29
Ubuntu 12.04 Simona PASS - 2012-11-29

Update Testing

  • Partial: 17.0b6, 17.0b5
  • Complete: <= 16.0b4
Platform Releasetest (report) Beta (report)
Windows XP PASS - 2012-11-28 RESULT - 2012-11-28
Windows Vista PASS - 2012-11-28 RESULT - 2012-11-28
Windows 7 32-bit PASS - 2012-11-28 RESULT - 2012-11-28
Windows 7 64-bit PASS - 2012-11-28 RESULT - 2012-11-28
Windows 8 32-bit PASS - 2012-11-28 RESULT - 2012-11-28
Windows 8 64-bit PASS - 2012-11-28 RESULT - 2012-11-28
Mac OS X 10.6 PASS - 2012-11-28 RESULT - 2012-11-28
Mac OS X 10.7 PASS - 2012-11-28 RESULT - 2012-11-28
Mac OS X 10.8 PASS - 2012-11-28 RESULT - 2012-11-28
Ubuntu 12.04 32-bit PASS - 2012-11-28 RESULT - 2012-11-28
Ubuntu 12.04 64-bit PASS - 2012-11-28 RESULT - 2012-11-28
Issues

Beta 3

Builds Build 1: FTP, HTTP
ID 20121205060959
Changeset 68cb786721c7 (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Build #1 Results (report)2012-12-05
Windows XP PASS - 2012-12-05
Windows Vista PASS - 2012-12-05
Windows 7 32-bit PASS - 2012-12-05
Windows 7 64-bit PASS - 2012-12-05
Windows 8 32-bit PASS - 2012-12-05
Windows 8 64-bit PASS - 2012-12-05
Mac OS X 10.6 PASS - 2012-12-05
Mac OS X 10.7 PASS - 2012-12-05
Mac OS X 10.8 PASS - 2012-12-05
Ubuntu 12.04 32-bit PASS - 2012-12-05
Ubuntu 12.04 64-bit PASS - 2012-12-05
Test Failures (known bugs)

Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows 7 Manuela PASS - 2012-12-06
Windows 8 Virgil PASS - 2012-12-06
Mac OSX 10.7 Simona PASS - 2012-12-06
Mac OSX 10.8 Paul PASS - 2012-12-06
Ubuntu 12.04 Ioana PASS - 2012-12-06

Exploratory testing around beta 3 changes- etherpad

Update Testing

  • Partial: 18.0b2, 18.0b1
  • Complete: <= 17.0b6
Platform Releasetest (report) Beta (report)
Windows XP PASS - 2012-12-05 RESULT - 2012-12-05
Windows Vista PASS - 2012-12-05 RESULT - 2012-12-05
Windows 7 32-bit PASS - 2012-12-05 RESULT - 2012-12-05
Windows 7 64-bit PASS - 2012-12-05 RESULT - 2012-12-05
Windows 8 32-bit PASS - 2012-12-05 RESULT - 2012-12-05
Windows 8 64-bit PASS - 2012-12-05 RESULT - 2012-12-05
Mac OS X 10.6 PASS - 2012-12-05 RESULT - 2012-12-05
Mac OS X 10.7 PASS - 2012-12-05 RESULT - 2012-12-05
Mac OS X 10.8 PASS - 2012-12-05 RESULT - 2012-12-05
Ubuntu 12.04 32-bit PASS - 2012-12-05 RESULT - 2012-12-05
Ubuntu 12.04 64-bit PASS - 2012-12-05 RESULT - 2012-12-05
Issues

Beta 4

Builds Build 1: FTP, HTTP
ID TBD
Changeset e904ee016207 (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Build #1 Results (report)2012-12-12
Windows XP 2 Fails - 2012-12-12
Windows Vista PASS - 2012-12-12
Windows 7 32-bit PASS - 2012-12-12
Windows 7 64-bit PASS - 2012-12-12
Windows 8 32-bit PASS - 2012-12-12
Windows 8 64-bit PASS - 2012-12-12
Mac OS X 10.6 PASS - 2012-12-12
Mac OS X 10.7 PASS - 2012-12-12
Mac OS X 10.8 PASS - 2012-12-12
Ubuntu 12.04 32-bit PASS - 2012-12-12
Ubuntu 12.04 64-bit PASS - 2012-12-12
Test Failures (known bugs)

Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows 7 Manuela PASS - 2012-12-13
Windows 8 Simona PASS - 2012-12-13
Mac OSX 10.7 Paul PASS - 2012-12-13
Mac OSX 10.8 Virgil PASS - 2012-12-13
Ubuntu 12.04 Virgil PASS - 2012-12-13

Update Testing

  • Partial: 18.0b3, 18.0b2
  • Complete: <= 17.0b6
Platform Releasetest (report) Beta (report)
Windows XP PASS - 2012-12-13 PASS - 2012-12-13
Windows Vista PASS - 2012-12-13 PASS - 2012-12-13
Windows 7 32-bit PASS - 2012-12-13 PASS - 2012-12-13
Windows 7 64-bit PASS - 2012-12-13 PASS - 2012-12-13
Windows 8 32-bit PASS - 2012-12-13 PASS - 2012-12-13
Windows 8 64-bit PASS - 2012-12-13 PASS - 2012-12-13
Mac OS X 10.6 PASS - 2012-12-13 PASS - 2012-12-13
Mac OS X 10.7 PASS - 2012-12-13 PASS - 2012-12-13
Mac OS X 10.8 PASS - 2012-12-13 PASS - 2012-12-13
Ubuntu 12.04 32-bit PASS - 2012-12-13 PASS - 2012-12-13
Ubuntu 12.04 64-bit PASS - 2012-12-13 PASS - 2012-12-13
Issues

Beta 5

Builds Build 1: FTP, HTTP
ID 20121219074241
Changeset 05ecd95e85da (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Build #1 Results (report) 2012-12-19
Windows XP PASS - 2012-12-19
Windows Vista PASS - 2012-12-19
Windows 7 32-bit PASS - 2012-12-19
Windows 7 64-bit PASS - 2012-12-19
Windows 8 32-bit PASS - 2012-12-19
Windows 8 64-bit PASS - 2012-12-19
Mac OS X 10.6 PASS - 2012-12-19
Mac OS X 10.7 PASS - 2012-12-19
Mac OS X 10.8 PASS - 2012-12-19
Ubuntu 12.04 32-bit PASS - 2012-12-19
Ubuntu 12.04 64-bit PASS - 2012-12-18
Test Failures (known bugs)

Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows 7 Mihaela PASS - 2012-12-20
Windows 8 Ioana PASS - 2012-12-20
Mac OSX 10.7 Paul PASS - 2012-12-20
Mac OSX 10.8 Simona PASS - 2012-12-20
Ubuntu 12.04 Manuela PASS - 2012-12-20
Mac OSX 10.6 (32bit mode) Manuela PASS - 2012-12-20

Update Testing

  • Partial: 18.0b4, 18.0b3
  • Complete: <= 17.0b6
Platform Releasetest (report) Beta (report)
Windows XP PASS - 2012-12-19 RESULT - 2012-12-18
Windows Vista PASS - 2012-12-19 RESULT - 2012-12-18
Windows 7 32-bit PASS - 2012-12-19 RESULT - 2012-12-18
Windows 7 64-bit PASS - 2012-12-19 RESULT - 2012-12-18
Windows 8 32-bit PASS - 2012-12-19 RESULT - 2012-12-18
Windows 8 64-bit PASS - 2012-12-19 RESULT - 2012-12-18
Mac OS X 10.6 PASS - 2012-12-19 RESULT - 2012-12-18
Mac OS X 10.7 PASS - 2012-12-19 RESULT - 2012-12-18
Mac OS X 10.8 PASS - 2012-12-19 RESULT - 2012-12-18
Ubuntu 12.04 32-bit PASS - 2012-12-19 RESULT - 2012-12-18
Ubuntu 12.04 64-bit PASS - 2012-12-19 RESULT - 2012-12-18

SVG Testing

There were a couple of bug fixes in this cycle which require a little bit of testing around. Of the two, bug 820061 dealing with SVG animations would be more worrisome. Please verify both bugs and test SVG animation with a few samples from the latest test suite cited in:

http://www.w3.org/Graphics/SVG/WG/wiki/Test_Suite_Overview

  • (Virgil)Compared test cases from 18b5(with SVG related patch) with previous builds (18b3-without patch):

-http://www.w3.org/Graphics/SVG/Test/20061213/htmlEmbedHarness/basic-index.html - """PASS""" - Ubuntu 12.04, Windows 7, Mac OS 10.8 -http://www.w3.org/Graphics/SVG/Test/20080912/htmlEmbedHarness/index.html - """PASS""" - Ubuntu 12.04, Windows 7, Mac OS 10.8

It will be sufficient to verify bug 813531 with the existing test case.

Issues

Beta 6

Builds Build 1: FTP, HTTP
ID 20121227074833
Changeset e904ee016207 (pushlog)
Checklist

Automated Testing

Due to bug 807099 the Mozmill CI will not run for this cycle. However, we will try to run test runs, triggered manually, for Windows 7 64bit, Mac 10.7.x, and Linux 64bit.

Platform Build #1 Results (report) 2012-12-28
Windows 7 64-bit RESULT - 2012-12-28
Mac OS X 10.7 RESULT - 2012-12-28
Ubuntu 12.04 64-bit RESULT - 2012-12-28

Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

In addition to manual spot-checks on all OS's below, we should run a breadth of tests on Windows 7 64bit, Mac OS X 10.7, Linux 64bit, that cover the most common use cases across critical areas of user interaction including video playback (Flash, Silverlight), private browsing, tabbed browsing, session restore - scenarios etherpad.

Platform Tester Result
Windows 7 spotcheck + in-depth manual testing Bogdan, Ioana PASS - 2012-12-28
Windows 8 Cornel PASS - 2012-12-28
Mac OSX 10.7 spotcheck + in-depth manual testing Paul, Ioana PASS - 2012-12-28
Mac OSX 10.8 Simona PASS - 2012-12-28
Ubuntu 12.04 spotcheck + in-depth manual testing Alexandra, Virgil PASS - 2012-12-28

Update Testing

Updates automation will not be run for this cycle due to bug 807099 so we will run a series of manual tests using a pair-wise approach that cover at least one type of scenario among the three main types of OS's.

  • Partial: 18.0b5, 18.0b4
  • Complete: <= 17.0b6
Platform Releasetest (report) Beta (report)
Windows XP RESULT - 2012-12-28 RESULT - 2012-12-28
Windows Vista RESULT - 2012-12-28 RESULT - 2012-12-28
Windows 7 32-bit RESULT - 2012-12-28 RESULT - 2012-12-28
Windows 7 64-bit RESULT - 2012-12-28 RESULT - 2012-12-28
Windows 8 32-bit RESULT - 2012-12-28 RESULT - 2012-12-28
Windows 8 64-bit RESULT - 2012-12-28 RESULT - 2012-12-28
Mac OS X 10.6 RESULT - 2012-12-28 RESULT - 2012-12-28
Mac OS X 10.7 RESULT - 2012-12-28 RESULT - 2012-12-28
Mac OS X 10.8 RESULT - 2012-12-28 RESULT - 2012-12-28
Ubuntu 12.04 32-bit RESULT - 2012-12-28 RESULT - 2012-12-28
Ubuntu 12.04 64-bit RESULT - 2012-12-28 RESULT - 2012-12-28
Issues

Beta 7

Builds Build 1: FTP, HTTP
ID 20121231071231
Changeset d5935d0242bc (pushlog)
Checklist

Automated Testing

Due to bug 807099 the Mozmill CI will not run for this cycle. However, we will try to run test runs, triggered manually, for Windows 7, Mac 10.7.x, and Linux (any).

Platform Build #1 Results (report) 2013-01-02
Windows 7 64-bit PASS - 2013-01-02
Mac OS X 10.7 PASS - 2013-01-02
Ubuntu 12.04 64-bit PASS - 2013-01-02

Manual Testing

The scope of this beta is three bugs:

  • IN PROGRESS - bug 825022 (mwobensmith) - pick up NSS roots module version 1.93
  • [DONE] - bug 815795 (juanb) - stack buffer overflow with canvas
  • [DONE] - bug 822682 (ashughes) - Firefox freezes on smartcard re-insertion

These need verification.

The rest is a manual spotcheck to make sure nothing is obviously broken.

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows 7 spotcheck juanb PASS - 2013-01-02
Windows 8 ashughes PASS - 2013-01-02
Mac OSX 10.7 spotcheck juanb PASS - 2013-01-02
Mac OSX 10.8 mwobensmith PASS - 2013-01-02
Ubuntu 12.04 spotcheck juanb PASS - 2013-01-02

Update Testing

  • Partial: 18.0b6, 18.0b5
  • Complete: <= 17.0b6
Platform Releasetest (report) Beta (report)
Windows XP PASS - 2013-01-02 RESULT - 2013-01-02
Windows Vista PASS - 2013-01-02 RESULT - 2013-01-02
Windows 7 32-bit PASS - 2013-01-02 RESULT - 2013-01-02
Windows 7 64-bit PASS - 2013-01-02 RESULT - 2013-01-02
Windows 8 32-bit PASS - 2013-01-02 RESULT - 2013-01-02
Windows 8 64-bit PASS - 2013-01-02 RESULT - 2013-01-02
Mac OS X 10.6 PASS - 2013-01-02 RESULT - 2013-01-02
Mac OS X 10.7 PASS - 2013-01-02 RESULT - 2013-01-02
Mac OS X 10.8 PASS - 2013-01-02 RESULT - 2013-01-02
Ubuntu 12.04 32-bit PASS - 2013-01-02 RESULT - 2013-01-02
Ubuntu 12.04 64-bit PASS - 2013-01-02 RESULT - 2013-01-02
Issues

Release Candidate

Builds Build 1: FTP, HTTP
ID 20130104151925
Changeset 650ead2dacc4
Checklist

Automated Testing

Platform Build #1 Results (report) 2013-01-06
Windows 7 64-bit RESULT - 2013-01-06
Mac OS X 10.7 RESULT - 2013-01-06
Ubuntu 12.04 64-bit RESULT - 2013-01-06


Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows 7 spotcheck Paul FAIL - 2013-01-07
Windows 8 Alexandra PASS - 2013-01-07
Mac OSX 10.7 spotcheck Manuela PASS - 2013-01-07
Mac OSX 10.8 Simona 'PASS - 2013-01-07
Ubuntu 12.04 spotcheck Ioana PASS - 2013-01-07

Etherpads

  • Spotchecking - link
  • Web fonts exploratory - link
  • WebGL exploratory - link

Update Testing

  • Partial: 17.0.1
  • Complete: <= 16.0.2
Platform Releasetest (report) Release (report)
Windows XP RESULT - 2013-01-07 RESULT - 2013-01-08
Windows Vista RESULT - 2013-01-07 RESULT - 2013-01-08
Windows 7 32-bit RESULT - 2013-01-07 RESULT - 2013-01-08
Windows 7 64-bit RESULT - 2013-01-07 RESULT - 2013-01-08
Windows 8 32-bit RESULT - 2013-01-07 RESULT - 2013-01-08
Windows 8 64-bit RESULT - 2013-01-07 RESULT - 2013-01-08
Mac OS X 10.6 RESULT - 2013-01-07 RESULT - 2013-01-08
Mac OS X 10.7 RESULT - 2013-01-07 RESULT - 2013-01-08
Mac OS X 10.8 RESULT - 2013-01-07 RESULT - 2013-01-08
Ubuntu 12.04 32-bit RESULT - 2013-01-07 RESULT - 2013-01-08
Ubuntu 12.04 64-bit RESULT - 2013-01-07 RESULT - 2013-01-08
Issues

Firefox 18.0.1

Builds Build 1: FTP, HTTP
ID 20130116073211
Changeset f64a3223c828
Pushlog Pushlog Changes
Checklist

Automated Testing

Platform Build #1 Results (report) 2013-01-16
Windows 7 64-bit PASS - 2013-01-16
Mac OS X 10.7 PASS - 2013-01-16
Ubuntu 12.04 64-bit PASS - 2013-01-16


Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows 7 spotcheck Manuela PASS - 2013-01-17
Windows 8 Ioana PASS - 2013-01-18
Mac OSX 10.7 spotcheck Alexandra PASS - 2013-01-17
Mac OSX 10.8 Simona PASS - 2013-01-18
Ubuntu 12.04 spotcheck Virgil PASS - 2013-01-17

Bug Verifications for 18.0.1

Fx18.0.1 Drivers:

  • VERIFIED FIXED bug 828632 - System Proxy Configuration Causes Memory Leak, syslogd load (raymond)
  • VERIFIED FIXED bug 828202 - System proxy settings issue with Firefox 18 (Manuela)
  • VERIFIED FIXED bug 828236 - issue with NTLM and some SSL Proxies (reporter)
  • VERIFIED FIXED bug 829646 - Proxy Auto Config - calling myIPAddress() in the top level (outside of FindProxyForURL) fails (Alexandra)

Ride-alongs:

  • VERIFIED FIXED bug 828954 - Firefox 18 breaks Unity content on OS X (juanb)
  • VERIFIED FIXED bug 818468 - Langpacks bundled in distribution/extensions are registered but disabled even if shown enabled (reporter)
  • VERIFIED FIXED bug 814434 - hidpi/retina url suggestion box pops up on wrong screen (mwobensmith)
  • VERIFIED FIXED bug 830303 - Fix plugin-container spawning on BSD after bug 753046 (reporter)
  • NEEDS CRASHSTATS bug 828184 - startup crash in nsPrefBranch::GetIntPref with QIPCAP.dll (Websense Endpoint) (juanb)
  • UNVERIFIABLE bug 828205 - crash in js:ebugScopeProxy::has with Firebug (Ioana)

Update Testing

  • Partial: 17.0.1, 16.0.2
  • Complete: < 16.0.2
Platform Betatest (report) Releasetest (report) Release (report)
Windows XP PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Windows Vista PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Windows 7 32-bit PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Windows 7 64-bit PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Windows 8 32-bit PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Windows 8 64-bit PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Mac OS X 10.6 PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Mac OS X 10.7 PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Mac OS X 10.8 PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Ubuntu 12.04 32-bit PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Ubuntu 12.04 64-bit PASS - 2013-01-17 RESULT - 2013-01-17 RESULT - 2013-01-17
Issues

18.0.2

Builds Build 1: FTP, HTTP
ID 20130201065344
Changeset c4fe2b219941
Pushlog Changes
Checklist
  • [DONE] Automated Testing
  • [DONE] Manual Testing
  • [DONE] Update Testing - betatest
  • [DONE] Bug Fix Verifications - By looking at crash-stats
  • [ON TRACK] Final sign-off sent and updates qualified on release channel

Automated Testing

Platform Build #1 Results (2013-02-01)
Windows XP PASS - 2013-02-01
Windows Vista PASS - 2013-02-01
Windows 7 32-bit PASS - 2013-02-01
Windows 7 64-bit PASS - 2013-02-01
Windows 8 32-bit PASS - 2013-02-01
Windows 8 64-bit PASS - 2013-02-01
Mac OS X 10.7 PASS - 2013-02-01
Mac OS X 10.8 PASS - 2013-02-01
Ubuntu 12.04 32-bit PASS - 2013-02-01
Ubuntu 12.04 64-bit PASS - 2013-02-01

Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows 7 Manuela PASS - 2013-02-04
Windows 8 Simona PASS - 2013-02-04
Mac OSX 10.7 Paul PASS - 2013-02-04
Mac OSX 10.8 Virgil PASS - 2013-02-04
Ubuntu 12.04 Bogdan PASS - 2013-02-04

Bug Fix Verifications

Drivers:

  • bug 806820 Crash in SuppressDeletedPropertyHelper (fixed by bug 831626)
    • We don't have good steps to reproduce either bug but Facebook is highly correlated

Ride-alongs:

Exploratory testing around Facebook - etherpad

Update Testing

  • Partial: TBD
  • Complete: TBD
Platform Betatest (report) Releasetest ([ report]) Release ([ report])
Windows XP PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Windows Vista PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Windows 7 32-bit PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Windows 7 64-bit PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Windows 8 32-bit PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Windows 8 64-bit PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Mac OS X 10.6 PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Mac OS X 10.7 PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Mac OS X 10.8 PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Ubuntu 12.04 32-bit PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Ubuntu 12.04 64-bit PASS - 2013-02-05 RESULT - YYYY-MM-DD RESULT - YYYY-MM-DD
Issues

Social API

18.0b4 Multi-window Changes

RE: bug 811247 In general the idea is to do some A-B testing, comparing 18.0b3 to 18.0b4.

Methodology

  • Start Firefox 18.0b3 with a new profile
  • Activate the Social API and note your current memory usage in about:memory
  • Run through the list of Things to Check below, make note of the panel loading times
  • Do some exploratory testing related to the toolbar button panels and multiple windows
  • Make note of the current memory usage in about:memory again
  • Start Firefox 18.0b4 with the same profile
  • Make note of the current memory usage in about:memory again
  • Run through the list of Things to Check below, make note of the panel loading times
  • Do some exploratory testing related to the toolbar button panels and multiple windows
  • Make note of the current memory usage in about:memory again
  • Clear the cache in 18.0b4 and minimize memory in about:memory
  • Make note of the current memory usage in about:memory again
  • Run through the list of Things to Check below, make note of the panel loading times
  • Do some exploratory testing related to the toolbar button panels and multiple windows
  • Make note of the current memory usage in about:memory again

What We're Looking For

  • Loading time regressions (reproduce in 18.0b4 but not 18.0b3)
  • UI/UX regressions (reproduce in 18.0b4 but not 18.0b3)
  • Stability regressions (reproduce in 18.0b4 but not 18.0b3)
  • Memory regressions (reproduce in 18.0b4 but not 18.0b3)

Things to Check

  • Clicking Friends toolbar button displays the Friends panel
  • Clicking Messages toolbar button displays the Messages panel
  • Clicking Notifications toolbar button displays the Notifications panel
  • Verify the UI of each panel is loaded properly
  • Verify you can scroll the panel if it's loaded with content
  • Verify you can select a panel item
  • Verify you can click the "Show All" link in each panel and the correct page loads
  • Verify panel behaviour switching back and forth between multple windows
  • Verify panel behaviour switching back and forth between multiple monitors

Platform Coverage (in priority order):

  • Windows: at least one of Win7, Win8, WinXP, Vista
  • Mac: at least one of OSX 10.8, 10.7, or 10.6
  • Linux: any recent version of Ubuntu

Results:

  • Ioana (Windows 7 32bit)
    • Load times:
      • 18.0b3: less than 1 second for each panel.
      • 18.0b4: less than 1 second for each panel.
      • 18.0b4 (no-cache): less than 1 second for each panel.
    • Memory (start | end):
      • 18.0b3: (202MB | 412MB)
      • 18.0b4: (198MB | 349MB)
      • 18.0b4 (no-cache): (111MB | 337MB) (111MB after cleaning the cache, 190MB after logging back in, 337MB after the exploratory)
    • Issues Found:
      • Opening chat windows from the messages notifications list displays inconsistent behavior (821284)
  • Virgil (Mac OS X 10.8)
    • Load times:
      • 18.0b3: less than 1 second for each panel.
      • 18.0b4: less than 1 second for each panel.
      • 18.0b4 (no-cache): less than 1 second for each panel.
    • Memory (start | end):
      • 18.0b3: (232.55MB | 270.55MB)
      • 18.0b4: (114.09MB | 135.61MB)
      • 18.0b4 (no-cache): (81.68MB | 126.36MB)
  • Simona (Windows 8 64bit)
    • Load times:
      • 18.0b3: less than 1 second for each panel.
      • 18.0b4: less than 1 second for each panel.
      • 18.0b4 (no-cache): less than 1 second for each panel.
    • Memory (start | end):
      • 18.0b3: (214MB | 463MB)
      • 18.0b4: (112MB | 320MB)
      • 18.0b4 (no-cache): (181MB | 400MB) (181MB after cleaning the cache, 219MB after logging back in, 400MB after the exploratory)
    • Issues Found:
      • Not all the notifications are displayed in the Social API sidebar (821316)
      • The panels are empty if the logging in is not kept and Firefox is closed (821323)
  • Manuela (Ubuntu 12.04)
    • Load times:
      • 18.0b3: less than 1 second for each panel.
      • 18.0b4: less than 1 second for each panel.
      • 18.0b4 (no-cache): less than 1 second for each panel.
    • Memory (start | end):
      • 18.0b3: (137MB | 145MB)
      • 18.0b4: (129MB | 134MB)
      • 18.0b4 (107MB | 124MB)
    • Issues Found:
      • Can't close Settings Menu when clicking outside the panel (821313)

Bugs

Triage

  1. QAWANTED: triage these bugs and test to fulfill the QA request
  2. UNCONFIRMED: triage these bugs for potential new regressions
  3. VERIFYME: verify the fix and do some exploratory testing around the issue
  4. FIXED: triage these bugs and test to ensure the fix is effective

New Issues

Bold any potential blockers. After every cycle make sure to call these out.

Social API

  • bug 821313 - Can't close Settings Menu when clicking outside the panel
  • bug 821316 - Not all the notifications are displayed in the Social API sidebar
  • bug 821323 - The panels are empty if the logging in is not kept and Firefox is closed
  • bug 821284 - Opening chat windows from the messages notifications list displays inconsistent behavior
  • bug 821642 - Scroll in Facebook notification panel makes background page to scroll as well

Menus

  • bug 814404 - [Mac OS] Missing menu items from Tools menu when focus is set to another Firefox window

Show PDF inline

  • bug 814686 - Extra space above the PDF documents
  • bug 814707 - Page doesn't fit the window correctly when using "Page fit" zoom option

Graphics

  • bug 815118 - Black pixels on Firefox button when hovering over tabs in taskbar
  • bug 815602 - Invalid drawable error in console after starting Firefox
  • bug 827364 - Fonts are not properly displayed
  • bug 827389 - Text from search box does not maintain the font and proportion of other text from the same page

Theme

  • bug 821240 - When Background Theme applied, Feedback button is improperly colored
  • bug 823560 - Collision between HTML5 videos and personas

Plugins

  • bug 821247 - Plugin Check displays that Java 7 Update 9 is outdated even if is up to date
  • bug 821304 - [Mac OS] Shockwave games no longer work in FF 18
  • bug 823533 - [Mac OS] Java doesn't work in 32-bit mode
  • bug 825172 - In full screen - video image freezes and the sound keeps on going if About Adobe Flash Player is selected
  • bug 825188 - Sound still plays after closing the video

Shell Integration

  • bug 823517 - Internet connection breaks after a crash in Firefox

Session Restore

  • bug 827374 - Menu Bar is not restored after Firefox crashes

Known Issues

Bold any potential blockers.

Themes

  • bug 616671 - Graphical glitch at the bottom of tabs when using darker Personas with tabs on top

HTML5 Video

  • bug 806095 - [10.8] Using JavaScript fullscreen API (mozRequestFullScreen) leaves part of the upper UI visible.
  • bug 708557 - Html5 youtube context menu not persistent when right clicking
  • bug 747909 - In Full Screen mode - copy actions are not allowed on HTML5 videos

Session Store

  • bug 791195 - Session is not restored after Firefox crash

WebGL

  • bug 787056 - MapsGL maps.google.com renders blurry on machines Chrome doesn't allow MapsGL enabling on
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_18/Test_Plan&oldid=512523"