Releases/Firefox 15/Test Plan

From MozillaWiki
Jump to: navigation, search

« Firefox 14 | Firefox 15 | Firefox 16 »

Contents

Firefox 15 Test Plan

Summary

Milestone Firefox 15
Status Currently in Beta -- shipping on August 28, 2012
Lead Anthony Hughes (irc: ashughes)
2nd Juan Becerra (irc: juanb)

The following is the test plan for Firefox 15 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 15. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

The following is a description of the tasks which need to be completed throughout this release.

  • 2012-04-24: [DONE] Nightly Merge
  • 2012-06-05: [DONE] Aurora Merge
  • 2012-06-22: [DONE] Aurora Testday
  • 2012-07-17: [DONE] Pre-Beta feature sign-offs
  • 2012-07-19: [ON TRACK] Beta 1
  • 2012-07-26: {{|}} Beta 2
  • 2012-08-02: {{|}} Beta 3
  • 2012-08-09: {{|}} Beta 4
  • 2012-08-16: {{|}} Beta 5
  • 2012-08-23: {{|}} Beta 6
  • 2012-08-27: {{|}} Release Candidate
  • 2012-08-28: {{|}} Final

Features

See the Feature Release Tracking wiki for a general overview

Feature Owner Status Tests
Show PDF Inline Mihaela Velimiroviciu ON TRACK MozTrap
Background Updates Vlad Ghetiu ON TRACK MozTrap
Debugger Ioana Budnar ON TRACK MozTrap
Responsive View Virgil Dicu ON TRACK MozTrap
Detailed Element Layout View Paul Silaghi ON TRACK MozTrap
Opt-in Activation for Plugins R2 Paul Silaghi ON TRACK MozTrap
Accessibility Improvements for Mac Vlad Ghetiu ON TRACK
Windows Start-up Performance Improvements Jason Smith ON TRACK
In-content preferences Mihaela Velimiroviciu ON TRACK MozTrap
Incremental Garbage Collection Ioana Budnar DELAYED to Fx?? MozTrap
Web Apps Integration to Desktop Jason Smith DELAYED to Fx16
Speedy Session Restore Paul Silaghi DELAYED to Fx??
Panel-based Download Manager Simona Badau DELAYED to Fx16 MozTrap
Developer Toolbar Virgil Dicu DELAYED to Fx16 MozTrap
Firefox Social Integration Anthony Hughes DELAYED to Fx16

Nightly

Merge

Functional Automation - result
Update Spotchecks - result


Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Endurance Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Mihaela Velimiroviciu) - Last Updated

Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Vlad Maniac / Alex Lakatos) - Last Updated
  • No issues found

Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs.

What to Do
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Anthony Hughes of the NEW issue
Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated

Result Spreadsheet


Aurora

Aurora Merge Sign-off

Automated Tests

Results of the Mozmill automation testruns.

Functional Tests
Platform Result: report Failures: known bugs
Windows XP PASS - 2012-06-06 no fails
Windows Vista PASS - 2012-06-06 no fails
Windows 7 32-bit PASS - 2012-06-06 no fails
Windows 7 64-bit PASS - 2012-06-06 1 fail
Mac OS X 10.6 PASS - 2012-06-06 2 fails
Mac OS X 10.7 PASS - 2012-06-06 1 fail
Ubuntu 11.10 32-bit PASS - 2012-06-06 no fails
Ubuntu 11.10 -bit PASS - 2012-06-06 no fails
Bugs
bug 761569 - testPrivateBrowsing/testCloseWindow.js
bug 761984 - testSearch/testFocusAndSearch.js
Endurance Tests
Platform Result: report Max Memory
Windows XP PASS - 2012-06-06 151 MB (+n/a from 14.0a2)
Windows Vista PASS - 2012-06-06 134 MB (+61% from 14.0a2)
Windows 7 32-bit PASS - 2012-06-06 139 MB (+46% from 14.0a2)
Windows 7 64-bit PASS - 2012-06-06 116 MB (+47% from 14.0a2)
Mac OS X 10.6 PASS - 2012-06-06 144 MB (+41% from 14.0a2)
Mac OS X 10.7 PASS - 2012-06-06 122 MB (+21% from 14.0a2)
Ubuntu 11.10 32-bit PASS - 2012-06-06 105 MB (+46% from 14.0a2)
Ubuntu 11.10 -bit PASS - 2012-06-06 129 MB (+30% from 14.0a2)
Bugs
bug 762215 - [cpg] Very large memory regression between Firefox 14 and 15

Feature Sign-offs

  • Log in to moztrap.mozilla.org
  • Select Run Tests
  • Select Product: Desktop Firefox
  • Select Versions: 15
  • Select Runs: Firefox 15 Aurora Merge Feature sign off
Feature Owner Win32 Mac Linux
Show PDF inline Mihaela Velimiroviciu PASS PASS PASS
Firefox Social Integration Simona Badau n/a n/a n/a
Incremental garbage collection Ioana Budnar FAIL PASS PASS
Silent Update: Background updates Vlad Ghetiu PASS PASS PASS
Windows Start-up Performance Improvements Jason Smith PASS n/a n/a
Opt-in activation for plugins (Phase 2) Paul Silaghi PASS PASS PASS
In-content preferences Mihaela Velimiroviciu PASS PASS PASS
Accessibility Improvements for Mac Vlad Ghetiu n/a n/a n/a
Developer Tool: Debugger Ioana Budnar PASS PASS PASS
Developer Tool: Responsive View Virgil Dicu PASS PASS PASS
Developer Tool: Detailed Element Layout View Paul Silaghi PASS PASS PASS
Web Apps Integration to Desktop Jason Smith PASS PASS PASS
Details

Report any detailed results or bugs here...

  • Firefox Social Integration Feature - The feature is still in definition, nothing is landed on Firefox 15.

Smoketests

Run the smoketests in MozTrap and report results below.

  • Log in to moztrap.mozilla.org
  • Select Run Tests
  • Select Product: Desktop Firefox
  • Select Versions: 15
  • Select Runs: Firefox 15 Aurora Merge Smoketests

Please email me feedback on using MozTrap and the test cases themselves

Platform Assignee Result: report
Windows 7 64-bit Simona PASS - 2012-06-08
Mac OS X 10.7.4 VladG PASS - 2012-06-08
Ubuntu 11.10 64-bit Mihaela PASS - 2012-06-08

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG

Do some smoketesting and exploratory testing to make sure the build works as expected.

  • [DONE] - VladG - ran the smoketests from MozTrap
  • [DONE] - VladG - exploratory on Youtube, Facebook, flash, silverlight, banking sites, etc

Exploratory Testing

Suggested areas to focus testing:

  • Plug-ins
    • PASS - Ioana, Simona
  • Audio/Video
    • FAIL - Ioana, Simona
    • 762903 - In Full screen "Hide Controls" is not applied on OGG videos
  • Social Networking
    • PASS - Ioana
  • Banking
  • Session Restore
    • PASS - Simona

Update Tests

auroratest
  • Win32: 14.0a2 en-US 2012-06-01 -> 15.0a2 en-US 2012-06-08 (PASS)
  • MacOS: 14.0a2 en-US 2012-06-02 -> 15.0a2 en-US 2012-06-08 (PASS)
  • Linux: 14.0a2 en-US 2012-05-31 -> 15.0a2 en-US 2012-06-08 (PASS)


Bugs Found

Full Query
ID Summary Priority Status
762868 Large text files are not printed entirely -- RESOLVED
762903 In Full screen "Hide Controls" is not applied on OGG videos -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Unprefixing Border-Image

Overview

In Firefox 15, the border-image is being unprefixed and will experience syntax changes as a result. The main syntax change is that border-image's border-style default is now none, not solid. However, as already noted from early testing, this results in regressions in top apps or sites. We need carefully identify what top sites are affected by this syntax change, determine the severity of the problem, and evangelize fixes to the site or app developer.

Top Sites Study

John Jensen did a study in the past that analyzed use of vendor prefixes across the web. We need to first take a look here to identify the list of sites that did end up using moz-border-image and see if the most important sites on the list regress by the new syntax changes. Testing of this then needs to exploratory test the following sites in this document here, specifically analyzing the layout of the site in regards to how border-image is used. For more information about this property, see this MDN document.

Important Bugs

Full Query
ID Summary Priority Status
730530 alliances.commandandconquer.com - Buttons not visible in Tiberium Alliances -- RESOLVED
748253 Lord of Ultima -moz-border-image does not render on buttons for FF 13 Aurora & FF 14 Nightly - Works on FF 12 Beta 6 -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Bug Triage

Query 06.18 - 06.22 06.25 - 06.29 07.02 - 07.06 07.09 - 07.13 Total
UNCONFIRMED 18 triaged 9 triaged 0 triaged 0 triaged 27 triaged
QAWANTED n/a n/a 1 resolved 6 resolved 7 resolved
FIXED 3 verified 0 verified 0 verified 0 verified 3 verified

Beta 1

Builds FTP, HTTP
ID 20120717110313
Changeset dc005f41ebdd
Checklist

Automated Tests

The following are the detailed results of the automated Mozmill test runs.

Platform Result: report
Windows XP PASS - 2012-07-17
Windows Vista PASS - 2012-07-17
Windows 7 32-bit PASS - 2012-07-17
Windows 7 64-bit PASS - 2012-07-17
Mac OS X 10.6 PASS - 2012-07-17
Mac OS X 10.7 PASS - 2012-07-17
Ubuntu 11.10 32-bit PASS - 2012-07-17
Ubuntu 11.10 64-bit PASS - 2012-07-17
Test Failures (known bugs)

Manual Smoketests

Run the Firefox 15 Beta 1 Smoketests in MozTrap and report the results below.

Platform Assignee Result: report
Windows XP Simona PASS - 2012-07-18
Windows 7 64-bit Paul PASS - 2012-07-18
Mac OS X 10.7 Mihaela PASS - 2012-07-18
Ubuntu 12.04 64-bit Ioana PASS - 2012-07-18

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-07-18
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-07-18
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-07-18
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-07-18
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-07-18
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-07-18
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-07-18

Spot check build after this test - PASS - VladG - 2012-07-08

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
    • When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Regression Testing

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo
Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Mihaela, Simona PASS - 2012-07-19 PASS - 2012-07-19 PASS - 2012-07-19
Windows 7 64-bit Paul PASS - 2012-07-18 PASS - 2012-07-18 PASS - 2012-07-18
Mac OS X 10.7 VladG PASS - 2012-07-18 PASS - 2012-07-18 PASS - 2012-07-18
Ubuntu 12.04 32-bit Virgil PASS - 2012-07-18 PASS - 2012-07-18 PASS - 2012-07-18

Update Testing

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-07-18 PASS - 2012-07-19
Windows Vista PASS - 2012-07-18 PASS - 2012-07-19
Windows 7 32-bit PASS - 2012-07-18 PASS - 2012-07-19
Windows 7 64-bit PASS - 2012-07-18 PASS - 2012-07-19
Mac OS X 10.6 PASS - 2012-07-18 PASS - 2012-07-19
Mac OS 10.7 PASS - 2012-07-18 PASS - 2012-07-19
Ubuntu 11.10 32-bit PASS - 2012-07-18 PASS - 2012-07-19
Ubuntu 11.10 64-bit PASS - 2012-07-18 PASS - 2012-07-19
Issues

New Issues

Bug Status
bug 775082 Video controls disappear after dragging the tabs Not a Fx15 regression, does not block release
bug 775114 Firefox does not hand off to Ubuntu archive utility for tar files Not a Fx15 regression, does not block release
bug 775488 Radio live streaming doesn't work with NoScript installed Not a Fx15 regression, does not block release

Beta 2

Builds FTP, HTTP
ID 20120724191344
Changeset 4d2c274ea4dc
Checklist

Automated Tests

The following are the detailed PASSs of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-07-26
Windows Vista PASS - 2012-07-26
Windows 7 32-bit PASS - 2012-07-26
Windows 7 64-bit PASS - 2012-07-26
Mac OS X 10.6 PASS - 2012-07-26
Mac OS X 10.7 PASS - 2012-07-26
Ubuntu 11.10 32-bit PASS - 2012-07-26
Ubuntu 11.10 64-bit PASS - 2012-07-26
Test Failures (known bugs)

Manual Smoketests

Run the Firefox 15 Beta 2 Smoketests in MozTrap and report the results below.

Platform Assignee Result: report
Windows XP Simona PASS - 2012-07-25
Windows 7 32-bit Mihaela PASS - 2012-07-25
Mac OS X 10.7 VladG PASS - 2012-07-25
Ubuntu 12.04 32-bit Virgil PASS - 2012-07-25

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-07-25
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-07-25
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-07-25
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-07-25
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-07-25
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-07-25
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-07-25

Spot check build after this test - PASS - VladG - 2012-07-25

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Regression Testing

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

  • Top AMO add-ons
  • AVG toolbar (installed via AVG software)
    • AVG Safe Search 12.0.0.1912 is not compatible with Firefox 15 beta.
    • AVG Security Toolbar can't be installed on Win XP - we get a blue screen while installing it.
  • Norton toolbar (installed via Norton software)
    • Not currently available. Investigating if we can get this software.
  • Yahoo toolbar
  • Yandex toolbar
  • Babylon toolbar
  • Ad-Aware Antivirus
    • The toolbar can't be installed on Firefox.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo
Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Simona PASS - 2012-07-26 PASS - 2012-07-26 PASS - 2012-07-26
Windows 7 32-bit Mihaela PASS - 2012-07-25 PASS - 2012-07-26 PASS - 2012-07-26
Mac OS X 10.7 VladG PASS - 2012-07-25 PASS - 2012-07-25 PASS - 2012-07-25
Ubuntu 12.04 64-bit Ioana PASS - 2012-07-25 PASS - 2012-07-25 PASS - 2012-07-26

Update Testing

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-07-26 PASS - 2012-07-27
Windows Vista PASS - 2012-07-26 PASS - 2012-07-27
Windows 7 32-bit PASS - 2012-07-26 PASS - 2012-07-27
Windows 7 64-bit PASS - 2012-07-26 PASS - 2012-07-27
Mac OS X 10.6 PASS - 2012-07-26 PASS - 2012-07-27
Mac OS 10.7 PASS - 2012-07-26 PASS - 2012-07-27
Ubuntu 11.10 32-bit PASS - 2012-07-26 PASS - 2012-07-27
Ubuntu 11.10 64-bit PASS - 2012-07-26 PASS - 2012-07-27
Issues

New Issues in Beta 2

Bug Status

Beta 3

Builds FTP, HTTP
ID 20120731150526
Changeset 21c5fe3f451e
Checklist

Automated Tests

The following are the detailed PASSs of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-07-31
Windows Vista PASS - 2012-07-31
Windows 7 32-bit PASS - 2012-07-31
Windows 7 64-bit PASS - 2012-07-31
Mac OS X 10.6 PASS - 2012-07-31
Mac OS X 10.7 PASS - 2012-07-31
Ubuntu 11.10 32-bit PASS - 2012-07-31
Ubuntu 11.10 64-bit PASS - 2012-07-31
Test Failures (known bugs)

Manual Smoketests

Run the Firefox 15 Beta 3 Smoketests in MozTrap and report the results below.

Platform Assignee Result: report
Windows XP Simona PASS - 2012-08-01
Windows 7 64-bit Paul PASS - 2012-08-01
Mac OS X 10.7 VladG PASS - 2012-08-01
Ubuntu 12.04 32-bit Virgil PASS - 2012-08-01

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-08-01
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-08-01
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-08-01
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-08-01
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-08-01
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-08-01
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-08-01

Spot check build after this test - PASS - VladG - 2012-08-01

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Regression Testing

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

  • Top AMO add-ons
  • AVG toolbar (installed via AVG software)
    • AVG Safe Search 12.0.0.1912 is not compatible with Firefox 15 beta.
    • AVG Security Toolbar can't be installed on Win XP - we get a blue screen while installing it.
  • Norton toolbar (installed via Norton software)
    • Not currently available. Investigating if we can get this software.
  • Yahoo toolbar
  • Yandex toolbar
  • Babylon toolbar
  • Ad-Aware Antivirus
    • The toolbar can't be installed on Firefox.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo
Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Simona PASS - 2012-08-01 PASS - 2012-08-01 PASS - 2012-08-02
Windows 7 64-bit Paul PASS - 2012-08-01 PASS - YYYY-MM-DD PASS - YYYY-MM-DD
Mac OS X 10.7 VladG PASS - YYYY-MM-DD PASS - YYYY-MM-DD PASS - YYYY-MM-DD
Ubuntu 12.04 64-bit Ioana PASS - 2012-08-01 PASS - 2012-08-02 PASS - 2012-08-02

Update Testing

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-08-02 PASS - 2012-08-02
Windows Vista PASS - 2012-08-02 PASS - 2012-08-02
Windows 7 32-bit PASS - 2012-08-02 PASS - 2012-08-02
Windows 7 64-bit PASS - 2012-08-02 PASS - 2012-08-02
Mac OS X 10.6 PASS - 2012-08-02 PASS - 2012-08-02
Mac OS 10.7 PASS - 2012-08-02 PASS - 2012-08-02
Ubuntu 11.10 32-bit PASS - 2012-08-02 PASS - 2012-08-02
Ubuntu 11.10 64-bit PASS - 2012-08-02 PASS - 2012-08-02
Issues

Beta 4

Builds FTP, HTTP
ID 20120808131812
Changeset 791245980a6e
Checklist

Automated Tests

The following are the detailed results of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-08-08
Windows Vista PASS - 2012-08-08
Windows 7 32-bit PASS - 2012-08-08
Windows 7 64-bit PASS - 2012-08-08
Mac OS X 10.6 PASS - 2012-08-09
Ubuntu 11.10 32-bit PASS - 2012-08-08
Ubuntu 11.10 64-bit PASS - 2012-08-08
Test Failures (known bugs)
  • /restartTests/testAddons_RestartlessExtensionWorksAfterRestart/test1.js (bug 719982)

Manual Smoketests

Run the Firefox 15 Beta 4 Smoketests in MozTrap and report the results below.

Platform Assignee Result: report
Windows XP Simona PASS - 2012-08-09
Windows 7 64-bit Paul PASS - 2012-08-09
Mac OS X 10.7 VladG PASS - 2012-08-10
Ubuntu 12.04 64-bit Ioana PASS - 2012-08-09

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-08-10
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-08-10
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-08-10
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-08-10
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-08-10
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-08-10
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-08-10

Spot check build after this test - PASS - VladG - 2012-08-10

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Regression Testing

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

  • Top AMO add-ons
  • AVG toolbar (installed via AVG software)
    • AVG Safe Search 12.0.0.1912 is not compatible with Firefox 15 beta.
    • AVG Security Toolbar can't be installed on Win XP - we get a blue screen while installing it.
  • Norton toolbar (installed via Norton software)
    • Not currently available. Investigating if we can get this software.
  • Yahoo toolbar
  • Yandex toolbar
  • Babylon toolbar
  • Ad-Aware Antivirus
    • The toolbar can't be installed on Firefox.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo
Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Simona PASS - 2012-08-09 PASS - 2012-08-09 PASS - 2012-08-10
Windows 7 32-bit Paul PASS - 2012-08-09 PASS - 2012-08-09 PASS - 2012-08-10
Mac OS X 10.7 VladG PASS - 2012-08-10 PASS - 2012-08-10 PASS - 2012-08-10
Ubuntu 12.04 32-bit Virgil PASS - 2012-08-09 PASS - 2012-08-09 PASS - 2012-08-09

Update Testing

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-08-10 PASS - 2012-08-10
Windows Vista PASS - 2012-08-10 PASS - 2012-08-10
Windows 7 32-bit PASS - 2012-08-10 PASS - 2012-08-10
Windows 7 64-bit PASS - 2012-08-10 PASS - 2012-08-10
Mac OS X 10.6 PASS - 2012-08-10 PASS - 2012-08-10
Mac OS 10.7 PASS - 2012-08-10 PASS - 2012-08-10
Ubuntu 11.10 32-bit PASS - 2012-08-10 PASS - 2012-08-10
Ubuntu 11.10 64-bit PASS - 2012-08-10 PASS - 2012-08-10
Issues

Beta 5

Builds FTP, HTTP
ID 20120814224555
Changeset 5b309ad1a88a (pushlog)
Checklist

Automated Tests

The following are the detailed results of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-08-15
Windows Vista PASS - 2012-08-15
Windows 7 32-bit PASS - 2012-08-15
Windows 7 64-bit PASS - 2012-08-15
Mac OS X 10.6 PASS - 2012-08-15
Mac OS X 10.7 PASS - 2012-08-15
Mac OS X 10.8 PASS - 2012-08-15
Ubuntu 11.10 32-bit PASS - 2012-08-15
Ubuntu 11.10 64-bit PASS - 2012-08-15
Test Failures (known bugs)

Manual Smoketests

Run the Firefox 15 Beta 5 Smoketests in MozTrap and report the results below.

Platform Assignee Result: report
Windows XP Simona PASS - 2012-08-16
Windows 7 64-bit Paul PASS - 2012-08-16
Mac OS X 10.7 Mihaela PASS - 2012-08-16
Ubuntu 12.04 32-bit Virgil PASS - 2012-08-16

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test - PASS - MihaelaV - 2012-08-17

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Regression Testing

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

  • Top AMO add-ons
  • AVG toolbar (installed via AVG software)
    • AVG Safe Search 12.0.0.1912 is not compatible with Firefox 15 beta.
    • AVG Security Toolbar can't be installed on Win XP - we get a blue screen while installing it.
  • Norton toolbar (installed via Norton software)
    • Not currently available. Investigating if we can get this software.
  • Yahoo toolbar
  • Yandex toolbar
  • Babylon toolbar
  • Ad-Aware Antivirus
    • The toolbar can't be installed on Firefox.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo
Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Simona PASS - 2012-08-16 PASS - 2012-08-16 PASS - 2012-08-16
Windows 7 32-bit Paul PASS - 2012-08-16 PASS - 2012-08-16 PASS - 2012-08-16
Mac OS X 10.7 Mihaela, Virgil PASS - 2012-08-16 PASS - 2012-08-16 PASS - 2012-08-16
Ubuntu 12.04 32-bit Virgil PASS - 2012-08-16 PASS - 2012-08-16 PASS - 2012-08-16

Update Testing

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-08-16 PASS - 2012-08-17
Windows Vista PASS - 2012-08-16 PASS - 2012-08-17
Windows 7 32-bit PASS - 2012-08-16 PASS - 2012-08-17
Windows 7 64-bit PASS - 2012-08-16 PASS - 2012-08-17
Mac OS X 10.6 PASS - 2012-08-16 PASS - 2012-08-17
Mac OS X 10.7 PASS - 2012-08-16 PASS - 2012-08-17
Mac OS X 10.8 PASS - 2012-08-16 PASS - 2012-08-17
Ubuntu 11.10 32-bit PASS - 2012-08-16 PASS - 2012-08-17
Ubuntu 11.10 64-bit PASS - 2012-08-16 PASS - 2012-08-17
Issues

Beta 6

Builds FTP, HTTP
ID 20120821170930
Changeset 2099c2bea208 (pushlog)
Checklist

Automated Tests

The following are the detailed results of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-08-22
Windows Vista PASS - 2012-08-22
Windows 7 32-bit PASS - 2012-08-22
Windows 7 64-bit PASS - 2012-08-22
Mac OS X 10.6 PASS - 2012-08-22
Mac OS X 10.7 PASS - 2012-08-22
Mac OS X 10.8 PASS - 2012-08-22
Ubuntu 11.10 32-bit PASS - 2012-08-22
Ubuntu 11.10 64-bit PASS - 2012-08-22
Test Failures (known bugs)

Manual Smoketests

Run the Firefox 15 Beta 6 Smoketests in MozTrap and report the results below.

Platform Assignee Result: report
Windows XP Paul PASS - 2012-08-22
Windows 7 64-bit Paul PASS - 2012-08-22
Mac OS X 10.7 Virgil PASS - 2012-08-22
Ubuntu 12.04 32-bit Virgil PASS - 2012-08-22

Regression Testing

Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Simona PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Windows 7 Mihaela PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Mac OS X ValdG PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Ubuntu 12.04 Ioana PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Operating System Tester Web Mail Mac 10.8 Signing Breakpad
Windows XP Simona PASS - 2012-08-22 N/A - YYYY-MM-DD PASS - 2012-08-22
Windows 7 Mihaela PASS - 2012-08-22 N/A - YYYY-MM-DD PASS - 2012-08-22
Mac OS X VladG PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Ubuntu 12.04 Ioana PASS - 2012-08-22 N/A - YYYY-MM-DD PASS - 2012-08-22

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

  • Top AMO add-ons
  • AVG toolbar (installed via AVG software)
    • AVG Safe Search 12.0.0.1912 is not compatible with Firefox 15 beta.
    • AVG Security Toolbar can't be installed on Win XP - we get a blue screen while installing it.
  • Norton toolbar (installed via Norton software)
    • Not currently available. Investigating if we can get this software.
  • Yahoo toolbar
  • Yandex toolbar
  • Babylon toolbar
  • Ad-Aware Antivirus
    • The toolbar can't be installed on Firefox.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

Web Mail

Test the following websites for rendering issues, functional issues, performance issues, and crashes. In particular make sure email, attachments, web messenger, and video chat are working as expected.

  1. Google Mail
  2. Yahoo Mail
  3. Hotmail
  4. Outlook.com

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-08-22
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-08-22
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-08-22
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-08-22
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-08-22
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-08-22
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-08-22

Spot check build after this test -PASS - VladG - 2012-08-22

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Breakpad End-to-End

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Update Testing

Note that we are doing partial updates for multiple source versions so we'll need to test on betatest first.

  • Partial: 15.0b4, 15.0b5
  • Complete: 15.0b3 and older
Platform Betatest: report Releasetest: report Beta: report
Windows XP PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Windows Vista PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Windows 7 32-bit PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Windows 7 64-bit PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Mac OS X 10.6 PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Mac OS X 10.7 PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Mac OS X 10.8 PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Ubuntu 11.10 32-bit PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Ubuntu 11.10 64-bit PASS - 2012-08-22 PASS - 2012-08-22 PASS - 2012-08-22
Issues

Release Candidate

Builds FTP, HTTP
ID 20120824154833
Changeset 450143d2d810 (pushlog)
Checklist

Automated Tests

The following are the detailed results of the automated Mozmill test runs.

Platform Results: report
Windows XP PASS - 2012-08-25
Windows Vista PASS - 2012-08-25
Windows 7 32-bit PASS - 2012-08-25
Windows 7 64-bit PASS - 2012-08-25
Mac OS X 10.6 PASS - 2012-08-25
Mac OS X 10.7 PASS - 2012-08-25
Mac OS X 10.8 PASS - 2012-08-25
Ubuntu 12.04 32-bit PASS - 2012-08-25
Ubuntu 12.04 64-bit PASS - 2012-08-25
Test Failures (known bugs)

Manual Smoketests

Run the Firefox 15 Smoketests in MozTrap and report the results below.

Platform Assignee Result: report
Windows XP Ioana PASS - 2012-08-26
Windows 7 64-bit Ioana PASS - 2012-08-26
Mac OS X 10.7 Virgil PASS - 2012-08-26
Ubuntu 12.04 32-bit Virgil PASS - 2012-08-26

Regression Testing

Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Ioana PASS - 2012-08-26 PASS - 2012-08-26 PASS - 2012-08-26
Windows 7 Ioana PASS - 2012-08-27 PASS - 2012-08-26 PASS - 2012-08-27
Mac OS X VladG PASS - 2012-08-27 PASS - 2012-08-27 PASS - 2012-08-27
Ubuntu 12.04 Virgil PASS - YYYY-MM-DD PASS - 2012-08-26 PASS - YYYY-MM-DD
Operating System Tester Web Mail Mac 10.8 Signing Breakpad
Windows XP Ioana PASS - 2012-08-26 N/A - YYYY-MM-DD PASS - 2012-08-26
Windows 7 Ioana PASS - 2012-08-27 N/A - YYYY-MM-DD PASS - 2012-08-26
Mac OS X VladG PASS - YYYY-MM-DD PASS - YYYY-MM-DD PASS - YYYY-MM-DD
Ubuntu 12.04 Virgil PASS - 2012-08-26 N/A - YYYY-MM-DD PASS - 2012-08-26

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

  • Top AMO add-ons
  • AVG toolbar (installed via AVG software)
    • AVG Safe Search 12.0.0.1912 is not compatible with Firefox 15 beta.
    • AVG Security Toolbar can't be installed on Win XP - we get a blue screen while installing it.
  • Norton toolbar (installed via Norton software)
    • Not currently available. Investigating if we can get this software.
  • Yahoo toolbar
  • Yandex toolbar
  • Babylon toolbar

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

Web Mail

Test the following websites for rendering issues, functional issues, performance issues, and crashes. In particular make sure email, attachments, web messenger, and video chat are working as expected.

  1. Google Mail
  2. Yahoo Mail
  3. Hotmail
  4. Outlook.com

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-08-27
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-08-27
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-08-27
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-08-27
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-08-27
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-08-27
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-08-27

Spot check build after this test - PASS - VladG - 2012-08-27

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Breakpad End-to-End

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Update Testing

Note that we are doing partial updates for multiple source versions so we'll need to test on betatest first.

  • Partial: 12.0, 13.0.1, 14.0.1
  • Complete: 13.0, 11.0, 10.0, 10.0.1, 10.0.2
  • Spotcheck: 4-9 -> 12 (complete) -> 15 (partial)
Platform Releasetest: report Release: report
Windows XP PASS - 2012-08-27 PASS - 2012-08-28
Windows Vista PASS - 2012-08-27 PASS - 2012-08-27
Windows 7 32-bit PASS - 2012-08-27 PASS - 2012-08-27
Windows 7 64-bit PASS - 2012-08-27 PASS - 2012-08-27
Mac OS X 10.6 PASS - 2012-08-27 PASS - 2012-08-27
Mac OS X 10.7 PASS - 2012-08-27 PASS - 2012-08-27
Mac OS X 10.8 PASS - 2012-08-27 PASS - 2012-08-27
Ubuntu 11.10 32-bit PASS - 2012-08-27 PASS - 2012-08-27
Ubuntu 11.10 64-bit PASS - 2012-08-27 PASS - 2012-08-27
Issues

15.0.1

Builds FTP, HTTP
ID 20120905151427
Changeset 0b774a1067fe (pushlog)
Checklist

Automated Tests

The following are the detailed results of the automated Mozmill test runs.

Platform Results: report
Windows XP PASS - 2012-09-06
Windows Vista PASS - 2012-09-06
Windows 7 32-bit PASS - 2012-09-06
Windows 7 64-bit PASS - 2012-09-06
Mac OS X 10.6 PASS - 2012-09-06
Mac OS X 10.7 PASS - 2012-09-06
Mac OS X 10.8 PASS - 2012-09-06
Ubuntu 12.04 32-bit PASS - 2012-09-06
Ubuntu 12.04 64-bit PASS - 2012-09-06
Test Failures (known bugs)

Manual Testing

Private Browsing

RE bug 787743 - Private Browsing mode not working in Firefox 15.0

Testing

Use Firefox 14.0.1 as your baseline comparison -- we are only concerned with shipping new regressions which did not exist in Firefox 14.

Basic Test:

  1. Clear your cache through Tools > Clear Recent History
  2. Open about:cache and verify the Disk Cache is empty
  3. Start Private Browsing mode
  4. Use Firefox, taking note of the websites you visit
  5. Stop Private Browsing mode
  6. Open about:cache and verify the Disk Cache and Memory Cache do not contain any entries referring to the sites you visited in Private Browsing mode
  7. Grep your profile to make sure no references to the website(s) you visited in Private Browsing mode are discoverable in the profile on your filesystem
  • cd ${PROFILE_DIRECTORY}
  • grep -r "sitename" .

Other things to watch for:

  • Visited URLs, loaded images, media, and other content, authenticated sessions, download manager entries, and cookies should NOT be discoverable outside of Private Browsing mode
  • Bookmarks, tags, and site-specific permissions set during Private Browsing SHOULD be discoverable outside of Private Browsing mode
  • Private Browsing data does not leak into History after exiting Private Browsing mode
  • Private Browsing data does not leak into Awesomebar after exiting Private Browsing mode
  • Private Browsing data does not leak into New Tab page after exiting Private Browsing mode
  • Private Browsing data does not leak into Home page after exiting Private Browsing mode
  • Private Browsing data does not leak across multiple subsequent Private Browsing sessions after exiting Private Browsing mode
  • Private Browsing data does not leak into Session Restore from restart, crash, about:sessionstore, software update, add-on install restart, system hard reset, system restore from sleep/suspend/hibernate, etc
Exploratory testing
Results
Platform Tester Try (builds) Candidate
Windows XP Ioana PASS - 2012-09-05 PASS - 2012-09-06
Windows Vista VladG PASS - 2012-09-05 PASS - 2012-09-06
Windows 7 32-bit Mihaela PASS - 2012-09-05 PASS - 2012-09-06
Windows 7 64-bit Mihaela PASS - 2012-09-05 PASS - 2012-09-06
Mac OS X 10.6 VladG PASS - 2012-09-05 PASS - 2012-09-06
Mac OS X 10.7 VladG PASS - 2012-09-05 PASS - 2012-09-06
Mac OS X 10.8 Virgil PASS - 2012-09-05 PASS - 2012-09-06
Ubuntu 12.04 32-bit Virgil PASS - 2012-09-05 PASS - 2012-09-06
Ubuntu 12.04 64-bit Ioana PASS - 2012-09-05 PASS - 2012-09-06
Issues Found

Update Testing

  • Partial: 13.0.1, 14.0.1, 15.0
  • Complete: 12.0, 13.0
Platform Releasetest: report Release: report
Windows XP PASS - 2012-09-06 PASS - 2012-09-06
Windows Vista PASS - 2012-09-06 PASS - 2012-09-06
Windows 7 32-bit PASS - 2012-09-06 PASS - 2012-09-06
Windows 7 64-bit PASS - 2012-09-06 PASS - 2012-09-06
Mac OS X 10.6 PASS - 2012-09-06 PASS - 2012-09-06
Mac OS X 10.7 PASS - 2012-09-06 PASS - 2012-09-06
Mac OS X 10.8 PASS - 2012-09-06 PASS - 2012-09-06
Ubuntu 12.04 32-bit PASS - 2012-09-06 PASS - 2012-09-06
Ubuntu 12.04 64-bit PASS - 2012-09-06 PASS - 2012-09-06
Issues

Hotfix

Hotfix to turn off Background Updates (see bug 790096).

Test Case

Hotfix installs for:

  • Firefox 15 - 18 users on...
  • Mac & Linux with...
  • add-ons installed in the app-bundle (how do you do this????) -- ex. Comcast software

Hotfixed is active if:

  • Hotfix add-on appears in Addons Manager
  • Background Update pref is turned off

Updates to check:

  • Updates not found via background update
  • Updates found via manual check for updates

Questions:

  • How do you install add-ons in the app-bundle? What is an app-bundle? and are there specific add-ons?
  • Hotfix installer dialog?
  • Background Update pref?
  • Expectation of pave-over install with "affected" version?
  • Expectation of pave-over install with "unaffected" version?

Expectations:

  • 15.0.1 release: hotfix applies, won't be able to test updates since none available
  • 15.0 release: hotfix applies, no background updates to 15.0.1
  • 15.0b6 beta: hotfix applies, no background updates to 16.0b3
  • 15.0a2 aurora: hotfix applies, no background updates to 17.0a2, hotfix removed after update
  • 16.0b2 beta: hotfix applies, no background updates to 16.0b3
  • 17.0a2 aurora: hotfix applies for those with app-bundle add-ons
  • 18.0a1 nightly: hotfix applies for those with app-bundle add-ons

Beta Feature Sign-off

Run the Firefox 15.0 pre-beta sign off tests in MozTrap and report results below.

Feature Owner Win32 Mac Linux
Show PDF Inline Mihaela Velimiroviciu disabled - 2012-07-13 disabled - 2012-07-13 disabled - 2012-07-13
Silent Update: Background updates Vlad Ghetiu PASS - 2012-07-13 PASS - 2012-07-13 PASS - 2012-07-13
Debugger Ioana Budnar PASS - 2012-07-13 PASS - 2012-07-17 PASS - 2012-07-13
Responsive Mode Virgil Dicu PASS - 2012-07-13 PASS - 2012-07-13 PASS - 2012-07-13
Detailed Element Layout Paul Silaghi PASS - 2012-07-13 PASS - 2012-07-13 PASS - 2012-07-13
Opt-in activation for plugins Phase 2 Paul Silaghi PASS - 2012-07-13 PASS - 2012-07-13 PASS - 2012-07-13
Accessibility Improvements for Mac Vlad Ghetiu N/A - 2012-07-13 PASS - 2012-07-13 N/A - 2012-07-13
Start-up Performance Improvements Jason Smith N/A - 2012-07-13 PASS - 2012-07-17 N/A - 2012-07-13
Panel-based Download Manager Simona Badau disabled - 2012-07-13 disabled - 2012-07-13 disabled - 2012-07-13
Incremental Garbage Collection Ioana Budnar disabled - 2012-07-13 disabled - 2012-07-13 disabled - 2012-07-13
Firefox Developer Toolbar Virgil Dicu disabled - 2012-07-13 disabled - 2012-07-13 disabled - 2012-07-13
In-content Preferences Mihaela Velimiroviciu disabled - 2012-07-13 disabled - 2012-07-13 disabled - 2012-07-13

Notes:

Bug Triage

  1. VERIFYME: verify the fix and do some exploratory testing around the issue
  2. QAWANTED: triage these bugs second and test to fulfill the QA request
  3. FIXED: triage these bugs third and test to verify the fix
  4. UNCONFIRMED: triage these bugs last and test for new regressions

Results

  • VERIFIED: 180 (46%)
  • QA-: 55 (14%)
  • QA?: 16 (4%)
  • UNTRIAGED: 139 (37%)
  • 390 Bugs Total

Verification Spreadsheet

Beta Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Known Issues

Bug Status Assignee
bug 785070 [Mac] plugins.click_to_play doesn't detect whether Silverlight is installed 2012-08-23: NEW - Fx14 regression Paul Silaghi
bug 777051 Clicking "Replay" button on HTML5 Youtube Videos makes the video freeze at the beginning 2012-08-16: NEW - not a regression (probably a change on youtube side)
bug 781470 HTML5 video play button doesn't fade out 2012-08-16: NEW - 12.0a1 regression
bug 781190 Video images freeze with the latest Flash 11.4.400.252 2012-08-16: NEW - plugin bug
bug 779492 Youtube pop-out video opens in the background 2012-08-16: NEW - plugin bug
bug 779465 Delayed scrolling in PDFs 2012-08-16: NEW - plugin bug
bug 778819 Can't toggle fullscreen via context menu in Flash 11.3 2012-08-16: WONTFIX
bug 775114 File-handler bindings broken for mozilla.org Firefox builds on Linux 2012-08-16: NEW - wontfix?
bug 775082 Video controls disappear after dragging the tabs 2012-08-16: NEW - 4.0b7 regression
bug 760417 Pop-out HTML5 video opens full-screen, still displays Full-screen button 2012-08-16: NEW - debugging
bug 757978 Uninstalling Firefox leaves behind a webapprt folder 2012-08-16: NEW - in-dev Tim Abraldes (DEV)
bug 747909 In Full Screen mode copy actions are not allowed on HTML5 videos 2012-08-16: NEW - regression? Simona Marcu (QA)
bug 692446 Firefox prompts to install WMP plugin even though it's installed 2012-08-16: NEW - in-dev Robert Strong (DEV)
bug 675260 Crash in DefaultFreeEntry coming from ShutdownNSS 2012-08-16: NEW - untrack? Brian Smith (DEV)
bug 667399 RealPlayer G2 LiveConnect plugin crash 2012-08-16: NEW - untrack?
bug 717178 Printing stops after 73%, and browser hangs up 2012-08-16: NEW
bug 675645 Firefox freezes when using Pop out while in Fullscreen mode on Youtube 2012-08-26: NEW

On Call

Please add your name below along with your availability and preferred contact method for the weekend of August 25th and 26th.

Name Available Aug 25th? Available Aug 26th? Contact
Anthony Hughes (:ashughes) Yes - all day Yes - all day phone (see phonebook)
Kevin Brosnan (:kbrosnan) Yes - all day Yes - all day phone (see phonebook)
Matt Brandt (:mbrandt) No Yes - all day phone (see phonebook)
Retornam (:retornam) Yes - all day Yes - halfday phone (see phonebook)
Tracy Walker (:tracy) Yes - all day Yes - all day phone (see phonebook)
Ioana Budnar, SV (:ioana) Yes - all day - 9 AM - 6 PM Romania Time (11PM - 8AM PST) Yes - all day - 9:30AM - 6:30 PM (11:30PM - 8:30AM PST) email
Paul Silaghi, SV (:Pauly) Yes - all day - 9 AM - 6 PM Romania Time (11PM - 8AM PST) No email
Virgil Dicu, SV (:virgil) No Yes - all day - 9:30AM - 6:30 PM (11:30PM - 8:30AM PST) email
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_15/Test_Plan&oldid=469590"