QA/Firefox3/TestPlan/MajorUpdate
Major Update Test Plan: Fx 2.0x - Fx 3.0x
Contents
Overview
We are offering major updates to Firefox 2.0.0.x users so they can upgrade to Firefox 3. There has been one successful Major Update so far, from 2.0.0.16 to 3.0.1. The current plan is to offer a Major Update from 2.0.0.18 to 3.0.4 during the next few weeks and then a second one from 2.0.0.19 to 3.0.5, which will hopefully be the final MU.
Scope
We are going to test basic major update functionality in some P1 locales, CJKT locales, and partner builds. We want to make sure basic functionality in the resulting build is the same as a fresh Fx3 install, and we must ensure user preferences are preserved.
Most of this testing will be manually executed. However the partner distribution builds will, in addition, be tested with some automation. The major update test plan for partner distributions is here.
We will need to test MU for CJKT builds as these have special preferences that need to be preserved (?) after the major update. For refence see: CJKT Matrix for Fx1.5x - Fx2 MU.
Schedule
The Major Update will probably ship in the week following Thanksgiving, December 1 through 5. This is also the time period when the first Firefox 2.0.0.19 and 3.0.5 builds will become available so testing needs to be finished for the MU before then.
Current schedule for QA is:
- Begin QA in Betatest channel: 11/18
- Release to Beta: 11/20
- Testing in Releasetest channel: 12/1
- Release: 12/2
We need to test the major update scenario in the betatest channel, release it to the beta channel and then do releastest and release following this.
Basic Tests
At a basic level we should make sure:
- Updates are presented
- In-line content for Details and EULA is localized
- Update does not break basic functionality for the user
- User data and preferences are preserved
Extended Tests
The extended tests take into account rich user profiles. For this, we will start with an Fx2 build with bookmarks, history, saved form data, saved passwords, feeds, extensions and themes (mix of compatible and incompatible with Fx3), plugins, default search engine preferences, and any other customizations a user may have for a typical Fx200x installation.
During the major update the extensions and themes should be checked for compatibility.
After going through the major update all (most) of the user's preferences should be preserved.
Instructions
- Download Firefox 2.0.0.20
- Install it, shut down and change the update channel to betatest
- it's in <app dir>/defaults/pref/channel-prefs.js
- Restart the browser and select "Check for Updates" from the Help menu
Current Results
We will track the 2.0.0.20 -> 3.0.10 Major Update testing results in the Fx20020-Fx3010MU Test Plan.
Previous Results
For previous iterations of MU testing results see the following links as they are produced:
- Fx3.0.6 - Fx3.1b3 MU Test Plan
- Fx20018-Fx304 testing
- Fx20016-Fx301 MU Results on beta all tier 1 l10n Fx20016 - 8.23.2008
- Fx2.0.0.16-Fx3.0.1 MU results on releasetest and betatest - as of 8.21.2008
- Fx2.0.0.16-Fx3.0.1 MU results on majortest - Current iteration as of 8.20.2008
- Fx2.0.0.14-Fx3.0 MU results on majortest - as of 7.16.2008
- Fx2.0.0.11-Fx3beta3 MU results on betatest - Previous iterations
Reference (legacy)
A detailed list of scenarios QA must watch out for is here: http://wiki.mozilla.org/Firefox3/UpgradePolicy
For the 1.5x to 2.0x major updates testing we created a number of tests in Litmus. They are in the 1.5x - 2.0x Major Update testgroup and FX 1.5-2.0 Major Update - Basic Tests and FX 1.5-2.0 Major Update - Extended Tests subgroups in the Firefox 2.0 Catchall test run.
The major update dialog will have an explanatory page, and it will be localized. For example, the 1.5x - 2.0x MU had dialogs that look like this: http://www.chevrel.org/mozco/20migration/display.php
The following is a list of tests taken from the "1.5x - 2.0x Major Update" test group in Litmus, as explained above.
check Details and EULA description check that you can only install if you accept license check eulas and descriptions are localized (Paul K., send a list of those that are ready) check major update works on different locales check interface elements work, buttons, links, text selection, scrolling, closing check "fallback" major updates (should there be a failure in the update download for example) check uninstalling and going back to previous build check only one type of update is presented at a time minor/major check major updates if you have multiple profiles check extension compatibility check check theme compatibility check check what tabs are default after you update. check these start pages are localized as before. check Windows bits are signed after major update. post: check minors are presented to the upgraded 2.0.0.1 users post: check major updates are presented only to those using the latest 150x post: check minor updates are presented for those users who said never post: check 200x users are not presented a major post: check majors a presented when user had said never, then manually checked for updates again (assuming major are once again available) check to see what happens when one type of update has been downloaded and not applied, and another type of update is made available. check to see if it is possible to download and say later and then regret and then install something better, like a major update. check what is left over after a "diff" (currently some crumbs are left for Win and Mac and it has been captured in a bug) check the state of the registry after a major update (and uninstall)