QA/Firefox3/TestResults/M11

From MozillaWiki
< QA‎ | Firefox3‎ | TestResults
Jump to: navigation, search

« QA/Firefox3/TestResults

Firefox 3 M11 Test Results

Related Pages

Links to separate test results. Some are referenced further below.

Contact Info

PM: Mike Beltzner (beltzner)
Build: John O'Duinn (joduinn)
QA: Tony (tchung)

Schedule

Code Freeze
1/22
Code Build
ETA 2/4
QA test signoff
ETA 2/10 Completed on 2/10, 5:40pm PST
  • Smoketests (2/5-2/6) 100%
  • BFTs (2/5-2/7) 100%
  • M11 and past focused Tests (2/5-2/10) ongoing
  • Top Websites (2/7-2/8) 100% Mac, 100% windows, 100% Linux
  • Major Updates (Fx2 -> Fx3 trial run) 100%
  • Update snippets from Alpha X, Beta 1,2 -> Beta 3 (2/7 - 2/8) 100% betatest, 100% releasetest, 100% beta
  • Memory Leak Testing against FFTs (handled by Tomcat on debug builds) 100%
  • P1 L10N spotchecks (2/8) 100%
  • M11 bug verifications (ongoing) ongoing
  • Beta 3 Testday (2/8) 100%
  • Release notes, Websites (2/9, 2/10) 100%

Release Notes

Build Location and IDs

RC3

ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b3-candidates/rc3/

  • Mac 10.4: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
  • Mac 10.5: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
  • Linux: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b3)Gecko/2008020513 Firefox/3.0b3
  • WinXP: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b3)Gecko/2008020514 Firefox/3.0b3
  • WinVista: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b3)Gecko/2008020514 Firefox/3.0b3

Strategy

  • Areas Covered:
    • Smoketests
    • BFTs
    • M11 and past focused Tests
    • Top Websites
    • Major Updates (Fx2 -> Fx3 trial run)
    • Update snippets from Alpha X, Beta 1,2 -> Beta 3
    • Memory Leak Testing (handled by Tomcat on debug builds)
    • P1 L10N spotchecks
    • M11 bug verifications
    • Beta 3 Testday
  • Areas Not Covered:
    • localization (String changes) - 3rd party will cover
    • Distribution

Test Results

Types of Tests Win XP Win Vista Mac Tiger Mac Leopard Linux
Smoketest marcia - PASS marcia - PASS Tomcat - PASS Tomcat - PASS Tomcat - PASS
BFT Tomcat (running leak tests on FFT) - PASS Tomcat (running leak tests) - PASS Tomcat (running leak tests on FFT) - PASS Marcia - PASS Tracy - PASS - beta3
Top Websites Testing (see below) Juanb - PASS N/A Juanb - PASS N/A Tracy - PASS
Updates Testing (see below) Tomcat - PASS Juanb - PASS Tomcat - PASS Marcia - PASS Tchung - PASS
l10N Pri 1 locales Tomcat - PASS N/A Tchung - PASS N/A Tracy - PASS
Bug Verifications Everyone - result Everyone - result Everyone - result Everyone - result Everyone - result

Top Sites Testing

Select the top 50 sites, and run through a combination of various web applications.

Testcases

  • Visit various sites on the list, verifying web compatibility
  • Detect Layout issues, Font issues, formatting problems, broken functionality
    • Some baseline comparisons is to use Fx 2.0.0.11, Safari, and Trunk nightly builds
  • Change browser font sizes, languages and verify web compatibility
  • Operate end to end functionality of a website. Examples:
    • track an auction item on ebay
    • save a book on amazon checkout
    • register for a new email account on Yahoo Mail
    • Create a document on Google docs
    • Visit a secure site and perform online banking

Notes: Fx3b3, problem rendering google maps street view bug 407953. This should be release noted.

Updates Testing

For b3, we will be doing updates of all Alpha and Beta 1 users to Beta 3. Tests will include different locales for beta 1, but only en-US for all past versions. We will be checking partial updates, complete updates, and failed status.

Will be available on betatest, releasetest, and beta channels

  • Betatest channel - PASS
  • Releasetest channel - PASS
  • Beta channel (live) - PASS

Testcases and Results are tracked here.

We will also do a special round of major update testing from Fx2-> Fx3 trial run. Testplan is held here.

  • Major Update - PASS

L10N P1 Spotchecks

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Automation inspection: supported locales will be ran through Minotaur, and results will be reviewed by axel and mic.

Axel's Localization Status Page

P1 Locales to be tested:

  • French (fr) pass tracy
  • Chinese (zh-CN) pass
  • German (de) pass tracy
  • Japanese (ja) pass
  • Spanish (es-ES) pass
  • Netherlands (nl) pass: but default Livemark wasn't loading tracy
  • Italian (it) pass tracy
  • Finnish (fi) pass tracy
  • Hindi (pa-IN) pass

Testcases and Results are tracked here.

Bug Verifications

Bugs that are marked RESOLVED FIXED that are checked in for beta. Please verify them.

Firefox bugs
Platform P1 bugs
Platform P2 bugs

Testday

A beta 3 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found on QMO.

Testday results are posted here.

Detailed Results

Tomcats Results

Smoketests RC 1

    • Linux
      • Date: 5.February 2008
      • Build id: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b3) Gecko/2008020421 Firefox/3.0b3
      • Status: PASS Results in Litmus
    • Mac 10.4
      • Date: 5.February 2008
      • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b3) Gecko/2008020418 Firefox/3.0b3
      • Status: PASS Results in Litmus (only known bugs found - no regressions)
    • Mac 10.5
      • Date: 5.February 2008
      • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020418 Firefox/3.0b3
      • Status: PASS Results in Litmus

Smoketest RC 2

    • Mac 10.4
      • Date: 5.February 2008
      • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b3)Gecko/2008020511 Firefox/3.0b3
      • Status: PASS Results in Litmus (only known bugs found - no regressions)
    • Mac 10.5
      • Date: 5.February 2008
      • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
      • Status: PASS Results in Litmus
    • Linux
      • Date: 6.February 2008
      • Build ID: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b3)Gecko/2008020513 Firefox/3.0b3
      • Status: PASS - Results in Litmus

Smoketest RC 3

    • Mac 10.4
      • Date: 7.February 2008
      • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b3)Gecko/2008020511 Firefox/3.0b3
      • Status: PASS Results in Litmus (only known bugs found - no regressions)
    • Mac 10.5
      • Date: 7.February 2008
      • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
      • Status: PASS Results in Litmus
    • Linux
      • Date: 7.February 2008
      • Build ID: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b3)Gecko/2008020513 Firefox/3.0b3
      • Status: PASS - Results in Litmus
    • Windows XP
      • Date: 7.February 2008
      • Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3
      • Status: PASS - Results in Litmus
    • Windows Vista
      • Date: 7.February 2008
      • Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3
      • Status: PASS - Results in Litmus
  • FFTS are in progress but they are more leak testing focused and will completed in the next weeks. So far it look good and no major regression found.
  • Bugs filed
    • bug 415826 Application Handler -> Web Feed -> Application details is empty
    • bug 416730 ###!!! ASSERTION: Someone forgot to properly cancel this request!
    • bug 416459 Undo does not restore Cut and Pasted Bookmark
    • bug 416383 Crash going back on and forth on www.strato.de
    • bug 416036 ###!!! ASSERTION: Oops! You're asking for a weak reference to an object that doesn't support that.: 'factoryPtr', file nsWeakReference.cpp,
    • bug 417101 Firefox crashed because of APNG after Update to Beta 3

marcia's Results

Smoketest RC 2

    • Win Vista Business
    • Tested on a QA Lab Machine
      • Date: 6.February 2008
      • Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3
      • Status: PASS Results in Litmus
      • Issues noted: Bug 415968
    • Win XP
    • Tested on a clean Win XP VM
      • Date: 6.February 2008
      • Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3
      • Status: PASS Results in Litmus
      • Issues noted: Mismatch between Text on Crash Reporter and Updater icons. They both say "Mozilla Foundation" while Firefox.exe says corporation. They are digitally signed by the Corporation. Filed Bug 415993

BFT

    • Mac 10.5
    • Intel build tested on a MacBook Pro; PPC build tested on a G5
      • Date: 5.February 2008
      • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3 and Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
      • Status: PASS Results in Litmus
      • Also Ran Content Handling portion of the FFT to account for the fact that there was minimal coverage in the BFT for that area.
      • Bugs Noted to Date:
        • Bug 416252 Save Page as invokes Tagging Downloaded Applications dialog
        • Bug 416048 All popup windows do not close when using File | Exit at popuptest.com
        • Bug 416021 Installation of incompatible theme shows "Waiting" in the Install Tab
        • Bug 415968 Inconsistent Migration Behavior on non-Vista/XP VMS
        • Bug 416392 [10.5] Difficult to tell which download is highlighted in DM (duped to another bug but I still think it is a bug)
        • Bug 416421 Firefox hangs when you try to quit the browser and have "Ask every time" set as default
        • bug 416850 Deleting a security exception and then re-adding it causes cert exception to remain valid
Noted in BFT but fixed in Nightly
  • 10.5-> Unable to move forward using the forward button in Places Manager
  • 10.5-> Unable to cut and paste a bookmark to a newly created folder in Places Manager

Tchung's Results

  • bug 412348 – Titles are missing in smart Bookmarks and Addressbar for urls w/out www. Found this to be happening when updating alpha1 -> beta 3. nominated to block beta 4.
  • bug 416444 – Gran Paradiso icon appears for bookmarks moved to desktop after updating to FFbeta3

Live Spot Checks

Live site downloads for M11, after bits are pushed out. Please cover en-US, and 2 or 3 additional Tier 1 locales as well. Pay attention to the following areas:

  • Check for signed installer and applications (windows only)
  • Firstrun page, whatsnew page, Home page, Search plugin
  • Installer, uninstall
  • General spotchecks and browser functionality


Windows XP -> abillings - PASS

  • en-US: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3
  • fr: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3
  • ja: Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3

Windows Vista -> stephend - PASS

  • en-US: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3
  • zh-TW: Mozilla/5.0 (Windows; U; Windows NT 6.0; zh-TW; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3
  • sv-SE: Mozilla/5.0 (Windows; U; Windows NT 6.0; sv-SE; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3

Mac Tiger -> Juan - Pass - Only ran a couple of locales, es-ES, and ja-jp-mac

Mac Leopard -> Marcia - PASS

  • Application icon missing - known issue
  • Bug 417107 filed against Frisian
  • frisian: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; @AB_CD@; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
  • cs: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; cs; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
  • uk: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; uk; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3

Linux -> stephend - PASS

  • en-US: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b3) Gecko/2008020513 Firefox/3.0b3
  • zh-CN: Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9b3) Gecko/2008020513 Firefox/3.0b3

Top Issues Reported in Hendrix Against Firefox 3 Beta 3

As of 2/19/08

  • Hotmail does not work right - 22 reports
  • Yahoo mail beta does not work right - 41 reports
  • I hate the new home button location - 42 reports

Top Issues in b3 in Reporter

Use the following link to track the top website issues in Firefox 3 Beta 3. [1]

As of right now, these include the following:

  • mail.google.com
  • www.google.com
  • mail.yahoo.com
  • labanquepostale.fr
  • msdn2.microsoft.com

We'll need to drill down into reports for the above domains to determine exactly where on the domain they're happening, however Gmail and Yahoo! Mail are pretty obvious.

Blog Posts of Note

  • http://www.dvorak.org/blog/?p=16066 Dvorak issue with missing back/forward buttons - mw22 has been able to reproduce on windows but I have not using a 2.0.0.x profile and then going to Beta 3

Scam sites charging for Firefox

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox3/TestResults/M11&oldid=84147"