QA/Firefox3/TestResults/M10

From MozillaWiki
< QA‎ | Firefox3‎ | TestResults
Jump to: navigation, search

« QA/Firefox3/TestResults

Firefox 3 M10 Test Results

Related Pages

Most of these pages are linked below in this document.

Contact Info

Build: John O'Duinn (joduinn)
QA: Tony (tchung)

Schedule

Code Freeze
12/4
Code Complete
12/7
Code Build
ETA 12/10 or 12/11, en-US & l10n
  • linux en-us & l10n 12/10
  • mac en-us & l10n 12/10
  • signed win32 en-us & l10n 12/12
QA test signoff
ETA 12/18
  • Smoketests, BFTs - 12/10 to 12/13 (3 testers)
    • Smoketest 100% Completed
    • BFTs 100% Completed
  • Mem Leaks tests - 12/13 (1 tester)
    • 100% Completed with smoketests
  • Top Extensions tests - 12/13 (2 testers)
    • 100% Completed.
  • Bug Verifications - 12/13 - 12/14 (4 testers)
    • In Progress. 65% done
  • Full Testday - 12/14 (community)
    • 100% Completed
  • L10n, Accessibility - 12/17 to 12/18 (2 testers + 1 accessibility tester)
    • Accessibility 100% Completed
    • L10n 100% done
  • Updates tests - 12/18 (3 testers)
    • 100% completed on betatest channel
    • 100% completed on releasetest channel
    • 100% completed on beta channel
  • Release notes, websites - 12/20 (2 testers)
    • 100% completed
Beta 2 Live
ETA 12/19

Release Notes

Build Location and IDs

Mac / Linux Build ID:

  • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b2) Gecko/2007121014 Firefox/3.0b2
  • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b2) Gecko/2007121014 Firefox/3.0b2
  • Build ID: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b2) Gecko/2007121016 Firefox/3.0b2

Windows Build ID:

  • Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b2) Gecko/2007121120 Firefox/3.0b2
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b2) Gecko/2007121120 Firefox/3.0b2

Strategy

  • Areas Covered:
    • Smoketests
    • BFTs
    • M10 and past focused Tests
    • Top Extensions
    • Minor Updates (beta1 and alpha 8 users -> beta 2)
    • Memory Leak Testing (handled by Tomcat on debug builds)
    • Accessibility
    • P1 L10N spotchecks (need list of supported locales)
    • M10 bug verifications
    • Beta 2 Testday
  • Areas Not Covered:
    • FFTs
    • localization (String changes) - 3rd party will cover
    • Distribution
    • Major Updates

Test Results

Types of Tests Win XP Win Vista Mac Tiger Mac Leopard Linux
Smoketest Tomcat - PASS Tomcat - PASS Tchung - PASS Marcia - PASS Tomcat - PASS
BFT Tomcat (running leak tests) - PASS Tomcat (running leak tests) - PASS. include Vista Litmus Test Suite PASS Tracy - PASS Marcia - PASS. Include Leopard Litmus Test Suite Tomcat (running leak tests) - PASS
Top Extensions Testing (see below) Abillings - PASS N/A Abillings - PASS N/A Tracy - PASS
Updates Testing (see below) Tomcat - Pass Marcia - Pass Tomcat - Pass Marcia - Pass Tchung - Pass
l10N Pri 1 locales Tomcat/Tracy (nl, fi, it) - pass - Tomcat (de, pa-IN) PASS N/A Tchung (ja-JP, es-ES, pa-IN, zh-CN) - PASS N/A Tracy (fr, zh-CN) - PASS
Accessibility (see below) marco - PASS Marco - PASS N/A N/A marco - PASS
Bug Verifications Everyone - result Everyone - result Everyone - result Everyone - result Everyone - result


Top Extensions Testing

Select a few top extensions from this list and run tests to install/update/uninstall/disable extensions. Available for download at AMO.

Or, refer to this list: Most-Used Addons

Testcases:

  • Enabling extensions and confirming web compatibility
  • Disabling or Uninstalling extensions and confirming proper messaging and behavior
  • Load about 10+ extensions, and fire them up all at once, verifying functionality and performance
  • Updating old extension versions to the latest if applicable
  • Switch profiles with different extensions, and confirm they do not get carried over.

Extensions Test Results - PASS

Updates Testing

For b2, we will be doing updates of all Alpha and Beta 1 users to Beta 2. Tests will include different locales for beta 1, but only en-US for all past versions. We will be checking partial updates, complete updates, and failed status.

Will be available on betatest, releasetest, and beta channels

  • Betatest channel - PASS
  • Releasetest channel - PASS
  • Beta channel (live) - PASS

Testcases and Results are tracked here.

Accessibility Testing

The accessibility testing team will put together a series of a11y test cases against b2, and run a check to see if there are any blocking issues. They will also verify bug fixes and call out any issues that would be critical to the release.

Current a11y Testplan and Results are tracked here. - PASS

L10N P1 Spotchecks

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Axel's Localization Status Page

Q: Will we be running minotaur automation for b2?

  • This depends on if Clint can get the Minotaur Windows VM up and running. If so, then tracy will run a full pass against the locale list.

P1 Locales to be tested:

  • UPDATE - not ready English (en-GB) Result
  • French (fr) PASS
  • Chinese (zh-CN) PASS
  • German (de) PASS
  • Japanese (ja) PASS
  • Spanish (es-ES) PASS
  • Netherlands (nl) PASS
  • Italian (it) PASS
  • Finnish (fi) PASS
  • Indian (pa-IN) PASS

I put all the beta1 available P1's through a Minotaur check against 2.0.0.9. That looked good; nothing glaringly bad jumped out at me. However, I need to get those files loaded up somewhere so Axel and Mic can have a look.

Testcases and Results are tracked here.

Bug Verifications

Bugs that are marked RESOLVED FIXED that are checked in for beta. Please verify them.

Testday

A beta 2 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found at the QMO post.

Testday results are posted here.

Detailed Results

Tomcats Results

  • Smoketests
    • Linux
      • Build ID : Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b2) Gecko/2007121016 Firefox/3.0b2
    • Windows XP (signed and unsigned)
      • Build ID: Mozilla/5.0 Windows; U; Windows NT 5.1; en-US; rv:1.9b2) Gecko/2007121120
    • Windows Vista (signed and unsigned)
      • Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b2) Gecko/2007121120 Firefox/3.0b2
      • Status : All PASS Results in Litmus
  • BFT's
    • Linux PASS
      • Build ID : Build ID : Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b2) Gecko/2007121016 Firefox/3.0b2
    • Windows XP PASS
      • Build ID : Build ID: Mozilla/5.0 Windows; U; Windows NT 5.1; en-US; rv:1.9b2) Gecko/2007121120
    • Windows Vista
      • Build id: # Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b2) Gecko/2007121120 Firefox/3.0b2 PASS
        • Bugs filed :
  • bug 407229 Breakpad fails to send on Fedora F8
  • bug 407905 Click on Download Manager Download finished notification has * problems when Pref is set to close Download Manager after download
  • bug 407929 Page Info "Block Cookies" should block cookies of the domain
  • bug 407192 Comment added to NS_ERROR_XPC_JS_THREW_STRING error in console when dragging bookmark from URL bar to toolbar
  • bug 407988 Windows Firefox 3 Beta 2 Builds broken !
  • bug 408443Update from Gran Paradiso Alpha 2, Alpha 3, and Alpha 4 -> FF Beta 2 cause [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIDocShellHistory.useGlobalHistory]

Tchung Results

  • Smoketests
    • Mac 10.4
      • Build ID : Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b2) Gecko/2007121014 Firefox/3.0b2
      • Status : PASS Results in Litmus
  • bug 408092 - Crash Reporter window is missing text for "Include URL in crash report". This is for pa-IN. minor bug.
  • bug 381357 - FR locale fails to load default RSS feeds. Still investigating if its a build problem or the feed site is down.

marcia Results

  • Smoketests
    • Mac 10.5.1
      • Build ID : Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b2) Gecko/2007121014 Firefox/3.0b2
      • Build ID: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9b2) Gecko/2007121014 Firefox/3.0b2 (spot check only, full smoketest not run)
      • Status : PASS Results in Litmus
      • Issues noted: Two crashes. Investigation revealed I hit bug 402505
  • BFT run (1/2 run on PPC mac, 1/2 on Intel Mac)
    • Mac 10.5.1
    • Build IDs: Same as above
      • Status: PASS Results in Litmus
  • Issues noted
    • At times my Error Console would fill quickly when loading certain sites. ebay.com was one example I would get this error all the time: Error: uncaught exception: [Exception... "Access to property denied" code: "1010" nsresult: "0x805303f2 (NS_ERROR_DOM_PROP_ACCESS_DENIED)" location: "<unknown>"]. The same error was generated for lots of other sites. bug 314787 is the only bug in Bugzilla I could find that included the error, but I got enough of them with sites such as espn.go.com that it was very noticeable. I filed a new bug to address this error - you can see it below in my list of bugs.
  • Bugs filed :
    • bug 407976 Tab Scrolling performances degradation with 42 tabs open
    • bug 408099 Installing Flashgot extension causes Firefox app to hang on restart
    • bug 408107 Behavior of "Recently Closed Tabs" is not consistent
    • bug 408280 Column headings do not persist during browsing session
    • bug 408281 onxblclick exception in Error Console during work in Places Organizer.
    • bug 408426 Website content not shown - "x-pointplus" is not "text/css" error
    • bug 408432 Unable to add a security exception to this page
    • Template:Bug 408852 Many instances of NS_ERROR_DOM_PROP_ACCESS_DENIED in console when loading espn.go.com
  • Bugs Noted that are problematic
    • Printing Bug where headers get cut off
    • bug 403532 Quicktime plug in content bleeds through adjacent tabs
  • Bugs Still Seen
    • bug 402965 History Information periodically disappears from History Sidebar
  • Icon Issues Noted
    • I added a comment to the Bug Tracking icon changes. The update icon in the add ons manager displays in a very poor resolution. I want to make sure it doesn't get missed as part of the visual refresh.

Marcoz results

Known issues (these should be folded into the release notes):

  • bug 407359 - Various issues with popups like the new AutoComplete popup on the location bar, the Add Bookmarks dialog (CTRL+D), the Download Manager's Info button. Some screen readers get confused by these panels announcing themselves as "menu". This issue is dealt with in the bug, but only for beta 3.
  • bug 407359 - Larry UI not keyboard accessible. This will also be dealt with in .
  • Site button has no keyboard shortcut, have to use Tools/Page Info dialog, Security and Privacy tab to get info.

Live Spot Checks for M10

Live site downloads for M10, after bits are pushed out. This includes verifying websites and release notes.

  • Windows XP -> abillings - Result
  • Windows Vista -> Tomcat - Result
  • Mac PPC/Intel-> tchung - Result
  • Mac Leopard -> Marcia - Result
  • Linux -> stephend - Result

DevTracking Bugs for M10

Refer to dev-tracking bugs at: [M10 firefox]|[M10 platform]

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox3/TestResults/M10&oldid=78359"