QA/Firefox3/TestResults/RC2

From MozillaWiki
< QA‎ | Firefox3‎ | TestResults
Jump to: navigation, search

« QA/Firefox3/TestResults

Firefox 3 RC2 Test Results

Related Pages

Links to separate test results. Some are referenced further below.

Project Release Tracking Page

Contact Info

PM: Mike Beltzner (beltzner)
Build: John O'Duinn (joduinn)
QA: Tony (tchung)

Schedule

Code Freeze
5/28
Code Build
5/29
QA test signoff
ETA 6/4
  • Smoketests PASS
  • BFTs PASS
  • RC2 and past focused Tests (ongoing) PASS
  • Update snippets from Alpha X, Beta X -> RC2 betatest, PASS releasetest, PASS, rc1->rc2 beta/release, result
  • P1 L10N spotchecks PASS
  • Top plugins PASS
  • RC2+ bug verifications (ongoing) result
  • RC2 Testday PASS
  • Release notes, Websites, live tests result

Release Notes

See list of release notes here

Build Location and IDs

*Note: Only Mac was respun for a second build. Windows and Linux were not affected by the bug in question.

Windows

  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0

Mac

  • RC2 Build 1
  • Leopard
    • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9) Gecko/2008052903 Firefox/3.0
  • RC2 Build 2
  • Leopard
    • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9) Gecko/2008053008 Firefox/3.0
  • RC2 Build 2
  • Tiger
    • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9) Gecko/2008053008 Firefox/3.0

Linux

  • Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9) Gecko/2008052912 Firefox/3.0

Test Coverage

  • Areas Covered:
    • Smoketests
    • BFTs
      • Use test profiles stored here.
    • Focused Tests
    • Update snippets from Alpha X, Beta X, RC1 -> RC2
    • P1 L10N spotchecks
    • Top Plugins (special coverage on bug 434914 for mac)
    • Bug verifications (RC2+ bugs)
    • RC2 Testday
  • Areas Not Covered:
    • FFT
    • Top Sites
    • Top Themes
    • Top Extensions
    • Accessibility Tests (litmus and automation)
    • Memory Leak Testing (handled by Tomcat on debug builds)
    • Bc's topsite spidertests against jemalloc/malloc builds
    • localization (String changes) - Localizers to cover
    • Distribution
    • Major Updates

Test Results

Types of Tests Win XP Win Vista Mac Tiger Mac Leopard Linux
Smoketest Tomcat - PASS juanb - PASS Tracy - PASS marcia - PASS Tomcat (Build 2) PASS Tracy - PASS
BFT Tomcat - PASS Juan - PASS Hasham/Marcia - PASS Tomcat - PASS Build 2 BFT PASS Tracy - PASS
Updates Testing Tchung - Betatest PASS, Releasetest PASS, Beta result Juanb - Betatest PASS, Releasetest PASS, Beta result Tracy - Betatest PASS, Releasetest PASS, Beta result Marcia - Betatest PASS, Releasetest PASS, Beta result Abillings - Betatest PASS, Releasetest PASS, Beta result
l10N Pri 1 locales Tchung - PASS N/A Tchung - PASS N/A Tchung - PASS
Top Plugins Hasham - PASS N/A Tracy - PASS N/A Tchung - PASS
Bug Verifications Everyone - result Everyone - result Everyone - result Everyone - result Everyone - result
Live Checks & Download Links Tomcat - result Juanb - result Tracy - result Marcia - result Abillings - result

Updates Testing

We will be doing updates of all Alpha and Beta users to RC2. We will be checking partial updates, complete updates, and failed status.

Will be available on betatest, releasetest, beta, and release (rc1 only) channels

  • Betatest channel - PASS
  • Releasetest channel - PASS
  • Beta channel (live) - PASS
  • Release channel (rc1 only) - PASS

Testcases and Results are tracked here.

L10N P1 Spotchecks

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Automation inspection: supported locales will be ran through Minotaur, and results will be reviewed by axel and mic.

Testcases and Results are tracked here - PASS

Top Plugins Testing

Query the top plugins supported for the web, and verify they are compatible with Firefox 3. See AMO list for recommended plugins.

Testcases

  • On a clean environment, install plugins
  • Test updating of old plugins
  • Disable plugins in Addons Manager, verify plugins stop working
  • Uninstall plugins, verify websites request a plugin to download
  • Verify websites with popular plugins:

Additional Plugin litmus cases can be found here.

Bug Verifications

Bugs that are marked RESOLVED FIXED that are checked in for RC. Please verify them.

RC2+ Bugs - http://tinyurl.com/47wglk
P1/P2 Firefox bugs - http://tinyurl.com/3bfgoa
P1/P2 Platform bugs - http://tinyurl.com/354n48

Testday

A RC2 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found on QMO.

Testday results are posted here.

Live Spot Checks

Check live site download links on all platforms. Please cover en-US, and 2 or 3 additional Tier 1 locales as well. Pay attention to the following areas:

  • Check for signed installer and applications (windows only)
  • Firstrun page, whatsnew page, Home page, Search plugin
  • Install, uninstall
  • General spotchecks and browser functionality
  • Confirm Build IDs

OS assignments:

  • XP - Tomcat - result
  • Vista - Juanb - pass
  • Mac Tiger - Tracy - PASS
    • Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; it; rv:1.9) Gecko/2008053008 Firefox/3.0
    • Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; da; rv:1.9) Gecko/2008053008 Firefox/3.0
    • Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; sv-SE; rv:1.9) Gecko/2008053008 Firefox/3.0
  • Mac Leopard - Marcia - result
  • Linux - Abillings - PASS

Detailed Results

Smoketests

  • bug 430737 – Update snippets sometimes don't propagate to the entire cluster. this happens in 2.x releases also. This delayed our website release for almost 1 1/2 hours.

Tracy's Results

  • Mac (Tiger) - Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9) Gecko/2008052903 Firefox/3.0
  • Status: PASS results in Litmus
  • Linux (FC7) - Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9) Gecko/2008052912 Firefox/3.0
  • Status: PASS results in Litmus

marcia's Results

  • Mac (Leopard) - Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9) Gecko/2008052903 Firefox/3.0
  • Status: PASS results in Litmus
  • Confirmed Bug 436210 which was seen during testing.

Tomcat's Results

  • Mac Leopard Build 2: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9) Gecko/2008053008 Firefox/3.0
  • Status: PASS results in Litmus
  • Windows XP : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0
  • Status: PASS results in Litmus

juanb's results

  • Vista Smoketest: PASS
    • Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0

BFT

hasham/marcia Results

  • marcia: Mac 10.4 - Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9) Gecko/2008053008 Firefox/3.0 and Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9) Gecko/2008053008 Firefox/3.0
  • Status: PASS
  • New bugs filed during testing
    • Bug 436935 NS_ERROR_NOT_IMPLEMENTED in nsLoginManager
  • Bugs Confirmed during testing
    • Bug 435863 Custom global shortcuts in Mac OS X do not work correctly with Firefox.
  • Bugs Seen during testing
    • Bug 436497 Google favicon needs to be updated.
    • Bug 435485 Can't reliably use "Move" command in Organize Bookmarks

Tomcat's Results

  • Mac 10.5 Build 1 - Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9) Gecko/2008052903 Firefox/3.0
  • Status: PASS results in Litmus
  • Confirmed Bug 432832 during the tests
  • Mac 10.5 Build 2- Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9) Gecko/2008053008 Firefox/3.0
  • Status: PASS results in Litmus


  • Windows XP : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0
  • Status: PASS results in Litmus

juanb's restuls

  • BFT on Vista: 90%
    • Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0
  • Notes:

tracy's results

  • Linux (FC7) - Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9) Gecko/2008052912 Firefox/3.0
  • Status: PASS results in Litmus
    • bug 436521 loop in plugin finder service says Flash is installed when it isn't

Accessibility testing

  • Windows: Pass results in Litmus.
  • Linux: Pass with the exception of testcase 5391 which is documented in bug 435179. Should be relnoted, nominated for fixing and inclusion in a 3.0.x release.
  • Verified bug 434002 using ChatZilla 0.9.82.1, which exposed this bug reliably for me previously.

L10n Testing (tchung)

  • Filed bug 436589 – Search plugin order differs on Linux / Windows versus Mac.

Hendrix Feedback =

  • Bugs Filed by marcia as a result of Hendrix review:
    • Bug 436568 [Vista] DEP may impede the browsing experience on Vista
    • Bug 436356 Bullet lists does not display correctly, works in FF 2
    • Bug 436139 Top part of website renders too high
    • Bug 437115 Site lays out properly using FF 2 but not FF3
  • Bugs with a lot of comments in hendrix
    • Bug 436686 On occasion Flash video will start with no sound, then stop after 2-3 seconds
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox3/TestResults/RC2&oldid=95600"