Fennec 5.0 Beta Test Plan

Overview

The goal of this Overall Test Plan is to lay out our testing strategy, determine who will own certain test areas and when they will do it and, finally, track the status. All of our releases are tested off of the "Fennec 5.0 Beta Catch-All Test Run" on the Litmus TCM. For more information about Fennec 5.0, take a look at the mobile team's Feature Planning page.

Project contacts

PM: Thomas Arend (tarend)
Dev Lead: Stuart Parmenter (stuart)
Build: John Oduinn (oduinn)
QA: Kevin Brosnan (kbrosnan)

Schedule

• Start Date: 5/27/2011
• End Date: 6/1/2011

Bugs in Release

• Bug 655864 - Aurora: password field is not masked immediately and shows artifacts, when typing past the end of the field
• Bug 655852 - Typing past the boundaries of the textbox will start clearing the beginning characters, instead of scrolling
• Bug 624163 - IME input does not work in rich editors with e10s
• Bug 641010 - crash [@ libdvm.so@0x35790][@ libdvm.so@0x35748 ][@ libdvm.so@0x35784 ][@ libdvm.so@0x357d0 ]
• Bug 658121 - fr locale has truncated strings
• Bug 658099 - ru locale has truncated strings
• Bug 658109 / Bug 646093 - it locale has truncated strings

Build Location and IDs

1. Go to the nightly builds folder in our ftp directory
2. Go to the "latest-mozilla-beta" folder
3. Download and install the candidate onto your Android phone/device

Test Areas

Smoketests

QA Task	Person/Group	How Often?
Smoketests	Waverley	Once
Android	Ioana
N900	Anna

Lite verification tests (or smoketests) will be first run when the builds are ready. Tests are available on litmus under the Fennec 2.0 Test Suite. In any case of failure in any Smoketest, blockers need to be raised ASAP before proceeding to the BFTs.

Tests are available on litmus under the Fennec 2.0 Test Suite.

Results details can be found in Waverley's results wiki.

Results:

Android

Results:

• Device: Samsung Google Nexus S (Android 2.3)
• Build: Mozilla /5.0 (Android;Linux armv7l;rv:5.0) Gecko/20110517 Firefox/5.0 Fennec/5.0
• Results Fennec 5.0b3 build 1 : PASS

Known bugs:

• bug 628269 - Pages are zoomed in at reload
• bug 639504 - Unable to zoom in/out a webpage on header/footer when a double tap gesture is made
• bug 625981 - places.sqlite-wal takes too much space for Fennec
• bug 656594 - Sync Now button is active even if the Sync is in progress

Note :

• no update test made - updates will be made only through the Android Market.

Maemo

• Device: Nokia N900 (Maemo GTK)
• Build: Mozilla/5.0 (Maemo; Linux armv7l; rv:5.0) Gecko/20110527 Firefox/5.0 Fennec/5.0
• Results Fennec 5.0b2 build 3 : PASS
• Issues:
  • Firefox Beta could not retrieve add-on. I can only install add-ons from AMO if "extensions.checkCompatibility.5.0=false"
• Reported bugs:
  • bug 661098: After sharing a webpage using facebook, the "Share link page" does not close
  • bug 661111: [Maemo] No JPAKE authentication code. Failed to connect to the server error

BFTs for 5.0 Features

QA Task	Person/Group	How Often?
BFTs	Ioana, Anna, Nicolae	Once

BFTs are a more comprehensive set of litmus tests that cover functional testing of features that have landed on Fennec 2.0. Tests are available on litmus under the Fennec 2.0 Test Suite.

Results details can be found in Waverley's results wiki.

Notes: Fennec 5.0b3 build 1 BFTs notes

Exploratory Testing

QA Task	Person/Group	How Often?
Exploratory Testing	QA team/Waverley	Daily

Exploratory testing around the bug fixes for this beta. Focus on IME keyboards, input forms and java crashers fixed during this beta cycle.

• focus on typing, deleting, overflowing textboxes to make sure we have not regressed anything
• Bug 624163 fixes a problem in rich editors (ehterpad and google docs are good test tests) with IME keyboards
• Java crasher read the bug for STR

Bug Verification

QA Task	Person/Group	How Often?
Bug Verification	Martijn (mw22)	Daily

All bugs fixed for this beta cycle should be verified fixed with a Firefox 5 beta 3 build.

L10N

QA Task	Person/Group	How Often?
L10N Litmus tests	nhirata	Daily

Review locales for broken or missing strings.

Test Run	ALL	bug 644587, bug 656686, bug 658054, bug 660388
	French	bug 660375,bug 660379
	Japanese	bug 661007, bug 661014, bug 660995
	Russian	bug 661277

Basically, bugs are about truncations in notifications and preferences in portrait view.

Note: http://mxr.mozilla.org/mozilla-central/source/mobile/locales/maemo-locales are all the locales that are suppose to be in the multilanguage pack.

Crasher Triage

QA Task	Person/Group	How Often?
Crasher Triage	nhirata	Daily

Socorro is Mozilla's primary web interface for triaging crash reports sent in by our user base. Due to the relatively large amount of changes coming in on a nightly basis, there is a need to check for crash reports coming in for every build. The owner of this task will triage through top crashers on a daily basis, file bugs if necessary and find steps to reproduce for those crashers if possible.

Results: https://wiki.mozilla.org/QA/Fennec/crash_rep_25may

Device Compatibility

QA Task	Person/Group	How Often?
Device Compatibility	AaronMT	Once

As part of our testing strategy, we will want to verify device compatibiltiy checking for regressions or major issues in Firefox by running through our compatbility criteria spreadsheet . Mobile users can also contribute towards device qualifying by submitting feedback via our untested device survey.

Website Compatibility

QA Task	Person/Group	How Often?
Website Compatibility	Aaronmt	Once

As part of our testing cycle we want to cover and include a general testing process that focuses on website compatbility with sites viewed in mobile Firefox. We want you to use mobile Firefox to assist us in identifying any noticeably major issues found in websites. Issues can range from pages that do not look right, layout and positioning problems, interactivity issues, font issues and pages that grealy differ from browser to browser and what you would expect. You can help us by completing a website exploration form here, and following the companion instructional test-plan.

Web Properties Review

QA Task	Person/Group	How Often?
Web Properties Review	kbrosnan	Once

Examine the copy for the staged websites that are part of the release promotion. We review the pages for technical errors in the copy. Additionally we help flag bugs and issues for notice in the release notes.

Market Install

QA Task	Person/Group	How Often?
Web Properties Review	kbrosnan	Once

Test to make sure upgrades and installs work from the market. Check that searching for the product returns the expected results.