QA/Thunderbird3/TestInstructions/Beta2

Thunderbird 3 Beta 2 Release Candidate Test Instructions

Overview

Release Candidate builds must be tested before releasing to the larger developer/public population. You, the volunteers and QA team, help ensure they receive a quality release by doing rigorous testing.

As always, if you need help please ask on IRC and in the newsgroups.

Several types of tests are available, and you can choose which to run based on your available time. BFT, L10n, and Mac platform tests are our greatest need. If you have tested before, we hope you attempt more in-depth testing this time around. Also, if you can't help before the last day of testing, please stop back later and run tests - additional test data is always helpful, and you will gain experience in testing. Thank you for your interest, and we hope, your enthusiastic participation.

Types of tests:

• BFT (Basic Functionality Tests) - touch most functional areas. If you do not have time to do BFT tests we hope that you ...
• Smoketest - tests a core group of functions (a much reduced BFT)
• Spot check - a visual "look see", splap it in and look around
• L10n (localization) - Litmus tests specific for localized builds. If you can test a localization, we would appreciate you doing so.

You are not required to run a full test set to completion. It is OK if you start testing but do not have much time, or think you might run out of time. We'd rather have you do a short test than do no testing at all. We mostly use litmus to record test results as you go through the list - you can stop whenever you need ... and return later to do more. Leave the test session open in the browser, or remember which test you did last, then start at the next step after starting up the litmus test process when you return.

Details

Important

• Use only specified builds - not nightly or previous alpha or beta build.
• File a bug whenever you encounter a problem, and mark as blocking bug 477222 so others are notified of problems found during QA.
• If litmus testcase:
  • is not run (you skipped it) - please do not comment, nor mark passing or failed.
  • fails
    1. file a bug (if one doesn't already exist), set blocking bug 477222
    2. put bug# in the litmus test report
  • is unclear
    1. set status to unclear, do not mark it passing
    2. please comment with steps and wording you think are more clear
• Review sections 1.1 to 1.6 of release information, especially the release notes.
• Contacts for getting help and providing feedback

Builds

Builds to be tested are listed in the release details. Please use a localized version if possible.

BFT

Basic Functionality Tests is organized into subgroups listed below. To shorten your testing time you can focus on a few subgroups as follows:

• * indicates priority items - if you have the resource required please do * groups before others
• All indicates items we hope everyone will test
• AL and MZ - please run the test group if your last name/surname is in the range A-to-L or M-to-Z. If you have time feel free to do all tests. If not, skip the ones not assigned to your name.

If you start running out of time and think you won't be able to return (or have time left after doing the assigned subgroups), you might return to the litmus menu "Select a Testgroup and Subgroup to Test" and pick one or two subgroups that show the least amount of testing coverage.

Starting with litmus:

1. You've already read the Beta 2 release notes, of course.
2. Login to litmus.mozilla.org (create an account if you don't have one).
3. Visit BFT tests.
4. Complete all information about your setup - buildid, platform, OS, and locale. You must relpace buildid - the automatic value in the field is incorrect. Click the blue-circle-i in litmus for help.
5. Click "submit configuration".
6. You're off and running! Expect 2-3 hours to do half the list (doing * items first).

Subgroup assignments for BFTs (list order per litmus)
 
  All-Install, shutdown, uninstall
  All-Vista Integrated Search (MS-Vista users only)
   AL-Migration (from other mail systems)
   AL-Import (from other mail systems)
   AL-Window configuration
   AL-Toolbars and menus
  All-Account settings & Preferences (Options)
   *  IMAP accounts
   *  POP accounts (exclude Global Inbox)
   *  Global inbox
  All-Mail composition
   AL-Spell checker
   AL-RSS account & subscriptions
   AL-Newsgroups
  All-Navigating and displaying messages
   MZ-Downloading and saving
   MZ-Image blocking
   MZ-Return receipts
   *  Proxies
   MZ-Offline, disk space
  All-Moving, copying, deleting messages
  All-Views and labeling messages
   AL-Message filters
   AL-Message search
  All-Address search
   MZ-Saved Search folders
   MZ-Message Grouping
   MZ-Message Aging
   MZ-Quicksearch
  All-Address books
   AL-Junk mail
   AL-Extensions (not for beta 1, reactivate test later)
   MZ-Theme management (not for beta 1, reactivate test later)
   MZ-Help
   MZ-Printing
  All-Master Passwords & password management
   *  Phishing, spoof detection
   *  Secure connections
   *  Digital signing, encrypting messages
   AL-Software Update

smoketest

Access litmus using the "Getting started" steps above for BFT, but use the smoketest URL: https://litmus.mozilla.org/run_tests.cgi?test_run_id=37. Should take about 1.5 hours. (Ignore "Subgroup assignments" listed above - it is not relevant)

Spot check

Spot check takes little time because is just a casual visual inspection and does not require the use of litmus. Install, poke around, inspect, test and try what you can, then visit spot check results and update the wiki.

locale

We need testers to smoketest or BFT L10n builds (per tests described above), then update the wiki at Thunderbird3/TestResults/Beta2/L10N to help us track which locales have been tested.

We also have L10n trunk baseline tests focusing on language string correctness and UI elements working and looking correct. Your testing results are recorded in litmus.