QA/Streamlined Stub Installer
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
06/02/2017 | 1.0 | Grover Wimberly IV | Created first draft |
Contents
Overview
Purpose
- Check Stub installer icon is per provided asset (not yet landed on Nightly)
- Check Stub installer name is “Firefox installer”
- Check caption logo is the release channel logo
- Check that pre-Win7 Windows releases get an error message (these users would get ESR from the download page)
- Check behavior on dismissal of UAC prompt (install to non privileged directory inside user Home directory) when admin user
- Check behavior on acceptance of UAC prompt (install to default directory) when admin user
- Check behavior on dismissal of UAC prompt (install to non privileged directory inside user Home directory) when non admin user
- Check behavior on acceptance of UAC prompt (install to default directory) when non admin user
- Check install wizard experience:
- The caption should change every 20 seconds, disappearing for .5 seconds before cycling to the next message, returning to the first when the third is done:
- Check that clicking OK redirects to the full installer download page. Cancel just exits the installer.
- Completing install opens Firefox if Firefox not already opened
- Completing install does not open Firefox if Firefox was already open
- If installation done on Win7+ 64 bit OS with 2GB+ of memory then install 64 bit Firefox (not yet landed on Nightly)
- Otherwise install 32 bit Firefox (if less than 2Gb or 32 bit OS) (not yet landed on Nightly)
- Check Installation directory
Scope
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- Functionality of the build on x32 and x64 bit OS with Administrator and Guest accounts
- User interface
Ownership
Softvision Las Vegas QA Team
Testing summary
Scope of Testing
In Scope
Install Experience
Out of Scope
N/A
Requirements for testing
Environments
Windows 7+ Pre-Windows 7 (To ensure functionality is disabled)
Channel dependent settings (configs) and environment setups
Nightly
Normal Installer
Beta
text
Post Beta / Release
text
Test Strategy
Risk Assessment and Coverage
ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
---|---|---|---|---|---|---|
RAC-1 | Risk description 1 | TO-1 | 2-Moderate | 1-Unlikely | 3-High | 6 |
RAC-2 | Risk description 2 | TO-1 | 3-High | 3-Almost Certain | 3-High | 27 |
RAC-3 | Risk description 3 | TO-2 | 2-Moderate | 2-Possible | 3-High | 12 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
Ref | Function | Test Objective | Evaluation Criteria | Test Type | RAC | Owners |
---|---|---|---|---|---|---|
1 | Name of the feature or sub-function being tested | The objective the test is trying to demonstrate | The criteria that will be evaluated to demonstrate the test is successful | Manual/ Automation/ Regression/ Performance/ Usability/ Security/ Telemetry | RAC-1, RAC-2, RAC-3 | Eng Team |
2 | Repeat for each feature/sub-function | |||||
3 |
Builds
This section should contain links for builds with the feature -
Nightly (Archive) - https://archive.mozilla.org/pub/firefox/nightly/2017/01/2017-01-13-04-02-04-oak/firefox-53.0a1.en-US.win32.installer-stub.exe Nightly (Public) - https://www.mozilla.org/en-US/firefox/channel/desktop/
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | ||
Study documentation/specs received from developers | ||
QA - Test plan creation | ||
QA - Test cases/Env preparation | ||
QA - Nightly Testing | May 29, 2017 | June 2, 2017 |
QA - Beta Testing | ||
Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail/ Google Docs |
Test case execution | TestRail |
Bugs management | Bugzilla |
Status
Overview
May-June 2017 - Nightly
References
- List and links for specs
List and links for available specs - documents, user stories, specifications
- Meta bug
Testcases
Test Areas
The installer stub.
Test suite
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project link Smoke Test suite - Link with the tests - if available/needed. Regression Test suite - Link with the tests - if available/needed.
Bug Work
Meta bug: 12345 - bug summary
Logged bugs ( blocking 12345 )
ID | Priority | Component | Assigned to | Summary | Status | Target milestone |
---|---|---|---|---|---|---|
15069 | P3 | Networking | jefft | File type url not correctly parsed and created | VERIFIED | M13 |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Bug fix verification
ID | Priority | Component | Assigned to | Summary | Status | Resolution | Target milestone |
---|---|---|---|---|---|---|---|
15069 | P3 | Networking | jefft | File type url not correctly parsed and created | VERIFIED | FIXED | M13 |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Sign off
Criteria
Checklist
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Merge to Beta Sign-off
Windows XP, 7, 8, 10
Test Rail - https://testrail.stage.mozaws.net/index.php?/plans/view/4418
Pre-Release Sign-off (2017-07-21)
Test Rail - https://testrail.stage.mozaws.net/index.php?/plans/view/5085
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | ||
Testing Infrastructure setup | Completed | |
Test Plan Creation | Completed | |
Test Cases Creation | Completed | |
QA Nightly - Full Testing | Skipped | |
QA pre-Beta Signoff | Sign-Off June 2, 2017 | |
QA Beta - Full Testing | TBA | |
QA pre-Release Signoff | TBA |