Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date	Version	Author	Description
06/27/2017	1.0	Justin Williams	Created first draft
08/14/2017	1.1	Justin Williams	Updated Draft

Overview

Purpose

When the stub installer runs it detects if you have an old Firefox version installed. If you do you are considered an updated user. If you do not and you have old profile data you are considered a new user. The stub installer will then prompt you with the proper screen (new or updated user) and ask if you want to clean your profile data (this is checked by default). This is done to determine which Photon Onboarding Tour is shown to you once Firefox is installed.

See: for more info

Ownership

Engineering lead
  Tim Chien

Engineering Team
  Matt Howell

UX Team: 
  Michael Verdi
  Bryant Mao

QA 
  Justin Williams

Testing summary

Scope of Testing

In Scope

The Firefox Stub Installer and the Profile Folder.

Out of Scope

Everything outside of the profile folder and the stub installer is out of scope. Fennec is also out of scope.

Requirements for testing

Environments

Windows 7 x64: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
Windows 10 x64: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz, 3601 Mhz
Windows 8.1 x64: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz, 3601 Mhz

Test Strategy

Risk Assessment and Coverage

ID	Description / Threat Description	Covered by Test Objective	Magnitude	Probability	Priority	Impact Score
RAC-1	Refresh a profile the user does not want to be refreshed.	TO-1	2-Moderate	1-Unlikely	3-High	6
RAC-2	Show the incorrect prompt.	TO-2	2-Moderate	2-Possible	3-High	12
RAC-3	Could potentially break the installer. Since the new code all runs before the installation starts, if any of it crashes/hangs/otherwise fails unrecoverably, then no installing can happen.	TO-3	3-High	1-Unlikely	3-High	9
RAC-4	Currently no automation test coverage		3-High	2-Possible	2-Medium	12

Values:

• Magnitude: 1- Low , 2-Moderate, 3-High

• Probability: 1-Unlikely, 2-Possible, 3-Almost Certain

• Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

• An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
• An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
• An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref	Function	Test Objective	Evaluation Criteria	Test Type	RAC	Owners
1	Proper refresh/non-refresh is determined by user	Refresh/Do not refresh a profile the user wants	This action is determined by the user	Manual	RAC-1	Eng Team
2	Verifying that the Installer works correctly by showing the correct prompt when determining if the user is a new user/updated user	Correct prompt is shown.	The installer is not broken and works as expect. New/Updated users are shown the proper prompt	Manual	RAC-2, RAC-3	Eng Team
3	Verify the changes to the installer are not identifiable to the user.	Installer is smooth and user friendly.	The installer runs smooth and installs Firefox as expected	Manual	RAC-3	Eng Team
4	Verify the correct pings are being sent to the server.	Ping information is being sent and received correctly.	Ping data is shown on about:telemetry and STMO	Manual		Eng Team

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	03/30/1017	11/14/2017
Study documentation/specs received from developers	08/08/2017	11/14/2017
QA - Test plan creation	06/27/2017	11/14/2017
QA - Test cases/Env preparation	08/15/2017	11/14/2017
QA - Nightly Testing	08/18/2017	08/24/2017
QA - Beta Testing
Release Date	11/14/2017	11/14/2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	Refresh Test Case Creation
Test case execution	Refresh Test Case Execution
Bugs management	Refresh Meta Bug
Trello	Refresh Trello Card

Status

Overview

Track the dates and build number where feature was merged to Release/Beta

Channel	Date feature was merged to channel	Build number
Nightly	08/11/2017	20170811100330
Beta
Release

Testcases

Test Areas

Test Areas	Covered	Details	Reviewed By
UI
Mouse-only operation	Y	used by majority of users when using the stub installer	Matt Howell
Display (HiDPI)	Y	used by a large chunk of users	Matt Howell
Usable with a screen reader	Y	needs to be compatible with different accessibility applications	Matt Howell
RTL build testing	Y	needs to be compatible with all locales	Matt Howell
Install/Upgrade
Feature upgrades/downgrades data as expected	Y	feature is dependent on upgrading	Matt Howell
Affects first-run or onboarding	Y	determines the tour based off of previous update version or profile information	Matt Howell

Sign off

Criteria

Checklist

• All test cases should be executed
• Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing
Win 7, 8.1, & 10

• Link for the tests run
  • Full Test suite, link to TestRail - Tests Runs and Results Refresh Test Run

Merge to Beta Sign-off
List of OSes that will be covered by testing
Win 7, 8.1, & 10

• Link for the tests run
  • Full Test suite

Merge to Release Sign-off
List of OSes that will be covered by testing
Win 7, 8.1, & 10 (x64 & x86)

• Link for the tests results

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)	[DONE]
Testing Infrastructure setup	[DONE]
Test Plan Creation	[DONE]
Test Cases Creation	[DONE]
Automation Coverage	n/a
Performance Testing	n/a
All Defects Logged	[DONE]
Critical/Blockers Fixed and Verified	[DONE]
Metrics/Telemetry	n/a
Basic/Core functionality Nightly testing	[DONE]
QA mid-Nightly Signoff	[DONE]	Email sent
QA Nightly - Full Testing	[DONE]
QA pre-Beta Signoff	[DONE]	Email sent
QA Beta - Full Testing	[DONE]
QA pre-Release Signoff	[DONE]	Email sent

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name	Title	Department	Approval Date	Method
Lawrence Mandel / RyanVM	QA Reviewer	Product Integrity		Email
Matt Howell	Software Engineer	Engineering	09/07/2017	Email
Romain Testard	PM	Product Management	09/07/2017	Email