Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name	Title	Department	Approval Date	Method
Cornel Ionce	QA Manager	Product Integrity	Date	Email
Patrick Brosset	Engineering Manager	Engineering	Date	Email
Bryan Clark	EPM	Product Management	Date	Email

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date	Version	Author	Description
05/22/2017	1.0	Iulia Cristescu	Created first draft
06/20/2017	2.0	Iulia Cristescu	Updated most of the sections
06/23/2017	3.0	Iulia Cristescu	Updated Approvals, Risk Assessment and Coverage, Test Areas and Sign Off Criteria sections

Overview

Purpose

DevTools Go Faster is about shipping DevTools as an on-demand add-on to the Firefox users. 99% of Firefox users don't use DevTools and for the users who do the 90% of them are on the Release channel. Converting DevTools to an add-on allows removing DevTools from most Firefox installs and ship DevTools to the developers faster than ever.

Scope

This wiki details the testing that will be performed by the project team for the DevTools Go Faster project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

What will be tested
How testing will be performed

Ownership

Stakeholders

Product Manager - Bryan Clark
Overall Product - Jeff Griffiths

Engineering

Engineering Manager - Patrick Brosset
Overall Engineer - Joe Walker
Engineering - Alexandre Poirot
Engineering - Honza
Engineering - Julian Descottes

QA

QA Manager - Cornel Ionce
PM for QA team - Florin Mezei
TL for QA team - Andrei Vaida
Leading QA efforts - Iulia Cristescu
Leading QA efforts (peer) - Alexandru Simonca

Testing summary

Scope of Testing

In Scope

DevTools lets you examine, edit, and debug HTML, CSS, and JavaScript on the desktop and on mobile.
DevTools will become a Mozilla extension that can be installed on demand by the user (using an UI that detects when someone needs DevTools) and will be updated outside of the Firefox's release cycle.
For existing Firefox DevTools users there should be no noticeable change in their daily experience.
DevTools Shim is responsible for installing the DevTools add-on as needed. The Shim will auto-install the add-on for existing DevTools users by default. The Shim is listening to the DevTools keyboard shortcuts and provides install methods at other various entry points like about:debugging. When new users use a DevTools keyboard shortcut Firefox will prompt them, asking if they want to install DevTools. If the the answer is yes, the DevTools add-on will be installed and will update itself in the background from that point on.

The testing efforts will be invested on the following areas:

previous usage detection;
add-on installation work flow and its integration with the current browser functionalities and UI;
DevTools basic and advanced functionality and usability, according to the existing requirements.

Out of Scope

The mobile implementation and testing are not targeted
Storage inspector, StyleEditor, Canvas Debugger, Shader Editor, Performance, Memory, Settings are not covered

Requirements for testing

Environments

Testing will be performed across platform (both x86 & x64 infrastructures), targeting:

Windows 10
Windows 7
Ubuntu 16.04
macOS 10.12

Test Strategy

Risk Assessment and Coverage

ID	Description / Threat Description	Covered by Test Objective	Magnitude	Probability	Priority	Impact Score
1	Compatibility across all available channels	Testing should cover all the Firefox channels (from Release to Nightly)	2-Moderate	2-Possible	3-High	4
2	Differences depending on channels (built in vs. add-on) and profiles (already opened DevTools vs. never opened DevTools)	Testing should also regard the previous usage detection process and the differences induced by the installation work flow	3-High	3-Almost Certain	3-High	12
3	Localization/RTL	Testsuite will include RTL/non-RTL differentiation	2-Moderate	2-Possible	3-High	12
4	Platform specific issues	Most users are using Windows but the team mostly works on OSX/Unix, so there is a risk for Windows only regressions. Therefore the testing process will assure full platform coverage	3-High	3-Almost Certain	3-High	12

Values:

Magnitude: 1- Low , 2-Moderate, 3-High

Probability: 1-Unlikely, 2-Possible, 3-Almost Certain

Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref	Function	Test Objective	Evaluation Criteria	Test Type	Owners
1	Previous usage detection	Verify if DevTools previous usage state is properly detected on Beta and Release builds and if the corresponding install process is initiated	If DevTools was previously used, the addon is automatically downloaded and installed. If the user never tried DevTools, it will go through the install process	Manual	Release QA
2	Install entry points	Verify if the entry points for users to install DevTools are captured	The corresponding install work flow is initiated when DevTools are launched, depending on the previous usage status	Manual	Release QA
3	Install work flow	Verify if the installation process is properly executed	Depending on the captured entry point, the corresponding install page is opened in a new tab and the installation process is successfully completed	Manual	Release QA
4	DevTools functionality and usability	Verify if all DevTools functionalities are working	Toolbox, Inspector, Console, Debugger, Netmonitor and all the related tools are fully functional and no regressions are triggered by the new install work flow	Manual	Release QA

Builds

DevTools Go Faster will be tested on the following builds:

Nightly - waiting for custom builds provided by Engineering
DevEdition - starting with the build from
Beta - starting with
Release - starting with

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	05/01/2017
Study documentation/specs received from developers	05/20/2017
QA - Test plan creation	05/22/2017
QA - Test cases/Env preparation	06/06/2017
QA - Nightly Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	TestRail
Test case execution	TestRail
Bugs management	GitHub & Bugzilla

Status

Overview

Released to Nightly starting with
Merged to beta starting with
Merged to Release starting with

References

List and links for specs:

Testcases

Test Areas

Test Areas	Covered	Details
Private Window	Yes
Multi-Process Enabled	Yes
Multi-process Disabled	No
Theme (high contrast)	No
UI
Mouse-only operation	Yes
Keyboard-only operation	Yes
Display (HiDPI)	No
Interaction (scroll, zoom)	Yes
Usable with a screen reader	No	e.g. with NVDA
Usability and/or discoverability testing	Yes	Is this feature user friendly
RTL build testing	Yes
Help/Support
Help/support interface required	Yes	Make sure link to support/help page exist and is easy reachable.
Support documents planned(written)	Yes	Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected	Yes
Does sync work across upgrades	No
Requires install testing	Yes	separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding	Yes	Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing	No	yes/no options, add comment with details about who will lead testing
Enterprise		Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration	No
Network proxies/autoconfig	No
ESR behavior changes	No
Locked preferences	No
Data Monitoring
Temporary or permanent telemetry monitoring	No	List of error conditions to monitor
Telemetry correctness testing	No
Server integration testing	No
Offline and server failure testing	No
Load testing	No
Add-ons		If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required?	No
Comprehensive API testing	No
Permissions	No
Testing with existing/popular addons	No
Security		Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review	No
Privilege escalation testing	No
Fuzzing	No
Web Compatibility		depends on the feature
Testing against target sites	Yes
Survey of many sites for compatibility	Yes
Interoperability		depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.	No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS	No
Interaction of this feature with other browser features	Yes

Test suite

Full Test suite - Testrail  
Smoke Test suite - Testrail
Regression Test suite - Link with the tests - if available/needed

Bug Work

Reference bugs

Sign off

Criteria

Checklist

All test cases should be executed
Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

Link for the tests run
- Full Test suite, link to TestRail - Tests Runs and Results link
- Daily Smoke, if needed/available
- Regression Test suite, if needed/available

Merge to Beta Sign-off
List of OSes that will be covered by testing

Link for the tests run
- Full Test suite

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff		Email to be sent
QA Nightly - Full Testing
QA pre-Beta Signoff		Email to be sent
QA Beta - Full Testing
QA pre-Release Signoff		Email to be sent

QA/DevTools Go Faster

Contents

Overview

Purpose

Scope

Ownership

Testing summary

Scope of Testing

In Scope

Out of Scope

Requirements for testing

Environments

Test Strategy

Risk Assessment and Coverage

Test Objectives

Builds

Test Execution Schedule

Testing Tools

Status

Overview

References

Testcases

Test Areas

Test suite

Bug Work

Sign off

Criteria

Results

Checklist

Navigation menu

Search