QA/DevTools Go Faster

From MozillaWiki
< QA
Jump to: navigation, search

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Cornel Ionce QA Manager Product Integrity Date Email
Patrick Brosset Engineering Manager Engineering Date Email
Bryan Clark EPM Product Management Date Email


Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
05/22/2017 1.0 Iulia Cristescu Created first draft
06/20/2017 2.0 Iulia Cristescu Updated most of the sections
06/23/2017 3.0 Iulia Cristescu Updated Approvals, Risk Assessment and Coverage, Test Areas and Sign Off Criteria sections

Overview

Purpose

DevTools Go Faster is about shipping DevTools as an on-demand add-on to the Firefox users. 99% of Firefox users don't use DevTools and for the users who do the 90% of them are on the Release channel. Converting DevTools to an add-on allows removing DevTools from most Firefox installs and ship DevTools to the developers faster than ever.

Scope

This wiki details the testing that will be performed by the project team for the DevTools Go Faster project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Stakeholders

  • Product Manager - Bryan Clark
  • Overall Product - Jeff Griffiths

Engineering

QA

Testing summary

Scope of Testing

In Scope

DevTools lets you examine, edit, and debug HTML, CSS, and JavaScript on the desktop and on mobile.
DevTools will become a Mozilla extension that can be installed on demand by the user (using an UI that detects when someone needs DevTools) and will be updated outside of the Firefox's release cycle.
For existing Firefox DevTools users there should be no noticeable change in their daily experience.
DevTools Shim is responsible for installing the DevTools add-on as needed. The Shim will auto-install the add-on for existing DevTools users by default. The Shim is listening to the DevTools keyboard shortcuts and provides install methods at other various entry points like about:debugging. When new users use a DevTools keyboard shortcut Firefox will prompt them, asking if they want to install DevTools. If the the answer is yes, the DevTools add-on will be installed and will update itself in the background from that point on.

The testing efforts will be invested on the following areas:

  • previous usage detection;
  • add-on installation work flow and its integration with the current browser functionalities and UI;
  • DevTools basic and advanced functionality and usability, according to the existing requirements.

Out of Scope

  • The mobile implementation and testing are not targeted
  • Storage inspector, StyleEditor, Canvas Debugger, Shader Editor, Performance, Memory, Settings are not covered

Requirements for testing

Environments

Testing will be performed across platform (both x86 & x64 infrastructures), targeting:

  • Windows 10
  • Windows 7
  • Ubuntu 16.04
  • macOS 10.12

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
1 Compatibility across all available channels Testing should cover all the Firefox channels (from Release to Nightly) 2-Moderate 2-Possible 3-High 4
2 Differences depending on channels (built in vs. add-on) and profiles (already opened DevTools vs. never opened DevTools) Testing should also regard the previous usage detection process and the differences induced by the installation work flow 3-High 3-Almost Certain 3-High 12
3 Localization/RTL Testsuite will include RTL/non-RTL differentiation 2-Moderate 2-Possible 3-High 12
4 Platform specific issues Most users are using Windows but the team mostly works on OSX/Unix, so there is a risk for Windows only regressions. Therefore the testing process will assure full platform coverage 3-High 3-Almost Certain 3-High 12

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 Previous usage detection Verify if DevTools previous usage state is properly detected on Beta and Release builds and if the corresponding install process is initiated If DevTools was previously used, the addon is automatically downloaded and installed. If the user never tried DevTools, it will go through the install process Manual Release QA
2 Install entry points Verify if the entry points for users to install DevTools are captured The corresponding install work flow is initiated when DevTools are launched, depending on the previous usage status Manual Release QA
3 Install work flow Verify if the installation process is properly executed Depending on the captured entry point, the corresponding install page is opened in a new tab and the installation process is successfully completed Manual Release QA
4 DevTools functionality and usability Verify if all DevTools functionalities are working Toolbox, Inspector, Console, Debugger, Netmonitor and all the related tools are fully functional and no regressions are triggered by the new install work flow Manual Release QA

Builds

DevTools Go Faster will be tested on the following builds:

  • Nightly - waiting for custom builds provided by Engineering
  • DevEdition - starting with the build from
  • Beta - starting with
  • Release - starting with

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 05/01/2017
Study documentation/specs received from developers 05/20/2017
QA - Test plan creation 05/22/2017
QA - Test cases/Env preparation 06/06/2017
QA - Nightly Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management GitHub & Bugzilla

Status

Overview

  • Released to Nightly starting with
  • Merged to beta starting with
  • Merged to Release starting with

References

List and links for specs:

Testcases

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled No
Theme (high contrast) No
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) No
Interaction (scroll, zoom) Yes
Usable with a screen reader No e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
RTL build testing Yes
Help/Support
Help/support interface required Yes Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Yes Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected Yes
Does sync work across upgrades No
Requires install testing Yes separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding Yes Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing No yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration No
Network proxies/autoconfig No
ESR behavior changes No
Locked preferences No
Data Monitoring
Temporary or permanent telemetry monitoring No List of error conditions to monitor
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites Yes
Survey of many sites for compatibility Yes
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features Yes

Test suite

Full Test suite - Testrail  
Smoke Test suite - Testrail
Regression Test suite - Link with the tests - if available/needed

Bug Work

Reference bugs

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite, link to TestRail - Tests Runs and Results link
    • Daily Smoke, if needed/available
    • Regression Test suite, if needed/available


Merge to Beta Sign-off
List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff Email to be sent
QA Nightly - Full Testing
QA pre-Beta Signoff Email to be sent
QA Beta - Full Testing
QA pre-Release Signoff Email to be sent