QA/Theming/Testplan
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
02/07/2017 | 1.0 | Krupa Raj | Created first draft |
02/09/2017 | 1.1 | Valentina V. | Updated |
02/09/2017 | 1.1 | Carciu Victor | Updated |
03/03/2017 | 1.1 | Carciu Victor | Updated - Added test cases summaries under Testcases Overview , Added some new items under Scope of Testing |
03/06/2017 | 1.1 | Krupa Raj | Updated - Removed some items to make the wiki less dense |
Contents
Overview
There is an effort to extend theming options present in Firefox today with a single, canonical and more capable alternative, based on WebExtensions. This document provides details about the testing efforts around this project.
Glossary
- LWTs: Light Weight Themes which are existing themes on AMO
- Static theme - new style themes which will be exactly like LWTs in functionality but will have a manifest.json
- Themextension: webextensions which make use of the themeing API. These are treated as webextensions. The term is not official and is Krupa's version of "fetch" ;)
- Complete themes: existing full themes which allow complete customization and will be deprecated after 57
Status
Themes can be tested in Firefox 55 by adding the boolean pref: extensions.webextensions.themes.enabled and setting it true.
Some example themes:
Ownership
Firefox Dev Lead: Michael de Boer; irc nick :mikedeboer
Webextensions Dev: Matthew Wein; irc nick :mattw
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vasilica Mihasca; irc nick :vasilica_mihasca
Add-ons QA: Valentina Virlics; irc nick :ValentinaV
Entry Criteria
- QA has access to all relevant documents/PRDs/tracker bugs
- The testcases have been reviewed by devs
- Feature has sufficient unit test coverage (determined by the devs)
- Features landing in Firefox have regression test coverage. This can be accessed by
- Entry Criteria for Dev has been satisfied
- Changes have landed in the nightly build and/or AMO dev
Current Status
- The feature is under development
Exit Criteria
- All the bugs have been triaged
- All P1 and P2 bugs have been resolved fixed
- All resolved bugs are verified
- The find/fixed rate is going down
- QA has had enough time to test the feature
Testing summary
Scope of Testing
In Scope
QA will test the following features in detail:
- Submission of static themes and themextensions
- Validation of static themes and themextensions
- Listing of LWTs and static themes indistinctly (on AMO)
- Management of static themes and themextensions
- Install and updates of static themes and themextensions
- Management of static themes and themextensions from the add-ons manager
- Using Firefox while a static theme/ themextension is installed
- Support for old-style LWTs
- Retiring Complete themes (Firefox displays a useful error message)
- Chrome parity - a chrome theme will work seamlessly on Firefox
- Android will support only actual LWTs
- RTL testing for Theming
Out of Scope
- Testing of individual themextensions and their functionality
- Load testing
- Performance testing
Requirements for testing
Environments
OSes covered: Windows, MacOSX, Linux Devices covered: Desktop, Android
Compatibility Matrix
Add-on Type | Supported Firefox versions | Supported platforms | Other Browsers |
---|---|---|---|
LWTs (old style) | All | Desktop and Android | Install not supported/appropriate messaging |
Static Themes | FF55 and greater | Desktop | Install not supported/appropriate messaging |
Themextensions | FF55 and greater | Desktop | Install not supported/appropriate messaging |
Complete Themes | supported until FF57 | Desktop | Install not supported/appropriate messaging |
Note: For static themes on Android, there is appropriate error messaging if any user tries to install.
Configs and environment setups
Set extensions.webextensions.themes.enabled to true
Test Strategy
Test Objectives
Ref | Function | Test Objective | Evaluation Criteria | Test Type | Owners |
---|---|---|---|---|---|
1 | AMO - Public pages | Test static themes/themextenions listed in AMO public pages | Users can browse, search and install static themes/themextenions on AMO | Manual and Automation | QA Team |
2 | AMO - Devhub pages | Submission & Management of Static themes | Users can successfully submit, edit, update & delete their static themes | Manual and Automation | QA Team |
3 | AMO - Linter | Validation of static themes and themextensions | Linter is updated to successfully validate the new static themes and themextensions | Manual | QA Team |
4 | AMO - Reviewer pages | Review of Static themes/themextensions | Reviewers can view, approve and reject static themes | Manual | QA Team |
5 | AMO - Admin | Admin access to static themes and themextensions | All the admin functionality is still applicable to static themes/ themextensions | Manual | QA Team |
6 | Add-ons Manager | All existing Add-ons manager functionality is retained | Users can browse, search and manage their installed static themes/themextensions via the Add-ons manager | Manual | QA Team |
7 | Backward Compatibility | Support for existing LWTs | Users can successfully install LWTs on all versions of Firefox | Manual | QA Team |
8 | Firefox with static theme/themextensions installed | Regression testing for Firefox with static theme/themextensions installed | There is no degradation of Firefox behavior with static theme/themextensions installed | Manual | QA Team
|
Builds
This section should contain links for builds with the feature -
- Links for Nightly builds
- Links for Aurora builds
- Links for Beta builds
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | ||
Study documentation/specs received from developers | ||
QA - Test plan creation | 02/07/2017 | |
QA - Test cases/Env preparation | ||
QA - Nightly Testing | ||
QA - Aurora Testing | ||
QA - Beta Testing | ||
Release Date | Firefox 55 |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail/ Google docs |
Test case execution | TestRail |
Bugs management | Bugzilla / Github |
Status
Overview
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Aurora Track the dates and build number where feature was merged to Release/Beta
Risk analysis
- Android does not support the new static themes. A developer wanting to upload a new theme will have to submit two separate themes (old and new) if they want to be listed for desktop and mobile
References
- PRD : https://docs.google.com/document/d/1ebB02HHc5Ni_ZyUwA41vJ9OlZsTEAMFOpXCaCqvuGTE/edit?ts=582dc369#heading=h.etihl2gdfzx6
- Engineering planning doc: https://docs.google.com/document/d/1ueD6V7aLLTuc1GAOxxQYcwl2HR-k62HHu3q8knTJ4FU/edit#
- AMO PRD: https://docs.google.com/document/d/1d9acBumpdK87sYg7Ah98BEKvL7b4tuO-RDpgqbm7FTs/edit#heading=h.ph0yhtsx9uv6
- Meta bug: Bug 1330328
bug 1330328 - (themingapi) Theming API
15 Total; 10 Open (66.67%); 4 Resolved (26.67%); 1 Verified (6.67%);
Testcases
Overview
Summary of testing scenarios available at https://public.etherpad-mozilla.org/p/themeing
Test Areas
We need to decide if this section is useful
Test Areas | Covered | Details |
---|---|---|
Private Window | Make sure that applying a theme or themextension works as expected in a private window | |
Multi-Process Enabled | Themes and Themextension work as expected with e10s | |
Multi-process Disabled | Themes and Themextension work as expected without e10s | |
UI | ||
Mouse-only operation | ||
Keyboard-only operation | ||
Display (HiDPI) | ||
Interraction (scroll, zoom) | Scroll and zoom work as expected with a theme or themextension applied | |
Usability and/or discoverability testing | Is this feature user friendly | |
RTL build testing | Theme and themextensions work as expected on RTL builds | |
Help/Support | ||
Help/support interface required | Make sure link to support/help page exist and is easy reachable. | |
Support documents planned(written) | Make sure support documents are written and are correct. | |
Install/Upgrade | ||
Feature upgrades/downgrades data as expected | ||
Does sync work across upgrades | ||
Requires install testing | separate feature/application installation needed (not only Firefox) | |
Affects first-run or onboarding | Check if there is a marketing plan to highlight themes in the onboarding experience | |
Does this affect partner builds? Partner build testing | yes/no options, add comment with details about who will lead testing | |
Enterprise | Raise up the topic to developers to see if they are expecting to work different on ESR builds | |
Enterprise administration | ||
Network proxies/autoconfig | ||
ESR behavior changes | ||
Locked preferences | ||
Data Monitoring | ||
Temporary or permanent telemetry monitoring | List of error conditions to monitor | |
Telemetry correctness testing | ||
Server integration testing | ||
Offline and server failure testing | ||
Load testing | ||
Security | Review should be done by the Security team | |
3rd-party security review | ||
Privilege escalation testing | ||
Fuzzing | ||
Interoperability | ||
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | ||
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | Make sure that the old LWTs still work as expected on Android; Not applicable for iOS | |
Interaction of this feature with other browser features |
Test suite
Full Test suite - Link with the gdoc, follow the format from link Smoke Test suite - Link with the gdoc, follow the format from link Regression Test suite - Link with the gdoc - if available/needed.
Bug Work
Tracker – Bug 1330328
Bug fix verification
Bug No | Summary | Status | Firefox Verion |
---|---|---|---|
123 | bug summary | NEW | Nighly 45 |
Logged bugs
Bug 111111
Bug 211111
Sign off
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
Merge to Aurora Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | ||
Testing Infrastructure setup | ||
Test Plan Creation | ||
Test Cases Creation | ||
Full Functional Tests Execution | ||
Automation Coverage | ||
Performance Testing | ||
All Defects Logged | ||
Critical/Blockers Fixed and Verified | ||
Metrics/Telemetry | ||
QA Signoff - Nightly Release | Email to be sent | |
QA Aurora - Full Testing | ||
QA Signoff - Aurora Release | Email to be sent | |
QA Beta - Full Testing | ||
QA Signoff - Beta Release | Email to be sent |