QA/Theming/Testplan

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
02/07/2017 1.0 Krupa Raj Created first draft
02/09/2017 1.1 Valentina V. Updated
02/09/2017 1.1 Carciu Victor Updated
03/03/2017 1.1 Carciu Victor Updated - Added test cases summaries under Testcases Overview , Added some new items under Scope of Testing
03/06/2017 1.1 Krupa Raj Updated - Removed some items to make the wiki less dense

Overview

There is an effort to extend theming options present in Firefox today with a single, canonical and more capable alternative, based on WebExtensions. This document provides details about the testing efforts around this project.

Glossary

  • LWTs: Light Weight Themes which are existing themes on AMO
  • Static theme - new style themes which will be exactly like LWTs in functionality but will have a manifest.json
  • Themextension: webextensions which make use of the themeing API. These are treated as webextensions. The term is not official and is Krupa's version of "fetch" ;)
  • Complete themes: existing full themes which allow complete customization and will be deprecated after 57


Status

Themes can be tested in Firefox 55 by adding the boolean pref: extensions.webextensions.themes.enabled and setting it true.

Some example themes:


Ownership

Firefox Dev Lead: Michael de Boer; irc nick :mikedeboer
Webextensions Dev: Matthew Wein; irc nick :mattw
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vasilica Mihasca; irc nick :vasilica_mihasca
Add-ons QA: Valentina Virlics; irc nick :ValentinaV

Entry Criteria

  • QA has access to all relevant documents/PRDs/tracker bugs
  • The testcases have been reviewed by devs
  • Feature has sufficient unit test coverage (determined by the devs)
  • Features landing in Firefox have regression test coverage. This can be accessed by
  • Entry Criteria for Dev has been satisfied
  • Changes have landed in the nightly build and/or AMO dev

Current Status

  • The feature is under development

Exit Criteria

  • All the bugs have been triaged
  • All P1 and P2 bugs have been resolved fixed
  • All resolved bugs are verified
  • The find/fixed rate is going down
  • QA has had enough time to test the feature

Testing summary

Scope of Testing

In Scope

QA will test the following features in detail:

  • Submission of static themes and themextensions
  • Validation of static themes and themextensions
  • Listing of LWTs and static themes indistinctly (on AMO)
  • Management of static themes and themextensions
  • Install and updates of static themes and themextensions
  • Management of static themes and themextensions from the add-ons manager
  • Using Firefox while a static theme/ themextension is installed
  • Support for old-style LWTs
  • Retiring Complete themes (Firefox displays a useful error message)
  • Chrome parity - a chrome theme will work seamlessly on Firefox
  • Android will support only actual LWTs
  • RTL testing for Theming

Out of Scope

  • Testing of individual themextensions and their functionality
  • Load testing
  • Performance testing

Requirements for testing

Environments

OSes covered: Windows, MacOSX, Linux 
Devices covered: Desktop, Android

Compatibility Matrix

Add-on Type Supported Firefox versions Supported platforms Other Browsers
LWTs (old style) All Desktop and Android Install not supported/appropriate messaging
Static Themes FF55 and greater Desktop Install not supported/appropriate messaging
Themextensions FF55 and greater Desktop Install not supported/appropriate messaging
Complete Themes supported until FF57 Desktop Install not supported/appropriate messaging

Note: For static themes on Android, there is appropriate error messaging if any user tries to install.

Configs and environment setups

Set extensions.webextensions.themes.enabled to true

Test Strategy

Test Objectives

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 AMO - Public pages Test static themes/themextenions listed in AMO public pages Users can browse, search and install static themes/themextenions on AMO Manual and Automation QA Team
2 AMO - Devhub pages Submission & Management of Static themes Users can successfully submit, edit, update & delete their static themes Manual and Automation QA Team
3 AMO - Linter Validation of static themes and themextensions Linter is updated to successfully validate the new static themes and themextensions Manual QA Team
4 AMO - Reviewer pages Review of Static themes/themextensions Reviewers can view, approve and reject static themes Manual QA Team
5 AMO - Admin Admin access to static themes and themextensions All the admin functionality is still applicable to static themes/ themextensions Manual QA Team
6 Add-ons Manager All existing Add-ons manager functionality is retained Users can browse, search and manage their installed static themes/themextensions via the Add-ons manager Manual QA Team
7 Backward Compatibility Support for existing LWTs Users can successfully install LWTs on all versions of Firefox Manual QA Team
8 Firefox with static theme/themextensions installed Regression testing for Firefox with static theme/themextensions installed There is no degradation of Firefox behavior with static theme/themextensions installed Manual QA Team


Builds

This section should contain links for builds with the feature -

  • Links for Nightly builds
  • Links for Aurora builds
  • Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation 02/07/2017
QA - Test cases/Env preparation
QA - Nightly Testing
QA - Aurora Testing
QA - Beta Testing
Release Date Firefox 55

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla / Github

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

  • Android does not support the new static themes. A developer wanting to upload a new theme will have to submit two separate themes (old and new) if they want to be listed for desktop and mobile

References

bug 1330328 - (themingapi) Theming API
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1330335 P3 Themes Theming API - Implement framework VERIFIED ---
1337334 P5 Themes Theming API Performance tracking NEW ---
1337335 P5 Themes Theming API privacy & security tracking RESOLVED ---
1341722 -- General Mike de Boer [:mikedeboer] Enable the extensions.webextensions.themes.enabled preference to ride the trains RESOLVED mozilla55
1347166 P5 Themes Theming API - Accessibility features NEW ---
1347171 P5 Themes Theming API - Google Chrome theme parity NEW ---
1347201 P5 Themes Theming API - Theming in-content about: pages NEW ---
1347205 P5 Themes Theming API - Experiments NEW ---
1348034 P5 Themes Theming API - Beyond LWT compatibility NEW ---
1348147 -- Frontend Documentation for new themes RESOLVED ---
1406454 P3 Extensions [meta] Get themes working on Android NEW ---
1417884 -- Themes [meta] Theming API polish NEW ---
1432172 P2 Themes [meta] Dark Theme Darkening RESOLVED ---
1451392 -- Themes [meta] Theming API UX NEW ---
1466496 P3 Theme & Visual Design Support the Theming API on iOS NEW ---

15 Total; 10 Open (66.67%); 4 Resolved (26.67%); 1 Verified (6.67%);


Testcases

Overview

Summary of testing scenarios available at https://public.etherpad-mozilla.org/p/themeing

Test Areas

We need to decide if this section is useful

Test Areas Covered Details
Private Window Make sure that applying a theme or themextension works as expected in a private window
Multi-Process Enabled Themes and Themextension work as expected with e10s
Multi-process Disabled Themes and Themextension work as expected without e10s
UI
Mouse-only operation
Keyboard-only operation
Display (HiDPI)
Interraction (scroll, zoom) Scroll and zoom work as expected with a theme or themextension applied
Usability and/or discoverability testing Is this feature user friendly
RTL build testing Theme and themextensions work as expected on RTL builds
Help/Support
Help/support interface required Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected
Does sync work across upgrades
Requires install testing separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding Check if there is a marketing plan to highlight themes in the onboarding experience
Does this affect partner builds? Partner build testing yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration
Network proxies/autoconfig
ESR behavior changes
Locked preferences
Data Monitoring
Temporary or permanent telemetry monitoring List of error conditions to monitor
Telemetry correctness testing
Server integration testing
Offline and server failure testing
Load testing
Security Review should be done by the Security team
3rd-party security review
Privilege escalation testing
Fuzzing
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS Make sure that the old LWTs still work as expected on Android; Not applicable for iOS
Interaction of this feature with other browser features

Test suite

Full Test suite - Link with the gdoc, follow the format from link
Smoke Test suite - Link with the gdoc, follow the format from link
Regression Test suite - Link with the gdoc - if available/needed.

Bug Work

Tracker – Bug 1330328

Bug fix verification
Bug No Summary Status Firefox Verion
123 bug summary NEW Nighly 45
Logged bugs

Bug 111111
Bug 211111

Sign off

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Daily Smoke, use template from link
    • Full Test suite, use template from link
    • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent