QA/Shield
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
5/12/2016 | 1.0 | Adrian Florinescu | Created first draft |
10/10/2016 | 1.1 | Adrian Florinescu | Updates |
11/10/2016 | 1.2 | Adrian Florinescu | Updates |
17/10/2016 | 1.3 | Adrian Florinescu | Updates |
02/21/2017 | 2.0 | Adrian Florinescu | Updates |
03/27/2017 | 2.1 | Adrian Florinescu | Updates |
Contents
Overview
Scope
This wiki details the testing that will be performed by the QA-Eng team for the SHIELD project. SHIELD is a system addon that addresses user attrition and satisfaction in Firefox. QA focus shall be given mainly to:
- SHIELD Recipe management
- SHIELD Control center (recipe administration center)
- Delivery system for the Shield recipes: how and if the filter expressions are targeting a certain niche of the FF users.
Ownership
Project Owner: Gregg Lind [:gregglind ]
Product Manager: Matt Grimes [:Matt_G]
Development: Michael Cooper [:mythmon]
Development: Michael Kelly [:Osmose]
Development: Gregg Lind [:gregglind]
PM, QA Engineeting Team - Rares Bologa
QA Engineering Team - Adrian Florinescu
QA Engineering Team - Ovidiu Boca
Testing summary
Scope of Testing
In Scope
Detail what is in scope from a testing perspective for the project team:
- Testing the admin interface : control center -> basic functionality of creating/updating a recipe from the functional point of view
- Minimal recipe delivery: should cover integration with the Firefox client
- Basic recipe filters: the testing will cover that the recipes are delivered and executed to the intended bucket of people.
- Basic performance variations on FF browser: the testing shall cover basic scenarios to identify that broken or extreme recipes do not affect negatively FF
Out of Scope
- Control Center
- Peer Review
- Preferences
Requirements for testing
Environments
TBD - Specify OSes that need to be covered Specify other configuration/environmental setup needed - The preliminary testing phase shall take place in Nightly 52, but the requirements for testing Shield as it graduates to system add-on will change. See Channel Configuration
Channel dependent settings (configs) and environment setups
Nightly
The System Add-on is to be tested with two configurations:
- 1. Mock test recipes - emulation of recipes
Documentation on: https://normandy-mock.dev.mozaws.net/
- Obtain a copy of Firefox with the SHIELD recipe client system add-on installed. You can check about:support to ensure that you have it.
- Set the extensions.shield-recipe-client.dev_mode preference to true to run recipes immediately on startup.
- Set the extensions.shield-recipe-client.logging.level preference to 0 to enable more logging.
- Set the security.content.signature.root_hash preference to 4C:35:B1:C3:E3:12:D9:55:E7:78:ED:D0:A7:E7:8A:38:83:04:EF:01:BF:FA:03:29:B2:46:9F:3C:C5:EC:36:04. This is the public hash used for verifying content signatures of the test data.
- The preference value for extensions.shield-recipe-client.api_url should be set for the mock recipe you want to run.
- 2. Staging server - using Shield System addon and the staging Control Center
- You need VPN access.
- Obtain a copy of Firefox with the SHIELD recipe client system add-on installed. You can check about:support to ensure that you have it.
- Set the extensions.shield-recipe-client.dev_mode preference to true to run recipes immediately on startup.
- Set the extensions.shield-recipe-client.logging.level preference to 0 to enable more logging.
- Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
- Set extensions.shield-recipe-client.api_url to https://normandy-admin.stage.mozaws.net/api/v1
- To access control center, https://normandy-admin.stage.mozaws.net/control/.
Aurora
text
Beta
1. Set the preference value for extensions.systemaddon.update.url set to https://aus5.mozilla.org/update/3/SystemAddons/53.0.0/20160922113459/default/en-US/beta-sysaddon/default/default/default/update.xml 2. Go to Tools -> Developer Tools -> Web Console -> Open "Toolbox Options" from "Advanced settings" mark as checked the last 3 available check boxes. (browser chrome....; remote debugging; worker debugging) 3. Open Browser Console and run this script: Components.utils.import("resource://gre/modules/AddonManager.jsm"); AddonManagerPrivate.backgroundUpdateCheck(); 4. Set the preference value for extensions.shield-recipe-client.api_url set to https://normandy.stage.mozaws.net/api/v1 5. Set the extensions.shield-recipe-client.dev_mode preference to true to run recipes immediately on startup. 6. Set the extensions.shield-recipe-client.logging.level preference to 0 to enable more logging. 7. Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90. 8. Set the preference value for extensions.shield-recipe-client.api_url set to https://normandy.stage.mozaws.net/api/v1
For mock recipes:
8'. Set the security.content.signature.root_hash preference to 4C:35:B1:C3:E3:12:D9:55:E7:78:ED:D0:A7:E7:8A:38:83:04:EF:01:BF:FA:03:29:B2:46:9F:3C:C5:EC:36:04. This is the public hash used for verifying content signatures of the test data.
Post Beta / Release
text
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
Ref | Function | Test Objective | Test Type | Risk Assessment and Coverage(RAC) | Owners |
---|---|---|---|---|---|
TO-1 | Control Center | To verify basic functionality of the admin interface | Manual | RAC-3 | Eng Team |
TO-2 | Recipe fetching and execution | To verify that recipes are fetched and executed when required | Manual | Eng Team | |
TO-3 | Filter validation | To verify that recipes are executed only when filter expression returns true | Manual | RAC-1 | Eng Team |
TO-4 | Heartbeat basic functionality | To verify the basic functionality of a heartbeat | Manual | RAC-4, RAC-5 | Eng Team |
TO-5 | Recipes/Heartbeat && FF integration | To verify that recipe execution and heartbeat display doesn't affect performance and visual expectations | Manual | RAC-4 | Eng Team |
TO-6 | Show heartbeat defaults | Checks: one heartbebat shown per 24hrs, display heartbeat again after 7 days + the other admin recipes display settings | Manual | - | Eng Team |
TO-7 | Sampling | Basic functionality of recipe ratio/buckets | Manual | - | Eng Team |
TO-8 | Test heartbeat/recipes edge cases - mock recipes | Covers mock testing of edge case scenarios | Manual | RAC-4 | Eng Team |
Risk Assessment and Coverage
ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
---|---|---|---|---|---|---|
RAC-1 | Recipes are not ran when filter == true | TO-03 | 3-High | 3-High | 3-High | 27 |
RAC-2 | Filter conditions are not returning the expected results | TO-3 | 3-High | 3-High | 3-High | 27 |
RAC-3 | Admin validation allows recipes values that break recipe workflow | TO-01 | 3-High | 3-High | 1- Low | 9 |
RAC-4 | Edge cases might break FF/Shield | TO-04, TO-05, TO-8 | 2-Moderate | 3-Almost Certain | 2-Medium | 12 |
RAC-5 | about: pages cannot be used as Heartbeat links | TO-04 | 2-Moderate | 2-Possible | 2-Medium | 8 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Builds
This section should contain links for builds with the feature -
- Links for Nightly builds
- Links for Aurora builds
- Links for Beta builds
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 10.05.2016 | 04.20.2017 |
Study documentation/specs received from developers | 10.05.2016 | 04.20.2017 |
QA - Test plan creation | 12.05.2016 | 04.20.2017 |
QA - Test cases/Env preparation | 06.23.2016 | 20.04.2017 |
QA - Nightly Testing | 11.20.2016 | 01.30.2016 |
QA - Aurora Testing | 01.30.2016 | 03.25.2017 |
QA - Beta Testing | 03.25.2017 | 04.20.2017 |
Release Date | TBD | TBD |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | Test Rail |
Test case execution | Test Rail |
Bugs management | Bugzilla / Git |
References
- List and links for specs
Testcases
Overview
Summary of testing scenarios - TBD
Test Areas
Test Areas | Covered | Details | Reviewer |
---|---|---|---|
Private Window | Yes | Osmose, Mythmon | |
Multi-Process Enabled | Yes | Osmose, Mythmon | |
Multi-process Disabled | Yes | Osmose, Mythmon | |
Theme (high contrast) | Yes | Osmose, Mythmon | |
UI | |||
Mouse-only operation | Yes | Osmose, Mythmon | |
Keyboard-only operation | Yes | Osmose, Mythmon | |
Display (HiDPI) | No | Osmose, Mythmon | |
Interraction (scroll, zoom) | Yes | the heartbeat bar shouldn't zoom (same behavior as the url bar) | Osmose, Mythmon |
Usable with a screen reader | No | Osmose, Mythmon | |
Usability and/or discoverability testing | Yes | Is this feature user friendly | Osmose, Mythmon |
Help/Support | |||
Help/support interface required | No | future plans to have something related to this, but not in the first versions | Osmose, Mythmon |
Support documents planned(written) | Yes | Make sure support documents are written and are correct. | Osmose, Mythmon |
Install/Upgrade | |||
Feature upgrades/downgrades data as expected | No | Osmose, Mythmon | |
Does sync work across upgrades | No | Osmose, Mythmon | |
Requires install testing | No | System addon | Osmose, Mythmon |
Affects first-run or onboarding | Yes | at least some tcs ran for first run | Osmose, Mythmon |
Does this affect partner builds? Partner build testing | Yes | Osmose, Mythmon | |
Enterprise | Raise up the topic to developers to see if they are expecting to work different on ESR builds | ||
Enterprise administration | No | Osmose, Mythmon | |
Network proxies/autoconfig | No | Osmose, Mythmon | |
ESR behavior changes | No | Osmose, Mythmon | |
Locked preferences | No | Osmose, Mythmon | |
Data Monitoring | |||
Temporary or permanent telemetry monitoring | Yes | Osmose, Mythmon | |
Telemetry correctness testing | Yes | Osmose, Mythmon | |
Server integration testing | Yes | Chris H handles it | Osmose, Mythmon |
Offline and server failure testing | Yes | Chris H handles it | Osmose, Mythmon |
Load testing | No | Osmose, Mythmon | |
Add-ons | If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | ||
Addon API required? | No | Osmose, Mythmon | |
Comprehensive API testing | No | Osmose, Mythmon | |
Permissions | No | Osmose, Mythmon | |
Testing with existing/popular addons | Yes | Pop-up blocker, privacy addons, profile json, test pilot integration testing: talk with BM team about effort | Osmose, Mythmon |
Security | Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | Osmose, Mythmon | |
3rd-party security review | TBD | osmoses to advice further on this | Osmose, Mythmon |
Privilege escalation testing | No | Osmose, Mythmon | |
Fuzzing | TBD | osmoses to advice further on this | Osmose, Mythmon |
Web Compatibility | depends on the feature | ||
Testing against target sites | No | Osmose, Mythmon | |
Survey of many sites for compatibility | Yes | scenarios with heartbeat redirect with addon blocker and blocked site | Osmose, Mythmon |
Interoperability | depends on the feature | ||
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | No | Osmose, Mythmon | |
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | No | Osmose, Mythmon | |
Interaction of this feature with other browser features | yes | test if it's possible to send to the about:accounts as a survey page | Osmose, Mythmon |
Test suite
Full Test suite - Link within TestRail link
Bug Work
Tracking bugs:
- Bug 1275996 - [Tracker] SHIELD System Add-on v1.0
- Bug 1308656 - [Land shield-recipe-client as system addon
- Bug 1342014 - [Shield] QA tracker for manual Testing Bug 1308656 - Land shield-recipe-client as system addon
Bug work
- bug 1342014 - QA tracker for manual Testing Bug 1308656 - Land shield-recipe-client as system addon]
34 Total; 8 Open (23.53%); 17 Resolved (50%); 9 Verified (26.47%);
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Checklist
Exit Criteria | Status | Notes/Details | |
---|---|---|---|
Testing Prerequisites (specs, use cases) | done | ||
Testing Infrastructure setup | done | ||
Test Plan Creation | done | ||
Test Cases Creation | done | ||
Full Functional Tests Execution | done | ||
Automation Coverage | yes | ||
Performance Testing | no | ||
All Defects Logged | no | defects logging split between Bugzilla and Git | |
Critical/Blockers Fixed and Verified | yes | ||
Daily Status Report (email/etherpad statuses/ gdoc with results) | Yes | when required | |
Metrics/Telemetry | N/A | ||
QA Signoff - Nightly Release | N/A | System Add-on | |
QA Aurora - Full Testing | N/A | System Add-on | |
QA Signoff - Aurora Release | N/A | System Add-on | |
QA Beta - Full Testing | N/A | System Add-on | |
QA Signoff - Beta Release | N/A | System Add-on |