QA/Activity Stream (system add-on)

From MozillaWiki
< QA
Jump to: navigation, search

Overview

Purpose

Activity Stream is a Firefox about:newtab and about:home replacement that aims to help users return quickly and easily to contextually relevant stuff on the web. It is currently implemented as an Add-on SDK/Jetpack add-on, and has been running as a TestPilot experiment for some months. more info

The Test Plan has been moved to: https://docs.google.com/document/d/1AkPOXuL-6m9Vhlnex9sC6bkTXKAwWcZSgoeZIzV93AQ/edit

Ownership

Engineering:

QA:

Communication channels:
IRC - #activity-stream
Contact email - Activity Stream (goes to Activity Stream staff)

Testing summary

Scope of Testing

In Scope

The scope of our testing is the Activity Stream desktop system add-on and its functionality. The testing effort for Activity Stream will be invested on the following areas:

  • integration: verify the add-on integration with the current browser functionalities and UI;
  • functionality: basic and advanced functionality to be verified according to the existing requirements;
  • usability: the Activity Stream New Tab content must be easy to use and straightforward;
  • testing how Bookmarks and History are imported in the add-on functionalities (from profile or Firefox account);

Out of Scope

The mobile testing is not in Desktop QA Engineering's scope.

Telemetry (server side) is covered by the Onyx subsystem.

Pocket (server side) is covered by Pocket internal test infrastructure.

Snippets (server side) is covered by Snippets subsystem.

Requirements for testing

Environments

Testing will be performed on the following machines:

  • Intel Core i5 @2.7 GHz, RAM 8GB, Intel Iris Graphics 6100 - Mac OS X 10.10, Mac OS X 10.11, Mac OS X 10.12
  • AMD FX-8320 @3.5 Ghz, RAM 16GB, ATI Radeon 3000 - Win 7 x64
  • AMD FX-8320 @3.5 GHz, RAM 16GB, nVidia 210 - Win 8.1 x64, Win 10 x64, Ubuntu 14.04, Ubuntu 15.04
  • Intel i5-3470 @3.2 GHz, RAM 16GB AMD R7 360 - Ubuntu 12.04 x32, Ubuntu 14.04 x64

All workstations are protected by Kaspersky Endpoint Security 10 v10.3.0

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Dependencies on profile History and Bookmarks TO-2 3-High 3-Almost Certain 3-High 27
RAC-2 Information should be displayed as soon as the browser starts TO-1 3-High 3-Almost Certain 3-High 27
RAC-3 Localization TO-5 3-High 3-Almost Certain 3-High 27
RAC-4 Accessibility TO-6 2-Moderate 2-Possible 3-High 12
RAC-5 Disable A-S from about:config TO-7 2-Moderate 2-Possible 3-High 12

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 Activity Stream functionality and content Verify that Activity Stream content is displayed in a new tab and about:home (TBD) Activity stream user content is displayed in each new tab or about:home (TBD) Manual Eng Team
2 Bookmarks & history Verify that profile / account bookmarks and history are correctly imported in the add-on All the previous stored data is displayed on the Activity Stream pages Manual Eng Team
3 Homepage override & about:home override (TBD) 1. Verify that Activity Stream content is displayed when navigating to clicking the "Home" button of about:home (TBD)
2. Verify that custom homepages can still be set. 		1. Clicking the Home button or navigating to about:home (TBD) correctly redirects to the Activity Stream page
2. Navigating to about:home correctly redirects to the custom set page 		Manual Eng Team
4 Private Window Verify that the Activity Stream content is not displayed when opening a New Private window or a New Tab in the Private window The privacy page is displayed when opening a New Private window or tab Manual Eng Team
5 Localization 1. Verify that the Activity Stream page and customization menu are localized.
2. Verify that the Activity Stream page and customization menu is localized in RTL locales and content is correctly displayed. 		1. The Activity Stream page and the customization menu are localized.
2. The Activity Stream page and the customization menu are localized and correctly displayed in RTL. 		Manual Eng Team
6 Accessibility 1. Verify that the Activity Stream page and customization menu are correctly rendered when using Windows High Contrast themes.
2. Verify that the Activity Stream page and customization menu are usable with a Screen Reader. 		1. Areas are correctly rendered when using the high contrast themes.
2. The Screen Reader can successfully read the elements on the page. 		Manual Eng Team
7 Disable / Re-enable the Activity Stream system add-on Verify that the system add-on can be disabled and enabled without any issues. 1. Disable the Activity Stream system add-on from the "about:config" page, using "browser.newtabpage.activity-stream.enabled" pref ("True" for enabled and "False" for disabled).
2. Verify that the browser UI returned to default functionality on the affected areas.
3. Re-enable Activity Stream system add-on and verify that the browser has changed back. 		Manual Eng Team
8 Memory usage Compare the memory usage of the add-on with measurements from the past. 1. Measure the memory usage from about:memory with the sysyem add-on turned OFF.
2. Measure the memory usage from about:memory with the system add-on turned ON.
3. Compare the memory profiles over time. 		Manual Eng Team

Builds

Activity Stream system add-on can be found in the next builds:

  • Nightly - starting with build from 2017-03-06 - a blank page
  • Nightly - starting with build from 2017-04-24 - Top Sites and Search were implemented


Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 02/15/2016 -
Study documentation/specs received from developers 02/15/2016 02/16/2016
QA - Test plan creation 01/25/2016 02/17/2016
QA - Test cases/Env preparation 02/18/2016 03/02/2016
QA - Nightly Testing 05/16/2017 -
QA - Beta Testing - -
Load Testing - -
Release Date - -

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management GitHub / Bugzilla

Status

Overview

References

List and links for specs:

Meta bug:

Testcases

Overview

- Summary of testing runs

Test Areas

Test Areas Covered Details
General
Private Window Yes
Multi-Process Enabled Yes
Multi-Process Disabled Yes
Theme (high contrast) Yes
Memory measurements (manual) Yes
L20n
Verify Activity Stream is available in core set of supported locales Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes Navigation, selection and link opening only.
Display (HiDPI) Yes Testing on 2k and 4k resolution displays.
Interaction (scroll, zoom) Yes
Usable with a screen reader Yes e.g. with NVDA
Usability and/or discoverability testing Yes The feature replaces the about:newtab page
Install/Upgrade/Uninstall
Feature upgrades/downgrades data as expected Yes
Does sync work across upgrades Yes Tested if data is kept after upgrade, after import or sync profile data.
Requires install testing No
Affects first-run or onboarding Yes Requires some fetching time if the profile used has stored history and bookmarks.
Does this affect partner builds? Partner build testing No
Verify that Activity Stream can be "disabled" Yes It should be able to be enabled and disabled by a toggle in about:preferences or by pref in about:config
Data Monitoring
Temporary or permanent telemetry monitoring Yes Not covered by the Engineering QA team
Telemetry correctness testing Yes Not covered by the Engineering QA team
Server integration testing Yes Not covered by the Engineering QA team
Offline and server failure testing Yes Not covered by the Engineering QA team
Load testing Yes Not covered by the Engineering QA team
Add-ons
If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular add-ons Yes
Security
Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review Yes Not covered by the Engineering QA team
Privilege escalation testing Yes Not covered by the Engineering QA team
Fuzzing Yes Not covered by the Engineering QA team
Web Compatibility
Testing against target sites No
Survey of many sites for compatibility No
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS Yes
Interaction of this feature with other browser features Yes

Test suite

Bug Work

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan</s>
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing
List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite - N/A

Merge to Beta Sign-off List of OSes that will be covered by testing

  • Link for the tests run - N/A
    • Full Test suite - N/A

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup No
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [IN PROGRESS]
Automation Coverage - Not on QA manual end.
Performance Testing - Not on QA manual end.
All Defects Logged
Critical/Blockers Fixed and Verified
Daily Status Report (email/etherpad statuses/ gdoc with results)
Metrics/Telemetry
Load testing
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent


Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Lawrence Mandel / RyanVM QA Manager Product Integrity Date Email
Tim Spurway Software Engineer Product Engineering Date Email
Nick Chapman / Maria Popova EPM Product Management Date Email


Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
05/18/2017 1.0 Ciprian Muresan Created the first draft for the Activity Stream system add-on
07/11/2017 1.1 Vlad Bacia-Mociran Updated the document with info regarding Memory measurements
08/18/2017 1.2 Vlad Bacia-Mociran Updated the document with info regarding "Approvals Required", "Risk Assessment", "Test Objectives" sections
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Activity_Stream_(system_add-on)&oldid=1182665"