QA/Mobile/FirefoxTVTestPlan

This is the Test Plan for Firefox for the Amazon Fire TV Stick (https://github.com/mozilla-mobile/firefox-tv)

Intro/Summary/Notes

Intro and Summary

• This wiki outlines the test requirements for Firefox for the Amazon Fire TV Stick releases.
• Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases

Notes

• Respository

Feature Lists

For the list of features and its status, please refer to here.

• NOTE: Firefox for the Amazon Fire TV Stick bugs are maintained in Github

Schedule

• 2.0 was released in March 2018
• Future milestones listed: here

QA Team

• Kate Glazko
• Andrei Bodea

Resource Plan

• Each QA person will be 100% allocated to this project

Test Strategy

Process

• When the development work on github issue is completed, whether it is before/after code freeze, QA-ready label may be applied to high-priority issues, indicating that they are available for testing. The issue should belong to a correct milestone before the development/QA work can be started.
• QA will verify the appropriate closed issues under the milestone, and upon completion, issues will be marked with QA-approved label and/or moved to "QA Verified" column in the project dashboard.
• If a closed issue under the milestone is not QA verifiable, or worth the investment for testing time any QA label applied will be removed and left in closed state.
• After the QA sign off, it is implied that all closed issues that are marked as QA-approved' and/or in "QA Verified" column in the project dashboard has been verified by QA team.
• When an issue is found against the new feature, the feature issue # should be referenced in the bug description.
• This does not apply to metabugs.
• For sprint status, please refer to the Firefox TV project dashboard.
• For QA Criteria, please see Focus Android QA Criteria

Manual Tests

UI Test suite for Focus for Android will be located in [1]. As a minimum, each of the P1 UI features will have corresponding test cases. The test suite will be executed by Aaron for the time being for acceptance testing.

Automated Tests

Automation tests will be mainly used for quick regression check of key UI features. Overall, automation tests will be added to the same Github repo, and executed on Firebase (For PRs) and Bitbar (For merge to master). The test result will be inspected for new failures, and github issue will be raised for tracking.

Automation tests are located here.

In addition to regularly run automated tests, QA will also generate screenshots for l10n verification upon request.

Code Analysis

• Codecov will be added to the github repo, and will display increase/decrease of unit test code coverage
• Findbugs addon is added to warn developers against possible code issues, as well as Lint.

Outside Scope

Following tests are outside the scope of the QA:

• Unit Tests
• Performance Tests
• Localization Tests
• Beta testing with a wider audience

Test Deliverables

• Completed TestRail Test suite which covers all features with UI aspects
• Automated Test suite in master branch, running on every checkin
• Github issues for every issue found
• Manual acceptance test report
• Signoff Decision

Dependencies/Risks

• Since the blocking status of web trackers is not exposed via adb log or UI, its validation will rely on unit tests
• The performance of Firefox-tv is not measured precisely, will rely on the tester feedback
• The automation/manual test suites will be built incrementally as new features are added to Github, and the late features may not be exposed to validation as much as earlier ones. The acceptance test after the feature completion date will be performed with the special emphasis on 1. key features and 2. most recently added features

Signoff/Exit Criteria

• No critical or major UI bugs
• No easily reproducible crashes