B2G/QA/Device Test Plan/Graphics
Objective
Graphics features on FFOS are difficult to verify due to its impact across different functional areas. We need a solid testing approach to make sure the new graphics features are implemented properly, by verifying new features and improvements as well as detecting key regression issues early.
Background Information
Graphics Features Backlog
Graphics backlog is tracked in this spreadsheet.
Features that are applicable to FxOS (marked on the 'FxOS Priority' column) and currently active or completed ('Status' column marked as active, completed or scheduled) should be subjected to FxOS QA activity.
Image comparison library to Python Marionette Framework
Please refer to this page for more information.
Graphics Smoke Test Suite
The purpose of smoke test is to detect graphics regression issues early on in the master branch. Sanity check of graphics performance will be tested on areas such as:
- Scrolling of homescreen
- Scrolling of web contents and rendered images
- Swiping between applications and between images
- Rendering of images and texts
- Manipulation of images and texts in applications, including browser app
- Rendering of open and closing of applications (including animations)
- Video playback
- Camera preview and image/video recording
- Orientation change
It is assumed that any new graphics feature implementation will affect one of more activities listed above.
Graphics Smoke Test Suite currently includes 9 test cases that can quickly identify graphics regressions. They can be found here.
The smoke test suite should run daily once automated, and for the cases that cannot be automated or yet to be automated, it will be verified on weekly basis as a minimum. Examples of manual-only tests include:
- Video playback
- Homescreen animation (the one that follows unlocking the screen, making sure there are at least 20 extra apps installed for a total of more than 60 icons on the home screen)
- Camera preview / image / video recording
Graphics Regression Test Cases
Currently, test cases for Firefox 2.0 which appear relevant to the graphics performance of FxOS has a tag 'gfxregtest' attached. There are 75 test cases, and selected based on following criteria:
- Renders the display images of application
- Involves the changes of the displayed images via scrolling, pinching, flicking, and other UI interactions
- Video Playback
Each feature listed in Graphics Features Backlog needs to be analyzed (when the feature is completed) to determine whether they need additional manual test cases defined. If so, new test cases would also contain 'gfxregtest' tag in mozTrap, as well as the tag indicating the Bug ID of the feature.
The Regression Test Suite will be executed 3 times per release cycle, as part of the full functional test run.
Tasks
Communicate with Graphics team
- Graphics Bug Triage: monitor incoming blocking-nominated bugs, and provide feedback during the triage session. Blocking criteria are:
- Feature regressing
- Blocking the successful completion of a common use case
- Risk of causing data loss, system freeze, or major user frustration
- Obvious rendering issues or performance issues such as checkerboarding or screen refresh issue
Any graphics bugs that fall under one or more of above criteria should be nominated(with ? flag) for the upcoming release in the b2g-release field. If a bug does not satisfy any of above criteria, mark it with 'backlog' flag.
- Discuss with Graphics team about each active features for FxOS (from graphics features backlog) for possible areas of weakness and test strategy. When a feature becomes active and FxOS relevant, it should be discussed. Record findings in Bugzilla and use them for testing when the feature is completed.
- Look for the status changes on graphics features in graphics features backlog on weekly basis, and put the testing notes in the comments of the corresponding bug. Create a query to list all active graphics feature bugs. Mark bugs with keywords if necessary.
- Monitor graphics regression bugs in Bugzilla and be up-to-date on the blocking bug status.
- This query looks for all open graphics bugs that are blocking or nominated for blocking.
- This query looks for all blocked/ nominated for blocked graphics bugs that are resolved in past week.
- Use above queries as templates to create additional queries
- Resource Estimate: 24 hrs (2 hrs per week on average)
- Deliverable: Testing notes for each feature in Bugzilla, if needed
Automate Graphics Smoke Test Suite
- Automate existing graphics smoke test suite from Section 2.4, and have the test result available on web for public viewing. Push it to master branch, if possible
- Resource Estimate: 80 hrs
- Deliverable: Python Marionette test suite in Git repository, Update wiki
Execution of Graphics Smoke Test Suite
- Set up daily execution of graphics smoke test suite once all test cases are automated.
- Resource Estimate: 24 hrs
- Deliverable: Test log, Bugzilla entry
Execute manual tests
- Execution of exploratory time-boxed (~6 hours, one full work day) testing when each graphics feature is completed. Testing will cover basic graphics features described in Section 2.3 After the testing is completed, log the results
- Resource Estimate: 40 hrs (6hrs per feature on average)
- Deliverable: Test log, Bugzilla entry
Maintain Python image comparison library
- Update the library to handle screen capture command change in 2.1
- Update/create new test cases if needed
- Bug fixes/improvements
- Resource Estimate: 40 hrs
- Deliverable: Check-ins in Git repository, Update wiki
Areas Not Covered
Following areas are not covered by this plan:
- Graphics Mochitests / Reftests
- Graphics performance measurements (e.g. APZ performance, FPS for image rendering, etc.)
Total estimated hours: 208 hours (approximately 40~50% of one person workload for the quarter)
Task Tracking
In the weekly meetings with the manager, mini-goals and progress status will be presented for feedback. This spreadsheet will be used to track goals and work progress.
