QA/CSS Grid Inspector Layout View
Approvals Required / Received
The following individuals are required to/have approved this Test Plan:
Name | Title | Department | Approval Date | Method |
---|---|---|---|---|
QA Manager | Product Integrity | Date | ||
Software Engineer | Engineering | Date | ||
EPM | Product Management | Date |
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
2017-06-12 | 1.0 | Bogdan Maris | Created first draft |
2017-06-22 | 1.1 | Bogdan Maris | Updated info (Test Strategy section) |
2017-07-07 | 1.2 | Bogdan Maris | Minor updates after mid-Nightly sign-off |
2017-09-08 | 1.3 | Bogdan Maris | Minor updates after pre-Release sign-off |
Contents
Overview
Grid Layout lets us properly separate the order of elements in the source from their visual presentation. As a designer this means you are free to change the location of page elements as is best for your layout at different breakpoints and not need to compromise a sensible structured document for your responsive design.
Purpose
The purpose of this testing is to make sure that the web developers don't run into issues when using CSS Grid Layout. Note that Jen Simmons is actively testing CSS Grid, giving feedback and logging bugs.
- The test scope, focus areas and objectives
- The test responsibilities
- The test strategy for the levels and types of test for this release
- The entry and exit criteria
- The basis of the test estimates
- Any risks, issues, assumptions and test dependencies
- The test schedule and major milestones
- The test deliverables
Scope
This wiki details the testing that will be performed by the project team for the CSS Grid Inspector Layout View project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- What will be tested
- How testing will be performed
Ownership
PM: Bryan Clark (irc: clarkbw)
Development: Gabriel Luong (irc: gl) Matteo Ferretti (irc: zer0 or matteo) Patrick Brosset (irc: pbro) Julian Descottes (irc: jdescottes)
Platform QA Team: Andrei Vaida (irc: avaida) - Team Lead Bogdan Maris (irc: bogdan_maris) Ciprian Georgiu (irc: ciprian_georgiu) - Peer
Testing summary
Scope of Testing
In Scope
Testing efforts will be focused on the impact that Grid overlay has on layout and functionality of its available settings across platforms.
Out of Scope
We consider out of scope testing grid properties in combination with different css/html elements, that will be best covered by a web developer.
Requirements for testing
Environments
OSs that need to be covered: * Windows 10 * Windows 7 * Ubuntu 16.04 * macOS 10.12
Channel dependent settings (configs) and environment setups
Nightly
- Feature fully enabled by default, no config or special environment setup required.
Beta
- Feature fully enabled by default, no config or special environment setup required.
Post Beta / Release
- tbd
Test Strategy
Risk Assessment and Coverage
ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
---|---|---|---|---|---|---|
RAC-1 | Users are encountering visual artifacts when Grid layout is active | TO-1 | 2-Moderate | 2-Possible | 3-High | 12 |
RAC-2 | Artifacts are left behind when grid overlay is toggled (enabled/disabled) | TO-1, TO-3, TO-4 | 3-High | 3-Almost Certain | 3-High | 27 |
RAC-3 | The Grid overlay is not changing alongside web elements when zooming in page | TO-1, TO-2 | 2-Moderate | 2-Possible | 3-High | 12 |
RAC-4 | Users are not able to change the Grid Overlay color based on their choosing | TO-4 | 2-Moderate | 2-Possible | 2-Medium | 8 |
RAC-5 | Learn more link is pointing to an invalid help page | 1-Low | 1-Unlikely | 1-Low | 1 | |
RAC-6 | Numbers are not displayed for grid lines | TO-4 | 2-Moderate | 2-Possible | 3-High | 12 |
RAC-7 | Grid areas are not displayed when available | TO-4 | 1-Low | 2-Possible | 2-Medium | 4 |
RAC-8 | Grid lines are not extended infinitely | TO-4 | 1-Low | 1-Unlikely | 1-Low | 1 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
Ref | Function | Evaluation Criteria | Test Objective | Test Type | RAC | Owners |
---|---|---|---|---|---|---|
TO-1 | CSS Grid graphics | Verify that no unwanted graphical glitches can be seen | Manual | RAC-1, RAC-2, RAC-3 | Release Team | |
TO-2 | Scroll/zoom when Grid is active | Verify that grid overlay is displayed correctly when zooming and scrolling | Manual | RAC-3 | Release Team | |
TO-3 | Overlay Grid is enabled and disabled | Verify that enabling/disabling Grid overlay works every time | Manual | RAC-2 | Release Team | |
TO-4 | Grid display settings | Verify that all settings for grid overlay work as expected | Manual | RAC-2, RAC-4, RAC-6, RAC-7, RAC-8 | Release Team | |
TO-5 | Grid overlay not available when CSS grid is not used in page | Verify that Grid overlay is only used in websites that use CSS Grid, otherwise display message to inform the user | Manual | Release Team |
Builds
This section should contain links for builds with the feature:
- Link for Latest Nightly builds
- Link for Beta 56 builds
- Link for Developer Edition 56 builds
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 2017-03-16 | 2017-09-26 |
Study documentation/specs received from developers | n/a | n/a |
QA - Test plan creation | 2017-06-12 | 2017-06-22 |
QA - Test cases/Env preparation | 2017-06-12 | 2017-06-26 |
QA - Nightly Testing | 2017-06-28 | 2017-07-28 |
QA - Beta Testing | 2017-08-02 | 2017-09-25 |
Release Date | 2017-09-26 |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail/ Google docs |
Test case execution | TestRail |
Bugs management | Bugzilla |
Status
Overview
Track the dates and build number where feature was released to Nightly
- 2017-06-20 merge to Nightly 56 (bug 1373492) [DONE]
Track the dates and build number where feature was merged to Beta
- 2017-08-02 merge to Firefox Beta 56. [DONE]
Track the dates and build number where feature was merged to Release
- 2017-09-26 merge to Firefox Release 56 [IN PROGRESS]
References
- List and links for specs
- Meta bugs
- bug 1181227 - [meta] CSS Grid Inspector
- bug 1347964 - [meta] Ship the new Inspector Layout panel
Testcases
Test Areas
Test Areas | Covered | Details |
---|---|---|
Private Window | Yes | |
Multi-Process Enabled | Yes | |
Multi-process Disabled | Yes | |
Theme (high contrast) | Yes | |
UI | ||
Mouse-only operation | No | The feature can be enabled and user by keyboard/touchscreen as well |
Keyboard-only operation | No | The feature can be enabled and used by mouse/touchscreen as well |
Display (HiDPI) | Yes | |
Interaction (scroll, zoom) | Yes | |
Usable with a screen reader | No | e.g. with NVDA |
Usability and/or discoverability testing | Yes | Is this feature user friendly |
RTL build testing | Yes | |
Help/Support | ||
Help/support interface required | TBD | Make sure link to support/help page exist and is easy reachable. |
Support documents planned(written) | TBD | Make sure support documents are written and are correct. |
Install/Upgrade | ||
Feature upgrades/downgrades data as expected | No | |
Does sync work across upgrades | No | |
Requires install testing | No | This is a feature that is included in Firefox, no 3party apps needed |
Affects first-run or onboarding | No | Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned. |
Does this affect partner builds? Partner build testing | No | yes/no options, add comment with details about who will lead testing |
Enterprise | Raise up the topic to developers to see if they are expecting to work different on ESR builds | |
Enterprise administration | No | Not yet available in ESR will probably be when it reaches 57 |
Network proxies/autoconfig | No | Not yet available in ESR will probably be when it reaches 57 |
ESR behavior changes | No | Not yet available in ESR will probably be when it reaches 57 |
Locked preferences | No | Not yet available in ESR will probably be when it reaches 57 |
Data Monitoring | ||
Temporary or permanent telemetry monitoring | No | List of error conditions to monitor |
Telemetry correctness testing | No | No telemetry is available for this feature AFAIK |
Server integration testing | No | |
Offline and server failure testing | No | |
Load testing | No | |
Add-ons | If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | |
Addon API required? | No | |
Comprehensive API testing | No | |
Permissions | No | Feature does not use permissions to work |
Testing with existing/popular addons | No | |
Security | Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | |
3rd-party security review | No | |
Privilege escalation testing | No | |
Fuzzing | No | |
Web Compatibility | depends on the feature | |
Testing against target sites | Yes | |
Survey of many sites for compatibility | No | This feature is only available on a handful of websites that use CSS Grid |
Interoperability | depends on the feature | |
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | No | |
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | No | |
Interaction of this feature with other browser features | No |
Test suite
Full Test suite - Testrail Link Smoke Test suite - Gdoc Link - only draft for now. Regression Test suite - Link with the tests - if available/needed.
Bug Work
Meta bugs:
- 1181227 - (meta) CSS Grid Inspector
- 1347964 - (meta) Ship the new Inspector Layout panel
- 1333706 - (meta) QA issues for CSS Grid Inspector
Logged bugs ( blocking 1333706 )
15 Total; 4 Open (26.67%); 7 Resolved (46.67%); 4 Verified (26.67%);
Bug fix verification
ID | Priority | Component | Assigned to | Summary | Status | Resolution | Target milestone |
---|---|---|---|---|---|---|---|
15069 | P3 | Networking | jefft | File type url not correctly parsed and created | VERIFIED | FIXED | M13 |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Sign off
Criteria
Checklist
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite, link to TestRail - Tests Runs and Results link
- Daily Smoke, if needed/available
- Regression Test suite, if needed/available
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | ||
Testing Infrastructure setup | ||
Test Plan Creation | ||
Test Cases Creation | ||
Automation Coverage | ||
Performance Testing | ||
All Defects Logged | ||
Critical/Blockers Fixed and Verified | ||
Metrics/Telemetry | ||
Basic/Core functionality Nightly testing | ||
QA mid-Nightly Signoff | [DONE] | Email sent 2017-07-07 |
QA Nightly - Full Testing | ||
QA pre-Beta Signoff | [DONE] | Email sent 2017-07-28 |
QA Beta - Full Testing | ||
QA pre-Release Signoff | [DONE] | Email sent 2017-09-08 |