Compatibility/Test Plan
Contents
Purpose
This purpose wiki is to serve as a general Test Plan of Mozilla’s Webcompat strategy. It defines the test schedule and the needed resources for the product test activities, testing levels and delivery, by supporting the following objectives: Validate the look and feel and user experience on a set of websites vis-a-vis to Web Compatibility; Recommend and describe the testing strategies to be used in the testing process; Offer support to the Dev team in reproducing issues logged (more info, further investigations).
Ownership
Overall Status
[LANDED] US, France, India, Germany, Taiwan
[NOT LANDED] Indonesia, Hong Kong, Singapore, Poland, UK, Russia
Testing Summary
Scope of testing
The testing will be focus on the strategy used to validate a set of web sites vis-a-vis to Web Compatibility. The goal of Web Compatibility testing is to ensure that the sites render without errors on the target web browser (Firefox Nightly) using Android devices. Also to ensure all the design and functionality is working fine. The objective is to improve the user experience of the Web when accessed from Android devices.
Environment
Testing will be performed on following:
- Android devices: Phone: 7.x
- Browser: Firefox Nightly, Release and Chrome
Specifications
- QA Engineering team will run Exploratory and Web Compatibility testing on a single OS/Platform (Android 7.x or above for Android client).
- The focus will be towards the latest official OS on the market.
- By Exploratory testing we understand “An informal test design technique where the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests”.
- By Web Compatibility testing we understand “A type of Web testing to check whether a set of sites is capable of running on different hardware, applications, network environments or Mobile devices. Compatibility Testing is a type of the Non-functional testing.”
Acceptance Criteria
QA Engineering team will consider a test as PASS if no differences are observed in Firefox Nightly vs. Chrome browser. Any issues that fall into one or more of the following categories:
- "Desktop site instead of mobile site"
- "Site is not usable"
- "Design is broken"
- "Video or audio doesn't play"
Or any other issue observed, will be logged using the webcompat.com form and the test will be considered FAIL.
Test Deliverables
- Test Plan
- Testing progress
Requirements for Test
Items to be tested
| Features / Functionality | Mozilla Firefox Nightly |
|---|---|
| Page Layout | X
|
| Home Page | X
|
| Browse Menu | X
|
| Navigation (3 levels deep)/Categories/Sub-categories/Links | X
|
| Search/Add Search Engine | X
|
| Filter/Sort/Drop-down Menus/Date-picker | X
|
| Pagination | X
|
| Overlays/Popups | X
|
| Items Alignment/Bullets/Lists | X
|
| Forms/Text Fields/Radio Buttons/Check Boxes | X
|
| Text/Padding/Font Size/Font Colo r/Font Style | X
|
| Background | X
|
| Play media (video, music, photos)/View photo carousel/Double tap photo to zoom | X
|
| View Articles | X
|
| Share media/articles (FB, Twitter, Mail) | X
|
| Add/Remove media/articles to/from Favorites/Edit Favorites/Wishlists | X
|
| Add/Move/Remove shopping item to/from Cart/Buy product (until checkout) | X
|
| Attach/Upload media | X
|
| Chat | X
|
| Phone/Redirect | X
|
| Charts | X
|
| Map/Find Location | X
|
| Download app link (if available) | X
|
| Ads display (not to hinder site navigation dismissive) | X
|
| Create account/Email confirmation link/Sign in/Sign out/User account details/Editing/Profile Page/Error handling | X
|
| Help/Contact/Subscribe/Customer feedback/About/Terms of Usage/Privacy | X
|
| Portrait / Landscape | X
|
| Toggle Mobile/Desktop | X
|
| Differences in the UI with respect to look and feel | X
|
Testing related notes:
- All Compatibility issues observed will have a corresponding bug in the Issues found section.
- All other observed Non-Compatibility issues will be added to the Comments section.
Details on the performed tests can be seen at the following link: https://docs.google.com/spreadsheets/d/1jpdTFvBF1_SMHbaCtWwuG6aAVCV35dTBS8l9rI4plQc/edit#gid=0
Items NOT to be tested
This list represents what features/items and combination of features/items will not be tested based on various reasons like:
- Desktop sites (full site/redirects);
- Check-out/Transaction process, unless specifically requested otherwise;
- Mobile application of the web site;
- Testing performed on multiple mobile device resolutions (limited time and resources, availability), unless specifically requested otherwise;
- Testing performed on emulators, unless specifically requested otherwise;
- Web sites that need special accounts (Banking/Insurance/Governmental/NSFW premium accounts);
- NSFW explicit content (videos/photos);
- Responsive design (Split screen mode - Android 7);
- Page Validation (enable/disable JavaScript);
- Flash content;
- Cookies enable/disable;
- CSS and HTML and XHTML Validation;
- SSL certificate Validation (Revocation, Untrusted, Self-signed, Certificates with invalid sub-domain, Expired);
- CSS version;
- DOCTYPE verification;
- HTML video format.
Testing Strategy
Testing activities will be planned according to the required set of locales to be tested, found in the Compatibility/Mobile Testing Surveys document (https://wiki.mozilla.org/Compatibility/Mobile_Testing_Surveys). Currently, the QA Engineering team proposes the following testing approach:
- QA Engineering team will run Exploratory and Web Compatibility testing on a single OS/Platform (Android 7.x or above for Android client).
- The focus will be towards the latest official OS on the market.
- By Exploratory testing we understand “An informal test design technique where the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests”.
- By Web Compatibility testing we understand “A type of Web testing to check whether a set of sites is capable of running on different hardware, applications, network environments or Mobile devices. Compatibility Testing is a type of the Non-functional testing.”
