QA/IntersectionObserver

From MozillaWiki
< QA
Jump to: navigation, search

Overview

Purpose

This wiki details the testing that will be performed by QA for the Intersection Observer API. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • Areas of risk
  • What will be tested
  • How testing will be performed
    • criteria
    • ownership
    • schedule

To help ensure the best possible release and long term maintenance of this feature.

Risk Analysis

Risk area Requirement Status
Ad intensive web pages No regression in interaction performance PASS on Nightly 54
Security Fuzz testing finds no security holes PASS on Nightly 54
Stability Crash rate does not increase TBD

Testing Strategy

Scope

Test Areas Covered
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Stability Yes
UI
Display (HiDPI) Yes
Interraction (scroll, zoom) Yes
Add-ons
ad-blocking Add-ons must be disabled/removed to test Yes
Security
Fuzzing Yes
Web Compatibility
Testing against target sites Yes

If it's not listed above, it is currently out of scope.

Objectives

Criteria Description Metric w/o IO API w/ IO API Criteria Met? QA Owner
Manual testing Interaction performance Pass Pass Yes on Nightly 54 Tracy
Unit testing Automated tests Pass Pass Yes on Nightly builds of 54 Tobias
Fuzz testing Automated tests Pass Pass [1] Yes on Nightly 54 Fuzzing team (Raymond)
Stability Telemetry/Crash reports tbd - telemetry experiment in bug 1341259 tbd - telemetry experiment in bug 1341259 soon to begin with 10% enabled Tobias
[1] tracking bug has no bugs hanging off of it

Testcases

Ad-hoc testing of Ad-intensive websites. In general news sites tent to be rich of Ads. Therefore the top news sites on Alexa are a great source for suitable websites.

[1] The top 500 news sites on the web

List of known sites using the Intersection Observer API:

Possible test scenarios:

  • Open a single news site. Wait till it is fully loaded to make sure all Ads are properly initialized. If there is content to scroll, scroll up and down a couple of times. Click on some articles to trigger a refresh.
  • Open a multiple news sites in different tabs. Wait till every page is fully loaded to make sure all Ads are properly initialized. Switch between tabs from time too time. Close and open new tabs randomly. If there is content to scroll, scroll up and down a couple of times. Click on some articles to trigger a refresh.
  • Repeat scenarios above but let the browser untouched for a longer period of time. Then proceed.

Environments

Full Testing will be performed using Nightly for Desktop builds on:

  • Windows
  • Mac OS X
  • Linux

Full Testing will be performed using Nightly for Mobile builds on:

  • Android

Channel dependent settings (configs) and environment setups

  • In Nightly (and DevEd when available)- dom.IntersectionObserver.enabled needs to be set to "true" in about:config
  • If any kind of Ad-blocker is installed/in use it needs to be uninstalled/disabled

Manual Testing Results

Ad-hoc testing around testcases and scenarios above in regular and private windows, in e10s and non-e10s windows and on normal and HiDPi displays.

Nightly

Platform IO disabled IO enabled QA'd by : build ID
Windows Pass Pass Tracy : 20170228030203
Mac Pass Pass Tracy : 20170228030203
Linux Pass Pass Tracy : 20170301110155
Android Pass Pass Tracy : 20170301110150

Aurora

Platform IO disabled IO enabled QA'd by : build ID
Windows (results) (results) TBD
Mac (results) (results) TBD
Linux (results) (results) TBD
Android (results) (results) TBD

Beta

Platform IO disabled IO enabled QA'd by : build ID
Windows (results) (results) TBD
Mac (results) (results) TBD
Linux (results) (results) TBD
Android (results) (results) TBD

Project Information

Ownership

Developer contacts: Tobias Schneider
QA: Tracy Walker
QA Peer: TBD

Builds

This section should contain links for builds with the feature

Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 20160128 tbd
Study documentation/specs received from developers 20170201 20170210
QA - Test plan creation 20170210 20170215
QA - Nightly Testing 20170227 20170301
Telemetry - Stability Experiment 20170304?? tbd
QA - Aurora Testing tbd
QA - Beta Testing tbd
Release Date tbd

References

  • List and links for specs
    • [2] Intersection Observer Spec
    • [3] Intersection Observer API