QA/Telemetry

From MozillaWiki
< QA
Jump to: navigation, search

Telemetry Experiments

Feature Status Lead Engineer QA Lead Status
Telemetry Experiments Incremental Improvements Benjamin Smedberg Benjamin Smedberg OK

Project Goal

Allow Mozilla to deploy experiments to a statistically-relevant population of users and measure the results.

Project Summary

With telemetry experiments enabled, Firefox will periodically retrieve a list of experiments from a secure server if the user meets the specified condition(s). Possible conditions include:

  • Product
  • Version
  • OS
  • Channel
  • Build ID
  • Language
  • Sample ratio

Each experiment will include the following information:

  • experiment ID
  • specific conditions for running the experiment
  • experiment start date & end date
  • experiment duration
  • XPI URL & hash

Test Plan

  • QA Testing Template can be found in the following etherpad: Here
  • Telemetry Experiment test plans can be found: Here

Experiments

You can view current and past telemetry experiments Here

  • Note: Some experiments that have already been expired may not be grayed out once expired when viewing the experiment list Here. The experiment is technically already disabled and once another experiment is pushed, the list will than be updated and the expired experiment will be grayed out.
Experiment Start Date End Date Max Runtime Channel Locale(s) Sample Rate Test Plan Current Status
Reorder about:newtab tiles 01-Jan-2014 15-May-2014 5.0 days nightly N/A 25% Here Completed
Search Experiment 01-Jul-2014 30-Sep-2014 28.0 days beta en-US 50% Here Completed
Automatic Translation 27-Jun-2014 01-Aug-2014 36.0 days aurora de 100% Here Completed
OOPP container unload timeout tester 01-Jul-2014 02-Sep-2014 42.0 days beta N/A 5% Here Completed
Automatic Translation 16-Jul-2014 25-Nov-2014 126.0 days beta pl 100% Here Completed
Automatic Translation 16-Jul-2014 25-Nov-2014 126.0 days beta tr 100% Here Completed
Automatic Translation 16-Jul-2014 25-Nov-2014 126.0 days beta vi 100% Here Completed
Invisible test for experiment branch correctness 14-Aug-2014 10-Sep-2014 7.0 days nightly N/A 50% Here Completed
Automatic Translation 16-Jul-2014 25-Nov-2014 126.0 days beta es-AR 100% Here Completed
Automatic Translation 16-Jul-2014 25-Nov-2014 126.0 days beta es-ES 100% Here Completed
New Tab Data 10-Dec-2014 10-Jan-2015 7.0 days beta N/A 25% Here Completed
Flash Protected-Mode Testing 15-Dec-2014 15-Jan-2015 32.0 days beta (Win Only) N/A 10% Here Completed
Plugin Video Usage Metrics 12-Jan-2015 23-Feb-2015 60.0 days beta N/A 10% Here Completed
Search Experiment 28-Jan-2015 07-Apr-2015 30.0 days release tr 70% Here Completed
Search Experiment 25-Mar-2015 07-May-2015 30.0 days release zh-TW 50% Here Completed
Multi-process Firefox A/B test 20-Sep-2015 01-Nov-2015 10.0 days aurora N/A 100% Here Completed
Search result pages in history 10-Nov-2015 24-Nov-2015 1.0 days beta N/A 10% Here Completed
Multi-process Firefox A/B test 14-Dec-2015 04-Jan-2016 10.0 days beta N/A 15% Here Completed
Unified URLBar Experiment 11-Jan-2016 24-Feb-2016 14.0 days beta N/A 10% Here Completed
Multi-process Firefox A/B test Beta 45 25-Jan-2016 11-Feb-2016 15.0 days beta N/A 50% Here Completed
Multi-process Firefox A/B test Beta 45 2nd Experiment 12-Feb-2016 25-Feb-2016 15.0 days beta N/A 50% Here Completed
E10s Checkerboard Experiment aka DisplayPort Size Tuning 25-Feb-2016 19-Mar-2016 7.0 days nightly N/A 50% Here Completed
Multi-process Firefox A/B test 3 (Disabled) 07-Mar-2016 24-Mar-2016 15.0 days beta N/A 50% N/A Completed
Plugin Block Experiment 02-May-2016 23-May-2016 21.0 days beta N/A 10% Here Deployed

Telemetry Experiments/FHR Documentation

QA Contact Information

If you have any questions or issues regarding telemetry, you can contact me via the following methods:

Internet Relay Chat (IRC): [Quickest Response]

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Telemetry&oldid=1163570"