QA/Telemetry/TelemetryTestCases

From MozillaWiki
< QA‎ | Telemetry
Jump to: navigation, search

Telemetry Experiments

Test Cases

  • Select Telemetry under the Products section on MozTrap <- [Currently in progress]

Because Telemetry Experiment sometimes involves multiple staging servers, building your own staging server, changing several preferences, hacking at the manifest before/after you build the experiment, coordinating with the developer, it's really difficult to create a step by step "test case". For each experiment, I've listed all the bugs where all of the testing has occurred in. These bug include a variety of test cases that could be used as examples for other similar experiments.

Experiments

Reorder about:newtab tiles

  • Metabug: 973990 - (telex) Telemetry Experiments v1 metabug

There wasn't really a main bug where most of the testing was done as the telemetry experiments infrastructure was still being developed and improved during this time. I've listed all of the bugs that have test cases associated with them for future reference.

Full Query
ID Summary Priority Status
973998 Telemetry experiments: initial server -- VERIFIED
985084 Experiment add-ons should be disabled at startup -- VERIFIED
986530 Deploy a test experiment to the nightly population -- VERIFIED
986677 Include time left in experiment in addon-manager P2 VERIFIED
988710 Telemetry experiments: local staging server not working with nightly -- VERIFIED
988873 Telemetry experiments: enable the experiments feature -- VERIFIED
990111 Addon Provider for previously-active experiments -- VERIFIED
992232 telemetry-experiment-server: change deployment repo and script -- VERIFIED
992258 Trigger a refresh of the experiments view in the addon manager when experiments change -- VERIFIED
994727 Telemetry experiments: Test and fix how disabling the feature works. -- VERIFIED
997188 testsuite connects to telemetry-experiment.cdn.mozilla.net -- VERIFIED
1004506 Telemetry experiments: NaN days remaining when having multiple experiments under about:addons -- VERIFIED
1005270 Trigger experiments initialization on opening the addon manager UI -- VERIFIED
1007663 Avoid showing empty experiments list -- VERIFIED

14 Total; 0 Open (0%); 0 Resolved (0%); 14 Verified (100%);


Search Experiment

  • Metabug: 1029746 - [meta] Search experiment
  • Testing was done in: 1029818 - [Search experiment] QA experiment add-on and sign-off for deployment
Full Query
ID Summary Priority Status
1017806 Telemetry Experiments: builtin support for multiple branches of an experiment -- VERIFIED
1029189 [UX] Design experience for new default search test -- RESOLVED
1029776 [Search experiment] Create experiment add-on and define user group (cell) on installation -- RESOLVED
1029792 [Search experiment] Implement notification UI -- RESOLVED
1029799 [Search experiment] Define experiment parameters -- VERIFIED
1029807 [Search experiment] Host update manifest -- RESOLVED
1029810 [Search experiment] Add the search experiment to the experiments server -- VERIFIED
1029818 [Search experiment] QA experiment add-on and sign-off for deployment -- VERIFIED
1030864 [Search experiment] Notify user about the end of the experiment -- VERIFIED
1031320 [Search experiment] Display notification UI for the correct groups -- RESOLVED
1033125 Deploy telemetry-experiment-server for search-beta experiment -- RESOLVED
1036719 [Search experiment] Search experiment should only target en-US -- VERIFIED

12 Total; 0 Open (0%); 6 Resolved (50%); 6 Verified (50%);


Automatic Translation Experiment [de]

  • Metabug: 973271 - Automatic Translation
  • Testing was done in: 1022411 - Build Telemetry Experiment for German translation trial
Full Query
ID Summary Priority Status
1021921 Story Breakdown - [Translation] Launch pre-trial of instant translation -- VERIFIED
1022410 Ensure all translation-related strings in Fx32 Aurora are localized in German -- VERIFIED
1022411 Build Telemetry Experiment for German translation trial -- VERIFIED
1022413 Update list of languages supported by automatic translation -- RESOLVED
1022856 implement translation provider attribution -- VERIFIED
1026853 Experiment is displayed as "pending removal" in detailed view -- VERIFIED
1032265 Deploy telemetry-experiment-server for translation-aurora experiment -- VERIFIED

7 Total; 0 Open (0%); 1 Resolved (14.29%); 6 Verified (85.71%);


OOPP container unload timeout tester

  • Metabug: 1018200 - Run an experiment in 32 beta to determine the best value for dom.ipc.plugins.unloadTimeoutSecs
  • Testing was done in: 1018200 - Run an experiment in 32 beta to determine the best value for dom.ipc.plugins.unloadTimeoutSecs
Full Query
ID Summary Priority Status
1007490 Option for timeout until an idle plugin-container is closed P3 RESOLVED
1011136 Add telemetry measure for plugin process launch count and timing -- RESOLVED
1017806 Telemetry Experiments: builtin support for multiple branches of an experiment -- VERIFIED
1018200 Run an experiment in 32 beta to determine the best value for dom.ipc.plugins.unloadTimeoutSecs -- VERIFIED
1042184 Deploy telemetry-experiment-server for translation + oopp unload experiments -- VERIFIED

5 Total; 0 Open (0%); 2 Resolved (40%); 3 Verified (60%);


Automatic Translation Experiment [pl, tr, vi]

  • Metabug: 973271 - Automatic Translation
  • Testing was done in: 1041598 - QA experiment for Vi/Tr/Po translation trial on beta
Full Query
ID Summary Priority Status
1035333 Build Telemetry Experiment for Vi/Pl/Tr translation trial on beta -- RESOLVED
1041598 QA experiment for Vi/Tr/Po translation trial on beta -- VERIFIED
1042184 Deploy telemetry-experiment-server for translation + oopp unload experiments -- VERIFIED

3 Total; 0 Open (0%); 1 Resolved (33.33%); 2 Verified (66.67%);


Automatic Translation Experiment [es-AR, es-ES]

  • Metabug: 1072741 - Build Telemetry Experiment for Spain/Argentina translation trial on beta
  • Testing was done in: 1082207 - QA experiment for es-ES/es-AR translation trial on beta
Full Query
ID Summary Priority Status
1072741 Build Telemetry Experiment for Spain/Argentina translation trial on beta -- RESOLVED
1082207 QA experiment for es-ES/es-AR translation trial on beta -- VERIFIED
1087039 Deploy telemetry-experiment-server for translation Es/Ar beta trial -- VERIFIED

3 Total; 0 Open (0%); 1 Resolved (33.33%); 2 Verified (66.67%);


New Tab Data

  • Metabug: 1062708 - Build Telemetry Experiment for new tab url data
Full Query
ID Summary Priority Status
1042214 Send detailed view ping of shown tiles -- RESOLVED
1062683 Remove urls from new tab pings -- RESOLVED
1062708 Build Telemetry Experiment for new tab url data -- RESOLVED
1111733 Test local UDP transfers when bigger payloads are sent by Telemetry Experiments -- RESOLVED

4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);


Flash Protected-Mode Testing

Full Query
ID Summary Priority Status
1108035 Hook functions in plugin-container for Flash to configure Flash protected mode off -- VERIFIED
1110215 Run an experiment: disable Flash protected mode and see how that affects the plugin crash/hang and BHR data -- RESOLVED
1110818 Measure subprocess crash rates in telemetry -- RESOLVED
1111791 Telemetry report: effect of the Flash protected-mode experiment -- RESOLVED
1112677 Deploy telemetry-experiment-server for flash protected-mode -- VERIFIED

5 Total; 0 Open (0%); 3 Resolved (60%); 2 Verified (40%);


Plugin Video Usage Metrics

  • Metabug: 1108668 - Telemetry experiment: determine the most popular Flash video sites
Full Query
ID Summary Priority Status
1108668 Telemetry experiment: determine the most popular Flash video sites -- RESOLVED
1119291 Telemetry ingestion: Flash video site ping -- RESOLVED
1119302 Implement nsIContentPolicy.shouldProcess for plugin resource loads -- RESOLVED
1122537 Deploy telemetry-experiments for bug 1108668 (Flash video telemetry) -- RESOLVED
1123888 Telemetry report: results of common Flash video sites -- RESOLVED

5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);


Search Experiment [tr]

  • Metabug: 1127076 - [experiment] test Yandex in default for Turkey
  • Testing was done in: 1127076 - [experiment] test Yandex in default for Turkey
Full Query
ID Summary Priority Status
1131864 Deploy telemetry-experiment-server for search engine experiment -- RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);


  • Experiment Staging Bug : 1131864 - Deploy telemetry-experiment-server for search engine experiment

Search Experiment [zh-TW]

  • Metabug: 1144367 - [experiment] test Yahoo in default for HK/TW (zh-TW)
  • Testing was done in: 1144367 - [experiment] test Yahoo in default for HK/TW (zh-TW)

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


  • Experiment Staging Bug : 1149264 - Deploy telemetry-experiment-server for search engine experiment

Multi-process Firefox A/B test

  • Metabug: 1193089 - [e10s][meta] Telemetry e10s experiment
  • Testing results can be found: Here
    • QA instructions can be found: Here
  • Experiment Staging Bug : 1217613 - Deploy telemetry-experiment-server for Aurora e10s experiment

Search result pages in history

  • Metabug: 1174937 - Telemetry experiments: share of searches recorded in Places that resolve to major web SRPs
  • Testing results can be found: Here
    • QA instructions can be found: Here
  • Experiment Staging Bug : 1214897 - Deploy telemetry-experiment-server for Search beahvior recorded in Places

e10s A/B on BETA

  • Note: the experiment was moved from beta43 to beta44
  • Metabug: 1222894 - [meta] Run an e10s A/B experiment on Beta 43
    • Testing results for beta43 can be found: Here
  • Metabug: 1229104 - [meta] Run an e10s A/B experiment on Beta 44
    • Testing results for beta44 can be found: Here
  • Experiment Staging Bug : 1232544 - Deploy telemetry-experiment-server for Beta e10s experiment

Unified URLBar Experiment

  • Metabug: 1219505 - Telemetry experiment for unified urlbar-searchbar with one-off search buttons
    • Testing results for beta43 can be found: Here

Multi-process Firefox A/B Experiment Beta

  • Metabug: 1238802 - Telemetry experiment for Multi-process Firefox A/B
    • Testing results for Beta 45 (45.0b1) can be found: Here

Multi-process Firefox A/B Experiment Beta 2nd Experiment

  • Metabug: 1244187 - Telemetry experiment for Multi-process Firefox A/B 2nd Experiment (E10S & Addons)
    • Testing results for Beta 45 (45.0b4) can be found: Here


DisplayPort Size Tuning

  • Metabug: 1251052 - Telemetry experiment for Measuring the effect of different displayport sizes on checkerboarding.
    • Testing results for Nightly 47 (47.0a1) can be found: Here


Plugin Blocking

  • Metabug: 1248813 - Telemetry experiment to measure whether blocking some Flash content will reduce browser jank and crash rate.
    • Testing results for Beta 47.0b3 can be found: Here


Unified Urlbar Shield Study

  • Metabug: xxxxxxx - Telemetry experiment to ensure telemetry data for Unified URLBar Shield Study.
    • Testing results for Unbranded Beta can be found: Here
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Telemetry/TelemetryTestCases&oldid=1155003"