Test Pilot/2016Q4

From MozillaWiki
Jump to: navigation, search

Test Pilot 2016 Q4 Objectives and Key Results

Final Status (Dec 31) Comments
Objective 1 Grow the Test Pilot user base. .26
KR1 30% improvement in Test Pilot experiment enable rate for new users 0.50 Starting rate is 8.02% which makes our target 10.42%. On Dec 31 the average for the quarter was 8.27% (an increase of 3.1%). However, the average for just December was 12.60% (an increase of 57.1%) so we definitely did some good, but that success is weighed down by the short period we had to measure after we pushed changes. Instead of a score of 0.03 I'd like to call this 0.5 for the positive trend going into Q1 (and a note about being able to measure within a quarter).
KR2 Reduce ratio of Test Pilot users (DAU) with no experiments installed by 30% 0.0 At start of quarter is 57.4%, so this moves the target to 75%. Calculations averaged over the last week of the year show this ended up at 56.2%. There was probably some effect from the holidays, but we definitely weren't at 75%.
KR3 At least 100 different accounts have commented on the Test Pilot Discourse forum over the course of a month. 0.27 We had 27 unique folks who were active in the month from mid-oct to mid-nov.
Objective 2 Provide evidence that Test Pilot is useful in helping to make product decisions .44
KR1 Publish conclusions from three experiments and graduate them from Test Pilot 0.33 Universal Search
KR2 Launch two new experiments 0.0 Closest was Cliqz, but that got bumped to Jan. Snoozetabs was functional, but quite a bit left on polish.
KR3 Build an offboarding process 1.0 Universal Search graduated.
Objective 3 Have the ability to run experiments outside of the US Market .75
KR1 Add localization support to Test Pilot and get 80% of users using Test Pilot in their first choice language. 1.0 Averaged over Q4 en-US is 85% of requests so calling this green. Note that at the end of December en-US is down to 74.71% of requests.
KR2 Launch at least one locale-specific experiment in one non-English locale 0.5 This was Cliqz which got bumped to Jan, but due to inter-team dependencies (and, in general, Test Pilot work completed) we decided to give 0.5 here.

Updates

Non-Goals