Test Pilot/2016Q3
From MozillaWiki
Test Pilot Q3 Objectives and Key Results
Final Score: 0.61
| Final Status (Sep 30) | Comments | ||
|---|---|---|---|
| Objective 1 | Grow the Test Pilot user base. | .72 | |
| KR1 | Grow total Test Pilot user base by 66% | 1.0 | We started the quarter with around 2000 DAU and on Sep 26th we had 21120. |
| KR2 | Add localization support to Test Pilot and get 80% of users using Test Pilot in their first choice language. | 0.0 | We deprioritized this completely. |
| KR3 | All new Test Pilot experiments support localization when appropriate. | 1.0 | We're marking this as a 1.0 because for our experiments it wasn't appropriate, because L10n was deprioritized (see KR2). |
| KR4 | Get 6% of our legacy users into the new Test Pilot program. | 0.6 | The peak usage over the last 30 days before the migration started was 735932 users, making our goal 44156 users. We ended up with 26254 new Test Pilot installs directly from this campaign, making for a 3.6% conversion. There are still 170k people with the old add-on installed. |
| KR5 | Remove Firefox Accounts integration. | 1.0 | |
| Objective 2 | Make it easy for existing Test Pilot users to learn about new experiments and start using them. | .77 | |
| KR1 | Get new experiment within 24 hours of launch at a rate 10% higher than our baseline engagement ratio. | 1.0 | this should have been "get x% of our users using a new experiment in y hours." Since the engagement ratio is based off of DAU and MAU it's not what we should use in the first 24 hours. NoMore404s had 10k users in the first week. |
| KR2 | Implement a notification system in Test Pilot | 0.8 | We did implement a push notification system, but it has some problems (weird empty messages) that we haven't solved and don't know how. We also added badging to our icon which has proven to be very successful for engagement (far more than the push notifications). |
| KR3 | 5% of active Test Pilot users share an experiment socially | 0.5 | This is really hard to measure as people could just be copying/pasting links to their social media. We've seen 363 "share" clicks from our site in the past 3 days, but weren't measuring earlier than that. 5% of our DAU (21120) is 1056 users. Q3 had 1500 sessions starting from facebook, 635 from twitter. |
| Objective 3 | Demonstrate that the Test Pilot experiment life cycle is valuable and sustainable. | .35 | |
| KR1 | Ensure at least one experiment graduates | 0.0 | |
| KR2 | Ensure at least one new experiment is publicly available | 4.0 | We actually shipped 4 new experiments! No More 404s, Tracking Protection, Min Vid, Page Shot (4.0 for good feelings, but 1.0 for actual score) |
| KR3 | Publish conclusions | 0.0 | |
| KR4 | Build an offboarding process | 0.38 | This is being tracked in Milestone TXP-18. We've done 3/8 of the issues so far. |
Updates
- Mid-Quarter check-in
- Q3 Review was on Oct 7, 2016
Non-Goals
The below were discussed as part of our goal setting but were not selected. They are included here for posterity and/or future planning.
- Objective: Expose standardized experiment performance data in an easily consumable format
- Key Results:
- Build a standardized “report card” with high level statistics
- Send an automated weekly summary email to executives
