So you want to measure shit?

Guidelines and recommendations for how to measure what you want in the spirit of our privacy operating principles. This document is Firefox- and Gecko-centric. If you want to measure stuff without the help of Firefox or Gecko, ask the UDC for advice.

Data and Surprises

First, it's important to know why there are so many different systems for measuring stuff.

• Types of data
• People don't want to be bothered
• People don't want to be surprised

Operating Principles

As we collect information about people, their interactions with our products, our products' interaction with the web, and other tasty nuggets, we need to be sure to keep the promises we make to our users and the web: insert principles here

What options are there?

We have built up a couple of nifty tools and even have some human resources to help you measure stuff! Not every tool is right for every job, however, so take a look in our toolbox:

Test Pilot: Total User Control

We spent a lot of time designing this system to be open, unsurprising, and powerful. Do you want to measure how people touch our software? Do you want to measure what they do on the web? Are you interested in the most intricate details of how the web works or how people play with it? This tool is for you. It comes with:

• Study Isolation: Your study is isolated from other data collection, so the sensitivity of data in your study isn't affect by the others. Not only that, your study doesn't have to conform to some pre-written set of allowable data, you set the limits yourself.
• Variable Sample Size: Go deep or go wide. Your choice.
• No Surprise Guarantee: you tell people exactly what you want to know and why up front, and then show what you got before the data comes to our servers.
• Kick-ass logo: who doesn't like a fox in a flight suit?
• Experts On Board: Our user research team can help you design the study and interpret its results.
• Survey Capability: you can also ask our users questions directly!

Here's how it works: you create a study (some JS code to measure stuff) and a user-readable description of what you're trying to figure out and how. We ship the study to users who have joined the Test Pilot program, and they're presented with the description. If they want to participate, Test Pilot starts collecting data on their computer. At the end of it, they're shown what we collected and then they have the option to release the data to us (or back out at the last minute). The user is at the controls here, deciding whether or not to send us data based not only on what we say but also on what we do.

Telemetry

If you ask users for the average time their Firefox spent laying out web sites, they will turn green and run away. You have a hunch it's slower than Talos says, but you want to see exactly how slow and maybe what parts of the process are the worst. If you know that stuff, you can get your crazy-smart engineers fixing the right parts of the code instead of just guessing at what may be broken. Telemetry is for you, and it comes with:

• MOAR DATA: Telemetry is distributed with every Firefox (and every Thunderbird!) More people are given the chance to help -- and they do.
• Histograms and Counters: You can collect a histogram of performance metrics for not only the average, but other statistics on data too. You can also increase a counter to see how many times something happened since they launched the program.
• Won't Bother Users: Once users agree to help, we do it all behind the scenes so they aren't bothered.
• Trending: You can see how the data you collect changes over time, since data is collected throughout the length of study.
• EASY: You can whip together telemetry probes in a few lines of code!

Here's how it works: you come up with an idea for a new histogram or counter probe, and suggest it to the user data team. Together, we quickly identify what it is you want to know, and how knowing this will be valuable to our users, and then you land your few lines of code in mozilla-central. That code ships to nightly users and then rides the train to full releases. As the data comes in daily, you can identify what it represents and leverage it to make our products rock!

Blocklist Ping

Metrics Ping

So what should I use?

You want your data, but you don't want to surprise our users. Which tool do you pick? Here's a general rule of thumb: Telemetry: Attributes of the software including performance statistics, machine/OS-related statistics, Test Pilot: Attributes of the web platform (stuff about web sites' content), user-specific data (site history, bookmarks, interface usage, patterns of interaction) Blocklist Ping: Stuff related to items we've blocked via the blocklist only. Metrics Ping: Mission-critical metrics for our continued existance (e.g., funding audits, user abandonment)