Connected Devices/Projects/Metrics

From MozillaWiki
Jump to: navigation, search

With ideas becoming more mature during the innovation process - and eventually becoming products - monitoring progress, measuring market fit or understanding the customer lifecycle becomes more and more important for Product teams. This is also known as Innovation Accounting. Even early in the innovation and product development process, testing hypotheses through experiments and creating insights from the data collected is an essential part of the Lean Startup approach.

For all of the Connected Devices products, collecting metrics is essential to guide decision making during the gating process. One problem with the set of Connected Devices products is that these products are going to be very different in terms of functionalities and technologies. Unlike FirefoxOS where one set of metrics was good for all supported devices, we will need very specific solutions for each product that will ideally follow a common structure of Innovation Accounting.

In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).

Rationale

Nearly all the Connected Devices trains are likely, at some point of time, to require collection of data. Data gathered from the projects is extremely useful to understand how people use our products but also to check how changes implemented in the Products affect how people use them.

In summary The idea of the CD Metrics Framework is providing all the CD trains an easy way to:

  • Collect metrics in the products
  • Get Access to the metrics information
  • Visualize the metrics information

The Framework

After firstly assessing Mozilla Telemetry Pipeline, and later several third party solutions the team decided to go for Google Analytics (GA) as the Framework for Collecting Metrics. The key reasons are:

  • Technical Fit: Google Analytics meets our needs both in terms of metric collections, accessing to raw data and visualizing it.
  • Adoption: The integration of GA with CD trains seemed to be the faster-one (e.g. in one day we managed to get some results in one of the trains: SmartHome).
  • Legally/Commercially: GA has been already vetted in Mozilla and we are already paying for a premium service

It’s important to clarify that the framework is using the Google Analytics Measurement Protocol that is more appropriate for IoT projects than Classic Google Analytics which is more e-Commerce related. The metrics are stored by GA in a Google BigQuery instance.

The Data can be later on retrieved via the Google Analytics Console that is very convenient to check if the data is being received and getting simple charts about the trains (e.g. how many users are currently “live”?).

In addition to the Google Analytics Console, the trains could also use Google Analytics Data Studio.

Furthermore, Re:Dash can be used to formulate SQL queries against the BigQuery Database. Re:Dash is a third party tool that can be used to get more sophisticated reports. Additional tools exist to create reports, that could be used by the CD trains, if they deemed they are necessary.

Additional Tools and Activities

Apart from identifying what is the best toolchain and framework for being used in CD trains, the CD Metrics Team has been working in different activities to ease adoption, ensure consistency and help with the visualisation activities.

  • Ease adoption: By Implementing a library for Rust and JS that simplifies the adoption of Google Analytics by the trains. (Note: The Rust library could be also used in other programming languages)
  • Ensure consistency: By documenting the parameters that can be used for collection of events
  • Ease visualisation: By evaluating Reporting/visualization tools to ensure that metric data can be easily analyzed by the trains and providing some example charts.

In other words, we have done different tasks to ensure all the trains can onboard easily and send information in a consistent way that can be used to get easily dashboards/reports that are also consistent cross-train.

Project Status and onboarding on the metrics framework

CD Metrics is a framework to ease data collection and some tools to ease data manipulation and visualization. The target of the metrics team is not creating reports on behalf of the trains but giving them access to the data and support to getting started with the framework tools. I.e. data ownership relies on the trains themselves: they are the ones that have full access to the data so they can view/query/report as they see fit.

A first version of the framework is already available, so any CD train could start the on boarding now. For doing so, we have developed a set of guidelines to help with the on boarding process to ingrate CD Metrics in your train: CD Metrics Integration

Why Not Telemetry?

The initial approach we explored was leveraging the unified Telemetry Pipeline to send data to the metrics infrastructure and process it. The idea was reusing an architecture already existing and widely used at Mozilla. In that way could rely on the capability the platform has for validating, sanitizing and even automatically analyzing device data.

After some months working to ensure the information sent was consistent, relying on the Telemetry "Ping" mechanism already used in many Mozilla products and defining one common Connected Devices Ping Format proposal that could be implemented by each train, we changed our approach.

The benefits of using Telemetry are clear for Gecko-based projects, but this might not apply to the CD trains:

  • The CD projects are intended to be initially small projects and Google Analytics and other 3rd party providers offer frameworks with lower learning and adoption curves. Using Telemetry could be an overkill to them.
  • Gecko and Telemetry are very coupled and some of the Telemetry advantages require Gecko. Indeed, some other projects in Mozilla that are not using Gecko have already started to use alternative solutions for Metrics.

Project Management

We use Trello board for project management. All ongoing tasks are listed there.

Sprint Backlog

Development process

Repositories

IRC

You can find us on irc.mozilla.org in the #cd-metrics channel.

Methodology

The project is in a very mature state, as the Framework is already been defined and tested with some projects (as project Haiku and project Magnet) so we have reduced the frequency of our meetings/standups... but we maintain our weekly meetings in order check if new requirements emerge or any other project need our help to integrate metrics.

  • Weekly meeting in CD_Metrics Vidyo room on Monday at 5:30 PM UTC (9:30 AM, PST), notes in Google doc

Calendar

The CD Metrics team has a public calendar with every team meeting Calendar

Adding Metrics to your Project/Train

The following link has instructions and details for adding metrics to your Project/train

Team

Retrieved from "https://wiki.mozilla.org/index.php?title=Connected_Devices/Projects/Metrics&oldid=1156666"