Identity/Metrics/FxA Metrics Nutshell Summary

From MozillaWiki
Jump to: navigation, search

Quick summary of what's planned (in priority order). Written for a meeting in Feb 2014.

Infrastructure

  • fxa-auth-server (and fxa-content-server) log events locally
  • heka processes those logs and sends data to log aggregator
  • more heka processing at the log aggregator, which creates elasticsearch indexes and mappings in elasticsearch db (as well as anonymization and other aggregation)
  • kibana dashboard(s) access elasticsearch db (all metrics described below can be expressed as counts/time)
  • log aggregator, elastic search & kibana are either on shared services, or an FxA specific mirror of that setup
  • a development environment will exist that has mirror of production data (to hack on heka filters, elasticsearch mappings, kibana dashboards etc.)

0. One Metric That Matters: Total number of Firefox Accounts

  • Web page that Mark can show to anyone -- this is the top line goal for FxA
  • At minimum, could be a single number
  • Display a graph over time

1. Accounts created per day

  • Also logins per day, and other events that correspond to fxa-auth-server endpoints
  • segmented by device (Android, Desktop, FxOS, browser), service (Marketplace, Sync, WMF, Settings), locale
  • derived from fxa-auth-server logs: log line at each endpoint contains the raw information, processed by heka, data eventually lands in an elasticsearch index.
  • device is derived from user-agent (How is device name derived? What devices do we explicitly want to track/monitor? -trink)
    • "agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
    • "agent":"Mozilla/5.0 (Windows NT 6.3; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0"
    • "agent":"curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5"
  • locale is sent on request header (What locales do we explicitly want to track/monitor? -trink)
  • service is passed in on some endpoints (Is there a fixed list of services? Again, if not which ones do we explicitly want to track/monitor -trink)

2. Active daily users

  • derived from fxa-auth-server logs: # calls to cert/sign endpoint, bucketed by userid.
  • alternative could be derived from verifier logs, again bucketed by userid
  • heka can do processing to count unique userid's/day, so that userid is not stored in elastic search
    • note: The sandbox is intentionally resource constrained, it is highly unlikely it will have enough memory to track every active user in a day as usage increases. -trink.

3. UX flow analysis

  • count number of people who make it to each stage of a given UX flow (per day)
  • note that it is not important to connect events in a flow for an individual user: associate each event with a flow type (create account, login, reset password) and assume that the flow happens in the context of one day
  • derived from fxa-auth-server logs initially, can add in events from fxa-content-server logs, etc. eventually (as long as the "flow type" matches)
  • not 100% clear to me yet whether we can identify the "flow" in the server, or if we need help by information passed in from the client
  • segmentations come along for the ride: service, device, locale

Additional segmentations

  • UX would like screen size segmentations, would require info from the client/device

UX Flow Timing Information

  • It would be great to get timing information for each stage in a UX flow (ms since the "start" of the flow), but this requires help from the client.
  • Getting great visualizations from this data might be a challenge in kibana (without some custom hacking)

Additional stages for UX flows, additional UX flows

  • Would be great to track # people who have opportunity to create account but then skip it (in FTU flows)
  • Would be great to tie email validation stage into FTU flow analysis (happens at different point in time -- possibly even on a different day)
  • Create and login flows are prioritized. Once things are up and running it should be easy to add password reset, etc.

Sync specific metrics

Retrieved from "https://wiki.mozilla.org/index.php?title=Identity/Metrics/FxA_Metrics_Nutshell_Summary&oldid=934824"