User:Polvi/InstrumentationMetrics
Contents
Process
We are following the well respected scientific method:
- Define the question
- Gather information and resources
- Form hypothesis
- Perform experiment and collect data
- Analyze data
- Interpret data and draw conclusions that serve as a starting point for new hypotheses
- Publish results
Details
Stolen from basil
There are several types of Firefox user data that would be useful to help guide product decisions and provide visibility about how users customize Firefox. There are the following "per user" data sets:
- Static metrics [S] (e.g. current number of bookmarks)
- Dynamic metrics [D] (e.g. average number of search done per session)
- Web metrics [W] (e.g. number of images per page)
- Unique metrics [U] (metrics only a browser could figure out, e.g. how often a user encounters Flash)
- Trend data of static and dynamic metrics [T] (e.g. average number of open tabs across sessions).
In addition, aggregated metrics from all user data collected can be calculated.
- Average static metrics [aS]
- Average dynamic metrics [aD]
- Average web metrics [aW]
- Average trend metrics [aT]
Other stats include low/high values across all users.
Metrics
Web interaction
Why Mozilla cares: With the browser, Mozilla can understand what the user encounters on the web.
- Which plug-ins, engines, renders, etc are used? [W]
- How many images are on the page? [W]
- How often are web standards found? [W]
Performance
Why Mozilla cares: Allows Mozilla to conduct real world performance tests.
- How quickly do pages render? [D] real world QA, will be able to track regressions in performance
- How much memory is being used? [D] QA, regression
- Number of crashes? [D] real world QA, regression tracking
Feature Use
Why Mozilla cares: Helps Mozilla determine which features are important, and which are not.
Feature use has been identified for wave #1 metrics collection
Search box
- How many times is the search box used? [init,D] validate 3rd party metrics
- instrument search box to collect a counter in daily buckets
- record which service was searched and quantity
- Which search is being used? [init,D] see how important other searches are, allows us to see how adding/removing new search boxes changes behaviour
Extensions
- How many extensions do people use? [init,S] just to know
- Addons v. other extension distributors [init,S] to see how important our distribution hub is
- parse extension update urls searching for addons (or not)
Themes
- How many users have a theme? How many? [init,S] to see how personas changes things
- number of themes
- non-default theme active?
Privacy
Are the privacy features of Firefox being taken advantage of?
- How often is "Clear private data" used? [S]
- How often does anti-phishing get triggered? [W]
Bookmarks
- How many bookmarks by type? [init,S] just to know
Pop-up blocking
- How many pop-ups are blocked? [init,W] marketing data
Tabs
- How many tabs are used? [init,D] just to know
- How are tabs opened/closed? [D]
UI use
Why Mozilla cares: Gives Mozilla insight to how users interact with the browser. With this information Mozilla can remove unneeded UI, and add UI where deemed helpful.
- How often do people decline updates? [D] UX
- How people launch Firefox? [can we even tell this?] retention
Toolbar
- How do people go back/forward on the web? [D]
Browsing
Why Mozilla cares: Will shed light on how users use the web. How do people browse the web? Will help us determine how to better serve the user.
- How many people switch the default homepage? [S] importance of the landing page
- Do users navigate with bookmarks? [D]
- How often is the search box used? [D] importance of the search box
- How long is a session? [D] importance of the browser
- Number of total sessions? [D]
- Time between sessions? [D]
- Where on page is clicked?
Environment
Why Mozilla cares: Allows Mozilla to understand the operating enviroment where browser is used. Profile information about where Firefox is used.
- How much memory is being used?
- What is the window resolution?
