This PRD is just a draft.

Overview

Rather than collecting metrics by hand and then entering them into a Google Docs spreadsheet, we should be collecting metrics on SUMO and provide a metrics dashboard to make analysis easier. The metrics we're looking to collect are [listed here].

Absolute must-haves

• Metrics should be able to be categorized so that they don't just all pile up on a single page
• Individual metrics may have different date ranges (e.g. last week, last month, etc.) and should be clearly labeled as such
  • For example, we might want to look at the traffic of the site on weekly basis, while we would want to look at CSAT trends on a monthly basis.
• Flexible framework where new metrics can easily be added. Ideally, there would be an admin panel where all metrics can be added/edited/removed dynamically
  • Properties of an individual metrics object would include title, display format (%, number, etc), category, and period length (e.g. day, week, month)
• Historic storage of metrics -- similar to how we're saving metrics over time on the Weekly Metrics spreadsheet
  • The data needs to be stored in a way that allows us plot the data (piecharts, histograms) in the future. E.g. stored in the db or in xml files or similar.

Should be considered for inclusion

• TikiWiki integration -- a lot of the data is coming straight from the TikiWiki db. Then there is a portion of data coming from site analytics software (in our case Omniture). Having the dashboard integrated in TikiWiki would simplify portability to other SUMO installations.
• (Firefox Support specific) Automatic import of data from Omniture -- there is an API to pull data from Omniture, so this should be possible. Alternatively, Omniture might be able to export/send data.
• Dynamic reporting -- i.e. specify what you want and what time period and have it generate just that report rather than only showing static reports
• Tabbed/folded interface (because otherwise it'd be very cluttered, or the page of stats will be very long)

Future ideas (not formal requirements)

• Graphing of data over time -- piecharts, histograms, etc.
• Access to individual results (specific pages or specific locales or specific users -- may need restrictions).
  • This means that you can say, click on "Firefox crashes when you open it" and see how many people get there via search vs from forums vs from front page vs Google and also what the CSAT data for that page was like and also how these things trended over time
  • Or see how CSAT data is in /fr/ vs /de/ or something. Tracking by users is necessary for karma anyway so setting up a way to track it here would give us a head start on that.

Needs decision or investigation

• Do we have any similar dashboard projects done in the past?
  • Any code we can reuse?
  • Any previous work on scope/implementation or good practices we can apply here?
• Static or dynamic data?
  • Do we store the data we periodically gather in a way that enables us to select date ranges for the data we're collecting?
  • Or are we just going for a list of SQL queries that are cron-jobbed daily/weekly/monthly and just stored statically?
• TikiWiki integration?
  • Having this integrated would make it easy for other installations of SUMO (e.g. Fennec, Thunderbird) to use the same features. Also, a lot of the data is coming from TikiWiki (although we probably should be using the db copy of the sumotools server for performance reasons; however that could be a pref for other SUMO installations).
  • Other SUMO installations might not have Omniture -- we would need an interface between website metrics and TikiWiki metrics so Omniture can easily be replaced with e.g. Urchin, Google Analytics, or whatever other sites might be using
    • We could store the data we get from the Omniture API separately so that the only interfacing can be done through an import script for Urchin or Google Analytics
    • Alternately, since the data may actually be different, the queries that rely on them can be dropped and replaced by appropriate queries by the individual implementation.

Storage

• We should figure out how to store this data. It doesn't seem a good idea to query the DB each time the page is generated (even if that's weekly) and have to requery to get old data. Instead, we should save the metrics in some sort of table or database structure that allows easy access.
• The exact schema used is pretty important and something we should all agree on first because it's likely that it will be hard to change down the line.
• Having per-article metrics that also have historical data would be a great idea but needs a good database schema to make for easy querying and data extraction. That way if we want to go back and look at top 10 articles in a given metric rather than just top 5, it's doable. This probably needs its own table(s).
• We should (I should) figure out how much data we're talking about here:
  • We have about 12 metrics (may be more in the future) per article, about 250 articles, daily. That's 3000 numbers to store (rows?) per day if we shove everything into one table. Does this scale? Is this idea feasible?
• Since none of the data refreshes more than daily, it may also be possible to shunt the current data run into a single static page that updates daily while still preserving the underlying data.

Mockup?

• Still entirely a draft but here's an initial idea (take 3):

• Tabs that correspond to Support/MetricsDashboardPRD/Metrics.
  • Groups of metrics are in tabs
    • Extra [+] button for easy addition of metrics (if possible)
  • Clear header showing date range these metrics apply for.
    • Change date range using a dropdown. Certain metrics are only collected weekly (marked with (-) in the metrics list) or weekly and monthly (marked with @) and thus can't be displayed when custom date ranges are used.
  • Metrics are shown in a list on each tab
    • Clicking on the metric expands it to show 5 rows of historical data, as a list. Also possible: linking to a second page.
    • Provide a link at the bottom of this table to get complete data.
    • Provide data and a change from the previous week: green for improving, red for worsening.
  • List based metrics are shown as a list with appropriate sorting data.