QA/Talkback/Metrics

« Back to QA/Talkback

Maintained by Samuel Sidler

In the past, there haven't been useful metrics on the crashes reported via Talkback. We'd like to track crashes better to determine where they're happening.

Q & A

1. What's the purpose of this page?

   This page provides a few basic metrics which we're going to use in the future to track the overall crashes we're seeing and where they're happening.

2. What metrics are you tracking currently?

   To start, we'd like to track if crashes are happening inside our code or the code of a plugin, extension, or OS. This is just a basic, best guess look at where the crash is happening, but since we'll be using the same metrics going forward, it should give us a good bit of consistent data to track.

3. How are you tracking "where" crashes are happening?

   Right now, we're simply doing a quick glance at the stack trace and making a best guess at where the crash is happening. If it's happening externally, we can usually see what plug-in/extension has caused it. However, if it's happening internally, we can, fairly easily, determine where it's happening and track those. This isn't being done in this first round yet, but might be integrated with Breakpad/Socorro in the future.

4. What questions are the graphs below asking?

   We're asking a few questions...

   • How do we track, between releases, if we're improving or regressing behavior and causing more crashes?
   • Are we seeing more problems in our code or in third-party code?
   • If its in third party code, is it being caused by an extension, a plug-in, or something else?
   • [etc. More here.]

5. Are all topcrashes represented in your metrics?

   No. Because of the way Talkback topcrash reports are created, there are various null pointers that get tracked in very general places instead of by specific stacks. This makes them hard to track. In the future, if the reports below are useful, we can go back and generate them. However, for this first run, we're just going to skip them. This includes the first two topcrashes and several others lower in the list.

6. Where is the topcrash data found?

   You can view the topcrash reports for Firefox 2.0.0.x on the Talkback website. The data updates once to twice a week. We update the spreadsheet with data about every week.

7. Show me sample data, pretty please?

   Sure thing! See below! :)

Sample Data

The following is sample data taken from Firefox 2.0.0.6 and 2.0.0.7.

The chart above shows the percentage of internal vs external crashes in 2.0.0.6 vs 2.0.0.7. Included are the tables of data which is used for generating this. Note that the percentages are off in that they don't include uncategorized crashes only crashes we can identify.

This chart shows where external crashes are occurring, whether it's a plug-in, extension, or otherwise. These are best guesses, though some are fairly obvious.