User:Staszyk/Measuring the retention rate

Scope

• Downloads
• Instalations
• Retention

Objectives

• make use of already available data and tools
• be able to easily repeat the analysis
• collect data for all locales to identify priorities.

Method

For every release, we measure:

1. firstrun/ pageviews
2. whatsnew/ pageviews
3. downloads
4. delta updates (e.g. 2.0.0.6 -> 2.0.0.7 binary patch)
5. full installer updates

Interpretation for the period A:

A1 - new users sum total who installed Firefox 2.0.0.6

A2 - current users sum total who updated to Firefox 2.0.0.6

A3 - total download number, which include:

• new users
• current users updating manually

A4 - total number of 2.0.0.5 users who updated to 2.0.0.6

A5 - total number of older releases' users who updated to 2.0.0.6

Interpretation for the period B:

B1 - new users sum total who installed Firefox 2.0.0.7

B2 - current users sum total who updated to Firefox 2.0.0.7

B3 - total download number, which include:

• new users
• current users updating manually

B4 - total number of 2.0.0.6 users who updated to 2.0.0.7

B5 - total number of older releases' users who updated to 2.0.0.7

The analysis

(B4 - A2) gives us an approximated number of users who updated their browsers for the first time (who thus may be considered new). Alternatively, as A2 should be equal to (A4 + A5), we may use (B4 - (A4 + A5)) (if Urchin data isn't too reliable for example). We should compare the result with B1, to have some idea about the accuracy.

The exception is for the users who won't update to 2.0.0.7 (because e.g. they won't be using their browsers for 6 weeks).

Next, we compare the result of the subtraction with the total number of 2.0.0.6 downloads (A3) or the total number of the /firstrun/ pageviews (A1) (the latter may be better, as downloads will also include users who update manually, by downloading and installing Firefox by hand).

To recap: we take the total number of 2.0.0.6 updates to 2.0.0.7 and diminish it by the total number of updates to 2.0.0.6. This gives us the number of news users, who downloaded, installed and used Firefox between the 2.0.0.6 and 2.0.0.7 releases. Next, we compare it with the number of users who downloaded and installed Firefox over this period, but didn't use it afterwards (which we know, as they don't update).

Advantages of the method

1. it doesn't require any new tools nor implementations
2. it easily allows to collect information for all the locales, since we already measure everything that is needed
3. it can show some unexpected fluctuations of the conversion and retention rates over time
4. it seamlessly integrates with the traffic tracking we've been already doing,
5. it regards users' population as a whole, a and not only it's random sample (this is where some inaccuracy might come from actually, but on the other hand it allows for quick comparisons with downloads figures (thanks to the download tracking that I've been working on) and installation figures (in-product pages hits).

Disadvantages

1. if a new user downloads and installs Firefox 2.0.0.6 and uses it rarely (very rarely), he may update directly to 2.0.0.8, once it's released. In this case, we don't count them in the analysis. Normally, the releases happen every 6-8 weeks, AFAIK, but if in case of an urgent security update, this might be a problem
2. the method measures the retention rate for consecutive releases only
3. it doesn't use samples but a population as a whole, which may constitute problems with applying statistical analysis to it.