THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

the tl;dr version

We've historically built large beta audiences composed primarily of early adopters and Firefox Faithful, and primarily used this audience as a testbed for sending us crash data which we then interpret to understand the effects of changes to our product that can't be adequately tested by our QA group due to limited coverage.

For Firefox 4 we need to take action to build an audience that is representative of our total user base (for QA coverage) as well as one that is intentionally inclusive of early influencers and web developers who we want to take advantage of our new features. We also need to build better mechanisms for accepting and processing quantitative and qualitative feedback.

the past

Why do we ship betas?

• build excitement and buzz
• get user feedback on new user interface features, perception of performance
• allow developers to experiment with new technology features (web developers, plugin developers, add-on developers)
• get wide compatibility testing with web sites, plugins, operating systems, hardware, software
• get wide QA testing by having many eyes on the product

Most of our current reasons for shipping betas focus on involving those users as part of a broad testing community since our QA group is so small, respectively. To date we've benefitted from betas somewhat as a marketing tool (we get a press cycle out of each one) but more as a user feedback tool.

Who do we ship to?

• tech press readers, early adopters
• Mozilla fans and close followers

In the past our outreach here has been less focused than we need it to be. We announce betas and invite the world to come join our testing group, but we don't do any analysis to see who ends up being part of our beta group, nor do we issue individual invitations to known and notable contributors.

My feeling is that we need the group to be representative of our actual user base (if we're using betas, as expressed above, to be an extended QA group) as well as inclusive of early adopters and developers who will be building websites using the new technologies. Not sure how best to accomplish this, but we need some stronger outreach when we launch betas to evangelize and get people included in the beta program.

Historically we get about 100k users on the beta when we first release it, and about 200k users are added every month. We'll need that ramp to go much faster this time around.

How do we talk about the beta?

• release notes
• firstrun and whatsnew page
• Mozilla Developer News and announcements

Back in Firefox 3 time, I began to realize that if we wrote the release notes and announcements in a slightly more tech-press friendly way, they would copy and paste what we were writing. So I did that, and we got a lot of focus on the specific items that were in a beta release that were new and notable.

For Firefox 3.5 we started to modify the firstrun and whatsnew pages with specific messages to testers, and to build excitement and emotional attachment to being in the beta group. We tried to make it easier for users to provide qualitative feedback, but the experience remained pretty poor (http://hendrix.mozilla.org) and underused.

How do we gather feedback?

• crash reports
• bug reports
• mozilla.feedback newsgroup (fed by hendrix.mozilla.org)
• other newsgroups (dev-apps-firefox, dev-platform)
• web forums (mozillazine, getsatisfaction, reddit, digg, facebook)

Again, aside from crash and bug reports, the collection effort has been unfocused. My feeling is that the spread of feedback represents the relative lack of ease of providing feedback directly to Mozilla itself; bug reports and even the mozilla.feedback newsgroup aren't the best user experience.

There have been many plans to build better feedback tools into Firefox, and while I support the idea I don't think we have a ton of time here, nor do I tend to think that creating new tools is always the best solution. Below I'll describe some of the projects underway to help us collect feedback in a more organized way.

the future

Here's what I'm thinking we need to do in order to address the issues identified in the analysis above. We need someone to own and drive this project (me, if needs be) and a bunch of contribution from the marketing and web development groups, as well as a more focused approach to beta milestones in which we know what feedback we'd like to collect from users before we ship them the milestone.

Goals

• gather quantitative data to assist with product design, engineering and qa decisions
• gather qualitative data to assist with product design and qa decisions
• gather feedback from web developers about new technologies
• build buzz around new product features and innovations
• build engagement with testing group and have them evangelize the new version

Projects Underway

• changing our updater code so we can send people to pages other than just firstrun or whatsnew when they upgrade - this is done, and gives us a lot more flexibility than we had. We can send someone to a different page based on what upgrade they've gotten, what OS they're running, etc. We can craft messages for each of those websites. Localization might be hard, here. (involved: rstrong, lmesa)
• integrating Test Pilot in beta builds to gather quantitative information. We could also use this for qualitative feedback. (involved: jinghua, jono)
• creating a JavaScript bookmarklet which can be included in the Bookmark Toolbar to provide a better lightweight "report a problem" or "send feedback" experience (involved: prouget, aakash, faaborg)

The other projects that I'm aware of are the ones that Slater and Laura are asking about for creating better product page resources on www.mozilla.com to describe what's happening in the beta, and Mozilla Hacks is looking to try and roll through the beta cycle with demos of the new web technology features.

Next Steps

• tie this all together into a plan with audience targets and goals
• figure out who's going to make sure we're co-ordinated on messaging, terms, and timing
• figure out if the bookmarklet + test pilot is enough for qualitative feedback, or if we need something more
• getting an engineering and qa liaison for Firefox so that we're pulling the sort of feedback that will be helpful for the decisions that they need to make