Thunderbird:UX:Test Pilot

Test Pilot For Thunderbird

We need more information about how our users work. To that end, we're doing a Thunderbird port of Test Pilot. If you're interested in helping with Test Pilot or Test Pilot studies for Thunderbird, and mailto:rebron@mozilla.com has offered to help you get started.

Code

Test Pilot for Thunderbird repository

Proposed Studies

Some ideas for studies

• Day in the Life of a Thunderbird user
• Menu and Toolbar usage study
• Main Window usage study
• Address book usage study

These studies will further be defined and deployed from the main Labs site: https://testpilot.mozillalabs.com/testcases/

Privacy Policy

• Thunderbird Privacy Policy
• Firefox Test Pilot Privacy Policy

The Test Pilot for Thunderbird Privacy Policy is in draft.

Test Pilot User Experience

Test Pilot will be bundled into Dailies, Earlybird, Beta, and then to all Thunderbird users starting with Thunderbird 9. Test Pilot for Thunderbird user experience will match Firefox and will give users the ability to participate in Test Pilot studies, surveys, and view results.

Access Points

There are three access points for Test Pilot for Thunderbird, 1) the Tools Windows, 2) Add-on Manager, and 3) Door Hanger notifications. There is no 'Add-on bar' for Thunderbird.

Tools Menu-Item

(note that the menu in the screenshot above is out of date, you now access Test Pilot from Tools | Feedback instead of Tools | Test Pilot)

• Main access point to Test Pilot will be through the Tools Menu Item. From here, users can go into the main window of Test Pilot of make quick selections to toggle off notifications.

Door Hanger Notification

• The Door Hanger notification is when we'll notify users when a new study is available. The user will click through to participate in the study. If the user clicks through the Privacy Policy link, the Door Hanger notification should remain. Clicking through to the study, will begin the Test Pilot study for the user. Users will be able to click, 'I don't want to participate in this study'.
• The Door Hanger notification will hang off the right most portion off the first Tab label.

Main Window - Current Studies

• This is the first screen that displays when users select Test Pilot.
• This is where users will be able to see their current studies, cancel studies, or interact with surveys.

Finished Studies window

• This is where users can view all the studies where they participated. Users can also click through to learn more about the purpose of the studies.

Study Findings window

• This is where users will be able to view results of the studies.

Settings window

• This is where users can set their preferences for notifications and sending us their data automatically.

Possible through automation

Thunderbird can collect these metrics automatically and without user intervention on machines where the user has installed the Test Pilot plugin.

• Performance
  • Number of messages per folder, folder size, number of folder, total size,...
  • Message load
  • Attachments, size
• Search
  • What kind of searches are used: what combination of search criteria, Global Search VS Quick filter?
• Crypto
  • SSL: Gather telemetry about the most popular ISPs that use SSL to protect communications with clients, namely SSL/IMAP and SSL/SMTP. Examples of data to collect would include ISPs that:
    • use self-signed certificates
    • use expired certificates
    • use revoked certificates (or have not set up revocation properly)
    • send incomplete cert chains
    • ask for SSL client-auth certs
    • use otherwise invalid certs (anything that generates an error)
  • S/MIME: Gather information about S/MIME usage and errors. (This might be tricky for a number of reasons. Need to think about this some more.)

Requires User Action

These ideas require the user to complete a survey of some sort.

• Backup
  • Find out what kind of email backup people would want, if any
  • Example Questions:
    • How often do you back up your email?
    • Have you ever lost old emails before?
    • How much would email backup be worth to you?

Which pref are really used

In order to clean up the UI (as Firefox do in Fx4), we need to know which pref have been changed by the user from their default value to a new value. This will allow us to track the prefs that are maybe not understandable or never used.For instance, do the user care about "marking the junk message as read" pref?

Other places we've asked

The Thunderbird UI Blog