Releases/Thunderbird 3.0b1/Post Mortem

« Back to Thunderbird 3 Beta 1 Release Page

What went well

1. The release in general - Standard8's documentation of a3 was extremely helpful
2. 38 locales
3. l10n opt-in process was a success
4. Only one respin
5. Lining up with FF 3.1 Beta helped keep strings and gecko stable
6. Build, Release, and web site changes went smoothly
7. Fewer late landings helped: make date decisions easier(?), stability

What could have gone better, and lessons learned

Leading up to the release

• Despite there being fewer late landings, several regressions from things like the js folder pane weren't discovered until very late in the cycle, including into build QA. The lessons are that:
  • we need to do (more) thorough test of big landings as soon as they land, and prior to freeze (the role of RC build QA should be to find stragglers, not to QA people's patches)
  • we should pretty much count on spending at least a week after the code freeze fixing bugs.

String Freeze

We need to make it clear that we're doing a slushy string freeze. There will be changes we want for the release after the string freeze that require string changes, and they will be managed by exception.

Conversely, we still need to be better about getting patches with string changes driven into the tree before the string freeze date.

Code Freeze

• Waited too long to decide to slip code freeze (and/or communicate the possibility that we would slip the code freeze)

• Should have realized ahead of time that the work week in MV the week before the code freeze was going to push the code freeze out.

Build

• Tagging
  • The process for picking the revisions we tag with could be a little clearer

• Building
  • No issues at all, went really smooth
  • Signing was also without issues

• L10N
  • Would be nice if shipped-locales (or a similar file) also kept track of the revisions we wanted to release with
  • Process change caused all complete MARs to be busted. rebuilding these was a bit of a pain (they'll need to be signed before Beta 2 too)

QA

• Better instructions, experienced returning testers, and QA tracking bug 463057 that anyone can cc:
  • helped everyone stay better informed and in touch
  • made for less burden on QA lead
  • improved quality of testing results
• respin bug 467813 also very helpful

Anything else

• b2 testing needs:
  • an early run at least a month before freeze of FFT for AB, accounts, and perhaps other subgroups, eg https://litmus.mozilla.org/show_test.cgi?id=5487 because of all the account and AB changes (see bug 469090 for example reason)
  • candidate testing should include full run of FFT (for each platform?). Perhaps also one platform run early, as cited above.

• We had announcement out & website updated before partial/full ftp propagation for builds.
  • Not too much of a problem, but we should try to remember that people will try and download as soon as we announce.
  • Need to make it clearer in the checklist that the process can take approx 6 hours or more (i.e. release overnight).

• Checklist wasn't fully completed.
  • IMHO we should be signing it off e.g. I think I saw Wayne asking if anyone had checked links on the website several hours after the release went out - did anyone do a full QA check of links whilst it was on staging?
  • Do we need to find an easier way of generating and handling the task list?
    • If we had a webapp (or something) where we could push buttons, to say this stage is done - add timestamp, notify relevant parties.

Actions

• AI:who:
• AI:wsmwk: Ask testers/revise doc to run build continuously if possible, not just run tests. Same for testdays. Also, they probably can't add the litmus test result #?url to bug.
• AI:wsmwk: add bug filing info to doc about setting fields (trunk, regression, etc)
• AI:wsmwk: schedule a bugday focused on trunk at some timely interval after. add to checklist.

Attendees

gozer, bienvenu, rebron, dmose, wsmwk, clarkbw