Releases/Thunderbird 3.1.6/Post Mortem

From MozillaWiki
Jump to: navigation, search

Post Mortem for the Thunderbird release of 3.1.6 and 3.0.10

Use extension 4401.

  • We're looking for:
    • How well did new processes introduced go?
    • Any problem/pain points?
    • Can we reduce time spent or speed up processes?

Leading up to the release (patch landing, code freeze etc)

  • We were mainly in MoCo's hands for the patches as the bug was in core.
  • Intentionally kept out of MoCo's way during their initial build period, as they had more users, and the tagging system requires it.
    • So we slept for a few hours, getting up early.
    • Possibly need to consider getting a couple more people familiar with QA & release.
  • Working out landings on mozilla-* was difficult.
    • Both mozilla-1.9.2 and mozilla-1.9.1 had their own relbranches for Thunderbird.
    • We ended up creating new relbranches to pick up the gecko version number change.
    • Moral: Avoid relbranches whenever possible.

Build

  • 21-hours from go for build to release.
    • That was a really good effort.
  • Thunderbird 3.1.6 build process wasn't auto-triggering builds after the initial tagging.
    • Standard8 suspects he didn't leave long enough after re-configuring, as the 3.0.10 build process worked fine.
  • Still feel like the automation needs too much watching.
    • We have tinderbox pages and buildbot showing the release. Standard8 was using tinderstatus as well.
    • Still felt like it was fragile.
  • No capability to do two builds at once.
    • This held us up at the signing stages, and with getting the builds tested.
    • We had an issue with generating the 3.0.x builds which meant that the windows repacks were delayed finishing until the update generation of the 3.1 builds was completed.
    • Standard8 has a fresh proposal for being able to do two builds at once, see bug 581107 comment 2
  • We should develop an automated way to remove the old nightly builds from latest-* on FTP when we bump the nightly version.
  • Signing issues:
    • Disk space check
    • USB signing lost
      • Issue: signing process is not restartable.
  • L10n verify slow on Mac minis
    • Slow due to lack of bandwidth
    • Possibly move to xserve or co-located hardware.

QA

  • Automated update test works really well
    • Need to really do it on other OSes
  • Eagerly waited for the second round of build to do virus check.
  • Lacked signing visibility.

Security

Website (Release Notes, What's New, L10n issues etc)

  • Made a mistake with combining 3.0.x and 3.1.x updates into one revision on svn.
    • We wanted to start pushing out 3.1.x updates earlier than 3.0.x because they were ready and we could QA them whilst we were waiting.
    • However, we also needed to at least publish the release notes as they are accessible in several places.
    • Standard8 ended up separating out the release notes for 3.1.x and pushing them to production separately, and then pushing the 3.0.x ones later.

Actual Release (Pushing out, updates, etc)

Anything else

  • Standard8 and Ludovic ended up being up for the majority of 21 hours.
    • Builds were started by Standard8 early UTC.
      • This meant the main go-live actions were late in the PDT day.
      • We would still have had to wait until after MoCo released, as Firefox needed the mirror bandwidth.
    • Possible remedies/future helpers:
      • More folks in PDT land familiar with QA & release process. Don't forget to ask for help.
      • Quicker builds/more parallel processes
        • If we had got the builds done quicker, then we could have QAed them earler.
        • Maybe need to see if we can share QA out more as well, so that two releases can be QAed at the same time.
        • If we'd had to wait for Firefox, we could have had a break for a few hours, with someone calling us when it was our turn.

Actions

  • Build: Investigate parallel builds bug 581107
  • Build: Investigate better ways of knowing when builds are complete
  • Build: Investigate restartable signing process
  • Build: Move Mac update verification to a machine located within the main network to ensure it has plenty of bandwidth.
  • Standard8: Document recommendations of website generation process above.
  • Standard8/Usul: Consider getting more folks familiar with QA & release process.

Attendees

Usul, jhopkins, gozer, Standard8

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Thunderbird_3.1.6/Post_Mortem&oldid=265696"