Releases/Firefox 3.0.1/Post-Mortem

This page is for notes from the Firefox 2.0.0.16/3.0.1 post-mortem meeting held on [to be decided].

Meeting Notes

From the dev/security view:

• -- We let the press dictate this release.
  • Action: Come up with estimated releases for next 4 months or so. This allows us to publicly say when such an exploit will be released. (ss and beltzner)
• -- The bug itself was revealed on release day, didn't decide to make it a short release until a month later.
  • Action: Same as above.

From the build view:

• -- Snippet generation was a little weird for 3.0.1 with RC2/3
  • Action: None. This was just a consequence of having a Mac-only RC3 of Firefox 3. Shouldn't be a problem in the future for update snippets.
• -- Pushed to mirrors and it stalled for some reason (2.0.0.16). Took 8 hours to get over 50%.
  • Action: In the future, we need to tell IT when we notice the problem. This will allow them to act on it. If it's the rsync issue on the mirrors, we can switch DNS servers. This time, we waited because we had extra time.
• Can we push to mirrors earlier?
  • Action: Yes. But we need to add an index.html file on the servers for minor releases. (ss)
• ++ No real humps along the way.

From QA of 2.0.0.x:

• ++ Some things got done earlier than expected. No real problems.

From QA of 3.0.x:

• -- Hand off from website changes were a little confusing.
  • Action: Need clearer communication from QA to build to show what "sign off on bits" means. (ss)
• -- Web page push takes too long.
  • Action: IT is making it easier and faster to push things. They can also push things immediately for us in the future. (ss, mrz)
• -- QA of web pages isn't as thorough as it could be.
  • Action: We need to make sure the entire release notes are looked at. (tchung)
• -- Some extensions failed to update properly (3.0.1)
  • Action: For minor releases, none. This is a consequence of 3.0.1 being the first minor update to 3.0. Some extensions had set "3.0" as their maxVersion instead of "3.0.*". We should've done an AMO check for such extensions in advance (proactive instead of reactive).
• ++ Our beta period caught the extension problem.
• -- Beta release notes didn't include details of what changed.
  • Action: In the future, we'll have a list for beta users. (ss)
• ++ No real issues with QA.

Agenda

• In sync releases? We kinda sorta did in-sync releases this time around but we need to decide for sure if we want to do them in the future. Pros: easier for disclosure/PR, soon only one team will own both releases. Cons: means we can't ship as they're ready, soon only one team will own both releases.

2.0.0.16-specific issues

3.0.1-specific issues

• On the beta release day, we had the following issues:
  • AUS snippets weren't showing bug 430737. oremj was out and temporarily fixing this issue (refreshing the files) wasn't documented well. How do we keep this from happening again?
  • Website changes that were pushed didn't appear in Europe because of an issue with the nl colo syncing. What happened and how do we keep it from happening?
• On release day:
  • Had issues with snippets appearing again
  • Had issues with the nl mirror sync again
  • I must have hit a bad mirror because at the very start of going live I could not get my Intel Mac Tiger machine to update. I experienced updates hanging, and it seemed as if the partial from 3.0->3.01 actually failed and went to the complete but I did not get notification. Unfortunately I did not have Live HTTP Headers turned on so I could not identify the bad mirror. Subsequent updates on the same machine a bit later were ok. The odd thing is I have never seen such a slow update when testing the various .dot releases. [marcia]