Releases/Thunderbird 12.0/BuildNotes
Until we're about to ship this page will serve as a spot to put down a rough checklist of what needs to be done to ship.
Contents
- 1 General Notes
- 2 Release Engineers
- 3 Bugs Hit
- 4 Build Data
- 5 Build 1
- 5.1 Signed off revisions
- 5.2 Start the Build
- 5.3 Tag
- 5.4 E-mail Metrics
- 5.5 E-mail mirrors@mozilla.org
- 5.6 Source
- 5.7 Builds
- 5.8 L10N repacks
- 5.9 Signing
- 5.10 L10N Verify
- 5.11 Updates
- 5.12 Major Updates
- 5.13 Add to Bouncer
- 5.14 Push to the internal mirrors
- 5.15 Final Verification
- 5.16 Push to the external mirrors
- 5.17 Push final snippets
- 5.18 Push MU snippets
- 5.19 Update symlinks
- 6 Redirects
General Notes
Release Engineers
rail, tracking bug: bug 744150
Bugs Hit
- there was no 12.0 milestone, had to use 12.0b4 milestone
- Updates failed
- bug 747689 - Thunderbird update verify fails trying to download 10.0
Build Data
Build # | Type | Build ID | Build machine | Time to build |
1 | Linux | 20120420153403 | momo-vm-linux-03 | 1 hrs, 43 mins, 28 secs |
Linux64 | 20120420153206 | momo-vm-linux64-06 | 1 hrs, 21 mins, 5 secs | |
Mac | 20120420154114 | tb2-darwin10-slave62 | 3 hrs, 37 mins, 27 secs | |
Windows | 20120420153905 | momo-vm-win2k3-08 | 1 hrs, 52 mins, 45 secs |
Build 1
Signed off revisions
- comm-release: http://hg.mozilla.org/releases/comm-release/rev/37620e34f3a4
- mozilla-release: Same changeset as for Firefox 12.0 final. Let buildbot create its own relbranch. (5bcfa0da3be9)
- L10n-changesets: sign-off 12.0 on the dashboard (I think there's one minor zh-TW change, ignore any pending). Took 12.0b4 changeset.
Start the Build
Preparation
- Clobberred comm-beta branch at https://clobberer-admin.mozillamessaging.com/ (at the top of the page, choose the production master, comm-release, any builder)
- Create release config patch
- Add L10N changesets
- Land configs
- Tag configs and tools using
hg tag THUNDERBIRD_12_0_RELEASE THUNDERBIRD_12_0_BUILD1
- Update and reconfig the master
ssh buildbot@momo-vm-03.sj.mozillamessaging.com cd ~/production/buildbot-configs hg pull -u # check if there are local chanes hg diff cd /buildbot/production ./buildbot.sh checkconfig ./buildbot.sh reconfig & tail -F buildbot-configs/thunderbird/twistd.log |grep configuration
- Start the automation
buildbot sendchange --username=rail --master=localhost:9010 --branch=releases/comm-release -m "Thunderbird 12.0 build 1 kick 1" doit
- Monitor the build progress at http://build.mozillamessaging.com/buildbot/production/ and manage it at https://buildbot-admin.mozillamessaging.com/production/
Tag
No problems
E-mail Metrics
Use the address "metrics-alerts < AT > mozilla < PERIOD > org"
Subject: Thunderbird 12.0
Thunderbird,thunderbird,thunderbird,12.0,12.0
E-mail mirrors@mozilla.org
Not necessary for rapid release betas or ESR builds; files only go as far as the 'internal mirrors'.
Coordinate yourself with the person creating the ESR and the latest release which normally go out at the same time.
They want to know approximately what time push to external mirrors will occur. Ideally this is a 24hr notice to them, with the estimated time.
Subject: Thunderbird 12.0 coming to mirrors on Apr 24, 2012 Body: We're planning on pushing our Thunderbird 12.0 release to mirrors ~24 hours before Tuesday, April 24, 2012. # If this is a chemspill, please mention that in the email as it will mean pinging # for CDN to be enabled when the push to internal mirrors happens If you believe these releases or these dates will cause any problems, please notify release at mozilla.com. If you have any problems with mirror status/weights/etc, please notify mirror-submissions at mozilla.org.
Source
No problems.
Builds
No problems
L10N repacks
- linux sanity check:
check-l10n-release-builds.pl --repo http://hg.mozilla.org/releases/comm-release --branch=release --revision THUNDERBIRD_12_0_RELEASE --min 164 --max 217 linux
- linux64 sanity check:
check-l10n-release-builds.pl --repo http://hg.mozilla.org/releases/comm-release --branch=release --revision THUNDERBIRD_12_0_RELEASE --min 164 --max 217 linux64
- win32 sanity check:
check-l10n-release-builds.pl --repo http://hg.mozilla.org/releases/comm-release --branch=release --revision THUNDERBIRD_12_0_RELEASE --min 180 --max 233 win32
- mac sanity check
check-l10n-release-builds.pl --repo http://hg.mozilla.org/releases/comm-release --branch=release --revision THUNDERBIRD_12_0_RELEASE --min 164 --max 217 osx
Signing
- Using Firefox signing machine
VERSION=12.0 BUILD=1 TAG=THUNDERBIRD_12_0_RELEASE REPO=releases/comm-release PRODUCT=thunderbird APP_NAME=mail EMAIL=release@mozilla.com KEYDIR=d:/2011-keys RELCONFIG=fake STAGE_HOST=stage.mozilla.org STAGE_USERNAME=tbirdbld CVS_USER=tbirdbld SSH_KEY=$HOME/.ssh/tbirdbld_dsa cd ${HOME}/hg-tools/ # if it exists, otherwise create a fresh clone hg pull && hg up mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python cd ~/signing-work/${PRODUCT}-${VERSION} cp ~/hg-tools/release/signing/* . rsync -av ~/hg-tools/lib/python/ ./lib/python/ make tb-all PRODUCT=${PRODUCT} VERSION=${VERSION} \ BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \ KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} \ APP_NAME=$APP_NAME TAG=$TAG \ STAGE_USERNAME=$STAGE_USERNAME \ STAGE_HOST=$STAGE_HOST SSH_KEY=$SSH_KEY \ CVS_USER=$CVS_USER
- run the same with "postsign" target
L10N Verify
Updates
- First run failed
Adding New File Contents/MacOS/components/libalerts.dylib done with patch 119/220 time (79.12s/16.19s/95.31s) (mar/patch/total) Extracting /buildbot/updates_release/build/temp/thunderbird/11.0.1/ftp/thunderbird-11.0.1.de.mac.complete.mar to /buildbot/updates_release/build/tmpgQPar3-fastmode/120/from Extracting /buildbot/updates_release/build/temp/thunderbird/12.0/ftp/thunderbird-12.0.de.mac.complete.mar to /buildbot/updates_release/build/tmpgQPar3-fastmode/120/to IN SHUTDOWN... Failed to remove patcher2.pid: No such file or directory FAIL shell call timed out after 10800 seconds at patcher2.pl line 1682. program finished with exit code 2 elapsedTime=10846.638534
- Commented out the following steps
#self.bumpPatcherConfig() #self.bumpVerifyConfigs()
- reconfigured and forced the updates builder
- second run failed as well
- Ran it manually on the same slave (no need to build update tools)
# cltbld@momo-vm-linux-03.sj.mozillamessaging.com cd /buildbot/updates_release/build rm -rf temp perl patcher2.pl --download --app=thunderbird --brand=Thunderbird --config=patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg export MAR=/buildbot/updates_release/build/mozilla/obj/dist/host/bin/mar find . -name '*.mar' | while read marname; do $MAR -t $marname > /dev/null; x=$?; if [ $x != "0" ]; then echo "FAIL $marname"; fi; done perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=thunderbird --brand=Thunderbird --config=patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg find temp/thunderbird/11.0.1-12.0/ftp/thunderbird/nightly/12.0-candidates/build1 -type f -exec chmod 644 {} \; find temp/thunderbird/11.0.1-12.0/ftp/thunderbird/nightly/12.0-candidates/build1 -type d -exec chmod 755 {} \; cd temp/thunderbird/11.0.1-12.0/ftp/thunderbird/nightly/12.0-candidates/build1 rsync -avP -e "ssh -oIdentityFile=~/.ssh/tbirdbld_dsa" --exclude=*complete.mar update tbirdbld@stage.mozilla.org:/pub/mozilla.org/thunderbird/nightly/12.0-candidates/build1/ cd - cd temp/thunderbird/11.0.1-12.0/ rsync -av -e "ssh -oIdentityFile=~/.ssh/tbirdbld_dsa" aus2.test/ tbirdbld@momo-build-adm-01.sj.mozillamessaging.com:/opt/aus2/snippets/staging/Thunderbird-12.0-build1-test rsync -av -e "ssh -oIdentityFile=~/.ssh/tbirdbld_dsa" aus2/ tbirdbld@momo-build-adm-01.sj.mozillamessaging.com:/opt/aus2/snippets/staging/Thunderbird-12.0-build1 bash /buildbot/updates_release/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest ssh -l tbirdbld -i ~/.ssh/tbirdbld_dsa momo-build-adm-01.sj.mozillamessaging.com "~/bin/pushsnip Thunderbird-12.0-build1-test"
Update Verify
- Forced update verifies failed trying to download 10.0 from a usual location, while the directory should be 10.0-real. Filed bug 747689
- No problems after landing bug 747689
Send a notification email
Let people know what's been done. Note: this is a wide list so keep traffic low.
To: thunderbird-drivers@mozilla.org Subj: Thunderbird 12.0 build1 is available on test channels
Major Updates
- Configs landed (as a part of 12.0 patch)
- Set clobberer
- forced the builder
- first build timed out trying to remove old builds
- forced again
- Second run failed with
wget of http://stage.mozilla.org//pub/mozilla.org/thunderbird/nightly/12.0-candidates/build1/macosx_info.txt failed. at MozBuild/Util.pm line 603. program finished with exit code 1
- Set clobberer
- Copied macosx_info.txt:
# tbirdbld@surf cd /pub/mozilla.org/thunderbird/nightly/12.0-candidates/build1/ cp macosx64_info.txt macosx_info.txt
- Forced 3rd builder
Major Update Verify
Add to Bouncer
- pushed a braindump patch
- Ran the following
# from my laptoop, in tools/release $ bash ../../braindump/releases-related/add_thunderbird_to_bouncer.sh Adding product: Thunderbird-12.0 Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="1673" name="Thunderbird-12.0"/></products> Adding product: Thunderbird-12.0-Complete Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="1674" name="Thunderbird-12.0-Complete"/></products> Adding product: Thunderbird-12.0-Partial-11.0.1 Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="1675" name="Thunderbird-12.0-Partial-11.0.1"/></products> Adding location for Thunderbird-12.0, linux: /thunderbird/releases/12.0/linux-i686/:lang/thunderbird-12.0.tar.bz2 Adding location for Thunderbird-12.0-Complete, linux: /thunderbird/releases/12.0/update/linux-i686/:lang/thunderbird-12.0.complete.mar Adding location for Thunderbird-12.0-Partial-11.0.1, linux: /thunderbird/releases/12.0/update/linux-i686/:lang/thunderbird-11.0.1-12.0.partial.mar Adding location for Thunderbird-12.0, linux64: /thunderbird/releases/12.0/linux-x86_64/:lang/thunderbird-12.0.tar.bz2 Adding location for Thunderbird-12.0-Complete, linux64: /thunderbird/releases/12.0/update/linux-x86_64/:lang/thunderbird-12.0.complete.mar Adding location for Thunderbird-12.0-Partial-11.0.1, linux64: /thunderbird/releases/12.0/update/linux-x86_64/:lang/thunderbird-11.0.1-12.0.partial.mar Adding location for Thunderbird-12.0, osx: /thunderbird/releases/12.0/mac/:lang/Thunderbird%2012.0.dmg Adding location for Thunderbird-12.0-Complete, osx: /thunderbird/releases/12.0/update/mac/:lang/thunderbird-12.0.complete.mar Adding location for Thunderbird-12.0-Partial-11.0.1, osx: /thunderbird/releases/12.0/update/mac/:lang/thunderbird-11.0.1-12.0.partial.mar Adding location for Thunderbird-12.0, win: /thunderbird/releases/12.0/win32/:lang/Thunderbird%20Setup%2012.0.exe Adding location for Thunderbird-12.0-Complete, win: /thunderbird/releases/12.0/update/win32/:lang/thunderbird-12.0.complete.mar Adding location for Thunderbird-12.0-Partial-11.0.1, win: /thunderbird/releases/12.0/update/win32/:lang/thunderbird-11.0.1-12.0.partial.mar
Push to the internal mirrors
Added the following entry to /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude:
- thunderbird/releases/12.0
# ~/bin/push-release /pub/mozilla.org/thunderbird/nightly/12.0-candidates/build1 /pub/mozilla.org/thunderbird/releases/12.0
Copy/paste the output and run it (took ~10mins)
Final Verification
No problems.
Push to the external mirrors
Removed the following entry to /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude:
- thunderbird/releases/12.0
Push final snippets
# ssh -l tbirdbld -i ~/.ssh/tbirdbld_dsa momo-build-adm-01.sj.mozillamessaging.com ~/bin/pushsnip Thunderbird-12.0-build1
Push MU snippets
# ssh -l tbirdbld -i ~/.ssh/tbirdbld_dsa momo-build-adm-01.sj.mozillamessaging.com ~/bin/pushsnip 20120421-Thunderbird-3.1.20-12.0-MU ~/bin/pushsnip 20120421-Thunderbird-3.1.20-12.0-MU-beta
Update symlinks
cd /pub/mozilla.org/thunderbird/releases/ rm latest && ln -s 12.0 latest
Redirects
Use the svn:sys/scripts/insert-website-redirect.pl to insert redirects into puppet/modules/apache/files/conf.d/live.mozillamessaging.com.conf