Releases/Firefox 3.0.11/BuildNotes
Be sure to use RELEASE_AUTOMATION_M14 in the buildbot master config and when tagging the Bootstrap config file, it fixes up the tagging of the changes from bug 483232 for partner repacks - nthomas
Contents
- 1 Build Engineers
- 2 Bonsai queries
- 3 Tags
- 4 Build data
- 5 Notes
- 5.1 Build 1
- 5.2 Build 2
- 5.2.1 Tag
- 5.2.2 Source
- 5.2.3 Build & Repack
- 5.2.4 XULRunner
- 5.2.5 Sign
- 5.2.6 L10nVerify
- 5.2.7 Generate Updates
- 5.2.8 Publish Updates to Test Channels (betatest & releasetest)
- 5.2.9 Update Verify
- 5.2.10 Stage
- 5.2.11 Partner Repacks
- 5.2.12 Push updates to beta channel
- 5.2.13 Sign Installers
- 5.2.14 Update Bouncer
- 5.2.15 Push to mirrors
- 5.2.16 Final Verification
- 5.2.17 Publish Updates to Release Channel
- 5.2.18 Release
- 5.2.19 Release matching XULRunner build
Build Engineers
catlee
Tracking release bug
Bonsai queries
Tags
Build 1:
Module | Branch | Tag | Pull date |
cvsroot/mozilla | HEAD | GECKO190_20090519_RELBRANCH FIREFOX_3_0_11_BUILD1 FIREFOX_3_0_11_RELEASE | |
l10n/l10n | HEAD | GECKO190_20090519_RELBRANCH FIREFOX_3_0_11_BUILD1 FIREFOX_3_0_11_RELEASE |
Build 2:
Module | Branch | Tag | Pull date |
cvsroot/mozilla | HEAD | GECKO190_20090519_RELBRANCH FIREFOX_3_0_11_BUILD2 FIREFOX_3_0_11_RELEASE | |
l10n/l10n | HEAD | GECKO190_20090519_RELBRANCH FIREFOX_3_0_11_BUILD2 FIREFOX_3_0_11_RELEASE |
Build data
Type | Build ID | SHA1 | Push date | Build machine |
Notes
Build 1
Setup before starting:
- On fx-win32-1.9-slave2:
- Deleted /e/xr19rel/WINNT_5.2_Depend
- Deleted /e/fx19l10nrel/WINNT_5.2_Depend
- Deleted /e/fx19rel/WINNT_5.2_Depend
- On fx-linux-1.9-slave2:
- Deleted /builds/verify/firefox-3.0.10
- Deleted /builds/updates/firefox-3.0.10
- Deleted /builds/source/firefox-3.0.10
- Deleted /builds/data/cltbld/firefox-3.0.10
- Deleted /builds/tinderbox/Xr-Mozilla1.9-Release/Linux_2.6.18-53.1.13.el5_Depend
- Deleted /builds/tags/FIREFOX_3_0_3_BUILD1
- Deleted /builds/tags/FIREFOX_3_0_4_BUILD1
- On fx-mac-1.9-slave2:
- Deleted /builds/verify/firefox-3.0.10
- Space free on slaves:
- fx-mac-1.9-slave2: 28G free on /
- fx-win32-1.9-slave2: 17G free on D, 14G free on E
- fx-linux-1.9-slave2: 21G free on /builds
- production-1.9-master reconfigured with change from https://bugzilla.mozilla.org/attachment.cgi?id=378318
- Kicked off automation:
buildbot sendchange --username=catlee --master=localhost:9989 -m"Firefox 3.0.11build1 release" go
Tag
No problems
Source
No problems
Build & Repack
- No problems
Sign
No problems
L10nVerify
No problems
Generate Updates
No problems
Publish Updates to Test Channels (betatest & releasetest)
No problems
Update Verify
Linux failed because pushing to hg would have created a new head. Clobbered /builds/verify/firefox-3.0.11/* on fx-linux-1.9-slave2 and rebuilt failed build. Second build passed.
Windows failed with "Step Updates died: shell call returned bad exit code: 1 at Bootstrap/Step.pm line 110.". Clobbered e:\builds\buildbot\trunk-automation\win32_update_verify\build and rebuilt. Second build failed with normal binary difference errors.
Mac passed on the first try.
Stage
No problems
Partner Repacks
First build failed because 7z couldn't be found. coop fixed this on the slave and forced a new build which ran successfully.
Push updates to beta channel
# cltbld@aus2-staging.m.o # make sure scripts are up to date cd bin cvs update cd /opt/aus2/snippets/staging/ ~/bin/backupsnip 20090519-Firefox-3.0.11-beta ~/bin/pushsnip 20090519-Firefox-3.0.11-beta
At this point the release process was interrupted due to bug 495648.
Build 2
Setup before starting:
- On fx-win32-1.9-slave2:
- Deleted /e/fx19rel/WINNT_5.2_Depend
- Deleted /e/fx19l10nrel/WINNT_5.2_Depend
- On fx-linux-1.9-slave2:
- Deleted /builds/verify/firefox-3.0.11
- Deleted /builds/updates/firefox-3.0.11
- Deleted /builds/source/firefox-3.0.11
- Deleted /builds/tags/FIREFOX_3_0_5_BUILD1
- Deleted /builds/data/cltbld/firefox-3.0.11
- Space free on slaves:
- fx-mac-1.9-slave2: 26GB free on /
- fx-win32-1.9-slave2: 16GB free on D, 14GB free on E
- fx-linux-1.9-slave2: 20GB free on /builds
- Kicked off automation:
buildbot sendchange --username=catlee --master=localhost:9989 -m"Firefox 3.0.11build2 release" go
Tag
No problems
Source
No problems
Build & Repack
No problems
XULRunner
Had to kick off manually. No problems after that.
Sign
No problems
L10nVerify
No problems
Generate Updates
No problems
Need to bring people who already updated to 3.0.11 build1 up to build2, so do this (after the automated run)
# connect as cltbld@fx-linux-1.9-slave2 cd /builds/updates/firefox-3.0.11/patcher/temp/firefox/3.0.10-3.0.11 for dir in aus2.test aus2.beta; do # copy complete snippets from 3.0.10 into a new dir for 3.0.11 mkdir -p b1-b2-${dir}/Firefox rsync -a --exclude 'partial.txt' ${dir}/Firefox/3.0.10/ b1-b2-${dir}/Firefox/3.0.11/ # use the buildid's from build1 mv b1-b2-${dir}/Firefox/3.0.11/Darwin_Universal-gcc3/2009042315 b1-b2-${dir}/Firefox/3.0.11/Darwin_Universal-gcc3/2009051908 mv b1-b2-${dir}/Firefox/3.0.11/Linux_x86-gcc3/2009042315 b1-b2-${dir}/Firefox/3.0.11/Linux_x86-gcc3/2009051908 mv b1-b2-${dir}/Firefox/3.0.11/WINNT_x86-msvc/2009042316 b1-b2-${dir}/Firefox/3.0.11/WINNT_x86-msvc/2009051909 done rsync -a -e 'ssh -i ~/.ssh/aus' b1-b2-aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20090605-Firefox-3.0.11build1-build2-test rsync -a -e 'ssh -i ~/.ssh/aus' b1-b2-aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20090605-Firefox-3.0.11build1-build2-beta
Then go to aus2-staging and push 20090605-Firefox-3.0.11build1-build2-test live.
Publish Updates to Test Channels (betatest & releasetest)
No problems
Update Verify
Linux failed because pushing to hg would have created a new head. Clobbered /builds/verify/firefox-3.0.11/* on fx-linux-1.9-slave2 and rebuilt failed build. The second build passed.
Windows failed with "Step Updates died: shell call returned bad exit code: 1 at Bootstrap/Step.pm line 110.". Clobbered e:\builds\buildbot\trunk-automation\win32_update_verify\build and rebuilt. The second build passed with the normal warnings about binary differences.
Mac passed.
Stage
No problems
Partner Repacks
Had to be forced manually, but ran without problems.
Push updates to beta channel
# cltbld@aus2-staging.m.o # make sure scripts are up to date cd bin cvs update cd /opt/aus2/snippets/staging/ ~/bin/backupsnip 20090605-Firefox-3.0.11build1-build2-test ~/bin/pushsnip 20090603-Firefox-3.0.11-beta ~/bin/pushsnip 20090605-Firefox-3.0.11build1-build2-beta
Sign Installers
Done manually using these installer-signing-instructions here.
- complete stage-merged:
# on stage cd /data/cltbld/firefox-3.0.11/ rsync -av batch1/mar/ stage-merged/ rsync -av batch1/stage-signed/ stage-merged/
- Create MD5 and SHA1 checksum files
# on stage cd /data/cltbld/firefox-3.0.11/stage-merged/ ~/bin/checksum-files .
- Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox . chmod 644 *SUMS
Update Bouncer
- Done.
- Note for next release: Do not remove the Check Now bit on the Firefox-3.0.10 Products until well after the change to the rsync module (to prevent the likes of bug 464566)
Push to mirrors
- push the stage-merged directory to the releases area:
# on stage rsync -av /data/cltbld/firefox-3.0.11/stage-merged/ /home/ftp/pub/firefox/releases/3.0.11/
- edit the exclude file /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.11) and remove the previous release (3.0.10).
Final Verification
- Verify that releasetest points to valid bounce links:
hg clone http://hg.mozilla.org/build/tools cd tools/release/updates cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' > update.cfg ./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
- Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
- A few 503 errors from mirrors.directorymix.com that were resolved when I went to check manually.
- Before pushing final updates, verify that "release" and "releasetest" channel match:
# on aus2-staging $ cd /opt/aus2/snippets/staging/20090603-Firefox-3.0.11 $ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20090603-Firefox-3.0.11-test/$a");' $
Publish Updates to Release Channel
# cltbld@aus2-staging cd /opt/aus2/snippets/staging ~/bin/backupsnip 20090603-Firefox-3.0.11 ~/bin/pushsnip 20090603-Firefox-3.0.11
Release
# cltbld@stage cd /home/ftp/pub/firefox/releases rm latest-3.0 && ln -s 3.0.11 latest-3.0
Release matching XULRunner build
XULRunner builds are produced by the automation, this is to groom them and push to the mirrors.
# cltbld@stage VERSION=1.9.0.11 FX_VERSION=3.0.11 BUILD=1 mkdir -p ~/xulrunner-${VERSION}/batch1/stage-unsigned/{runtimes,source,sdk} cd ~/xulrunner-${VERSION}/batch1/stage-unsigned rsync -av --exclude=*sdk* --exclude=*info* \ /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ runtimes/ rsync -av --include=*sdk* --exclude=* \ /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ sdk/ cp /home/ftp/pub/firefox/releases/${FX_VERSION}/source/firefox-${FX_VERSION}-source.tar.bz2 \ source/xulrunner-${VERSION}-source.tar.bz2 cd ~/xulrunner-${VERSION}/batch1 rsync -av stage-unsigned/ stage-signed/
Then create detached signatures per usual process (with PRODUCT=xulrunner, VERSION=1.9.0.11, don't need BUILD or TAG), skip the win32 signing section. Back on stage:
cd ~/xulrunner-${VERSION}/batch1/stage-signed cp /home/ftp/pub/firefox/releases/${FX_VERSION}/KEY . ~/bin/checksum-files . chown -R cltbld:xulrunner . find . -type f -exec chmod -v 644 {} \; find . -type d -exec chmod -v 755 {} \; cd ~/xulrunner-${VERSION}/batch1 rsync -nav stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/
If all is fine then
rsync -av stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/
Ping mfinkle or Mossop to update the links in these two wiki pages for the version change