Releases/Firefox 3.6b4/BuildNotes
NB: Text that is italicized indicates tasks still to be done, or information that needs to be filled in.
Contents
- 1 Build Engineers
- 2 Signed-off Revision(s)
- 3 Tags
- 4 Build data
- 5 Notes
- 5.1 Build 1
- 5.1.1 Tag
- 5.1.2 Source
- 5.1.3 Build/Repack
- 5.1.4 Socorro product data updates
- 5.1.5 Sign
- 5.1.6 L10nVerify
- 5.1.7 Generate updates and push betatest snippets
- 5.1.8 Update verify
- 5.1.9 Respin of fr
- 5.1.10 Update Bouncer
- 5.1.11 Final checks before push
- 5.1.12 Push to mirrors
- 5.1.13 Final Verification
- 5.1.14 Push updates to release channel
- 5.1.15 Release
- 5.1.16 XULRunner build
- 5.1 Build 1
Build Engineers
Tracking Bug: bug 530112
- PST: lsblakk
- EST:
- NZDT: nthomas
Signed-off Revision(s)
Build 1: af0e2b9566cc
Tags
On mozilla-central:
Build # | Tag | Changeset |
1 | GECKO192b4_20091124_RELBRANCH | af0e2b9566cc |
FIREFOX_3_6b4_BUILD1 FIREFOX_3_6b4_RELEASE | 4c488520d1bf |
Build data
Build 1:
Platform | Build ID | Build machine |
Win32 | 20091124213835 | moz2-win32-slave33 |
Mac | 20091124201530 | moz2-darwin9-slave09 |
Linux | 20091124201751 | moz2-linux-slave12 |
Wince | 20091124202522 | moz2-win32-slave42 |
Notes
Done on pm01
Build 1
- Clean up previous releases build dirs, just for the slaves we used for 3.6b3 build1. Used cssh for linux/mac, RDP as Administrator on win32.
- Linux: linux-slave01,02,05,06,07,08,09,10,18,20,22,28; rm -rf /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
- Mac:darwin9-slave02,05,06,07,09,10,12, xserve07,09; rm -rf /builds/slave/{macosx_build,macosx_repack,l10n_verification, macosx_update_verify}/*
- Win32: win32-slave01,02,05,06,07,08,09,10,16,31,40; rm -rf /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify,wince_build}/*
- bump mozilla-1.9.2 to 3.6b5pre with patch
- Get review and land [buildbot-configs patch], and re-enable ftp poller patch
- Closed the mozilla-1.9.2 tree
- Tagged build/{buildbot-configs,buildbotcustom,tools} with FIREFOX_3_6b4_RELEASE
- ONE-OFF - land sendchange hang fix from bug 528212 and restart master
- Sendchange was
buildbot sendchange --username=nthomas --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6b4 build1" ShakeAndBakeBaby
Tag
- slave: moz2-linux-slave08
- Land version bumps patch
- Re-opened mozilla-1.9.2
- No problems
Source
- slave: moz2-linux-slave42
- No problems
Build/Repack
- slaves
- Linux and Win32 no problems.
- Mac l10n bombed out because the slaves had not been cleaned up. Fix that (can't get l10n_verify and update verify on moz2-darwin9-slave05 because it needs rebooting) and then
# cltbld@production-master cd ~/tools hg pull hg up -r FIREFOX_3_6b4_RELEASE cd buildbot-helpers/ python force_release_l10n.py -m http://localhost:8010 -t FIREFOX_3_6b4_RELEASE -v -b releases/mozilla-1.9.2 -p macosx -n nthomas
- had failed to clean up bm-xserve07 and 09 so retrigger the locales that 15 locales that failed (af ar be bg bn-BD bn-IN ca cs cy da de el en-GB eo es-AR written to shipped-locales-macosx)
python force_release_l10n.py -m http://localhost:8010 -t FIREFOX_3_6b4_RELEASE -v -s shipped-locales-macosx -p macosx -n nthomas
- Manual verification that all l10n builds present:
# stage cd /home/ftp/pub/firefox/nightly/3.6b4-candidates/build1 # each should be 68 = en-US + 67 l10n builds for p in linux-i686 mac unsigned/win32; do echo `ls $p | grep -v xpi | wc -l` $p done
- Removed en-US.xpi from the 3.6b4-candidates/build1/ directory on stage
Socorro product data updates
Since Socorro 1.1 went live this is done with an Admin control panel, so asked Sam Sidler (ss) to enable those versions. That requires at least one crash report so created those using the CrashMe extension with linux 3.6b4 and 3.6b5pre builds.
Sign
No problems
L10nVerify
- slave: bm-xserve12
- Got "only in..." messages for locales which are new in b4 or missed b3
- no real differences
Generate updates and push betatest snippets
- slave: moz2-linux-slave01
As well as the automated updates we'll need to connect 3.6b1 to b4 for af bg da, similar to what we did last time.
Fill in b1 locales
- slave: moz2-linux-slave01
Needed to generate b1 -> b4 for 3.6b1 af bg da. Ran steps manually with this patcher config
Moved the original run out of the way:
cd /builds/slave/updates/build mv temp temp.original-run cd patcher-configs mv moz192-branch-patcher2.cfg moz192-branch-patcher2.cfg-original-run cd ../ #download builds perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg #create patches perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg #chmod partial mars: find temp/firefox/3.6b1-3.6b4/ftp/firefox/nightly/3.6b4-candidates/build1 -type f -exec chmod 644 {} \; #chmod partial mar dirs: find temp/firefox/3.6b1-3.6b4/ftp/firefox/nightly/3.6b4-candidates/build1 -type d -exec chmod 755 {} \; #upload partial mars: rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' --exclude=*complete.mar temp/firefox/3.6b1-3.6b4/ftp/firefox/nightly/3.6b4-candidates/build1/update ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6b4-candidates/build1/ #upload snippets: rsync -av temp/firefox/3.6b1-3.6b4/aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091125-Firefox-3.6b1-3.6b4-locales #upload test snippets rsync -av temp/firefox/3.6b1-3.6b4/aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091125-Firefox-3.6b1-3.6b4-locales-test
Then on aus2-staging as cltbld,
cd /opt/aus2/snippets/staging # Didn't run backupsnip because the automation one was running at the time ~/bin/pushsnip 20091125-Firefox-3.6b1-3.6b4-locales-test/
Generate 3.6b3 -> b4 for WinCE
- slave: moz2-linux-slave01 (same slave as the automated run)
# move the previous run aside cd /builds/slave/updates/build mv temp temp.b1 cd patcher-configs wget -Omoz192-branch-patcher2.cfg https://bug530112.bugzilla.mozilla.org/attachment.cgi?id=414504 cd /builds/slave/updates/build perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg find temp/firefox/3.6b3-3.6b4/ftp/firefox/nightly/3.6b4-candidates/build1 -type f -exec chmod 644 {} \; find temp/firefox/3.6b3-3.6b4/ftp/firefox/nightly/3.6b4-candidates/build1 -type d -exec chmod 755 {} \; cd /builds/slave/updates/build/temp/firefox/3.6b3-3.6b4/ftp/firefox/nightly/3.6b4-candidates/build1 rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' --exclude=*complete.mar update ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6b4-candidates/build1/ cd /builds/slave/updates/build/temp/firefox/3.6b3-3.6b4 rsync -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091125-Firefox-3.6b4-WinCE rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091125-Firefox-3.6b4-WinCE-test
# cltbld@aus2-staging # backupsnip in automation was still running, didn't do one ~/bin/pushsnip 20091125-Firefox-3.6b4-WinCE-test
Update verify
- All platforms' partials had differences in removed-files caused by packaging fixes landed for the 3.5 -> 3.6 MU, bug 515421. They are not a real issue. Eg linux
diff -r source/firefox/removed-files target/firefox/removed-files 500a501,502 > res/broken-image.gif > res/loading-image.gif 533a536,537 > icons/mozicon16.xpm > icons/mozicon50.xpm 537a542 > old-homepage-default.properties
- All platforms: 3.6b1 ka failed because we still don't have that locale back
- Linux: No real problems.
- Mac: got an error checking b2 fr where complete mars were 0-byte, which I assumed was because the files got removed while they were respun. Lead to the discovery that we need to generate b1 and b2 snippets for fr (see below)
- Win32: Three chk file differences as normal
Respin of fr
A problem was discovered with the fr build, so we've got to respin it.
clean up the slaves
Because we can't predict which slave it will happen on we have to clean up all of the l10n dirs on the slaves.
- Linux: 01,02,05-10
- Mac: moz2-darwin9-slave02,05,06,07,09,10,bm-xserve07,09
- Win32: win32-slave01,02,05-10
force the build
Used the web interface to force the fr build for each platform with the following properties:
- en_revision = FIREFOX_3_6b4_RELEASE
- l10n_revision = FIREFOX_3_6b4_RELEASE
- locale = fr
builds went fine, uploads failed with (FIXME):
Traceback (most recent call last): File "/usr/bin/post_upload.py", line 262, in ? func(options, upload_dir, files) File "/usr/bin/post_upload.py", line 156, in ReleaseToCandidatesDir os.chmod(os.path.join(root, d), 0755) OSError: [Errno 1] Operation not permitted: '/home/ftp/pub/firefox/nightly/3.6b4-candidates/build1/contrib/solaris_pkgadd' Encountered error while uploading Command ['ssh', '-o', 'IdentityFile=~/.ssh/ffxbld_dsa', 'ffxbld@stage.mozilla.org', 'post_upload.py -p firefox -v 3.6b4 -n 1 --release-to-candidates-dir "/tmp/tmp.IqJdUJ9377/" "/tmp/tmp.IqJdUJ9377/linux-i686/fr/firefox-3.6b4.tar.bz2" "/tmp/tmp.IqJdUJ9377/linux-i686/xpi/fr.xpi"'] returned non-zero exit code: 1 make: *** [l10n-upload-fr] Error 2
Had to finish the upload with:
cd /builds/slave/linux_repack/build/mozilla-1.9.2/dist/update/linux-i686 rsync -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' -av fr ffxbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.6b4-candidates/build1/update/linux-i686/
...and equivalent commands for the other platforms. Had to upload the win32 installer, too.
signing
- Done mostly manually:
- Only pulled win32 en-US (build1) and linux/mac/win32 zh-CN (build2), MD5SUMS, SHA1SUMS into unsigned-build2:
PRODUCT=firefox VERSION=3.6b4 BUILD=1 TAG=FIREFOX_3_6b4_RELEASE REPO='releases/mozilla-1.9.2' EMAIL=bhearsum@mozilla.com KEYDIR=d:/2009-keys mkdir ~/signing-work/${PRODUCT}-${VERSION}-fr2 cd ~/signing-work/${PRODUCT}-${VERSION}-fr2 cp ~/hg-tools/release/signing/* . make setup PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 rsync -av -e ssh --include=*SUMS --include=xpi --include=linux-i686/ --include=mac/ --include=unsigned/ --include=*irefox* --include=win32 --include=*fr* --include=update/ --exclude=* cltbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ unsigned-build${BUILD} rsync -av -e ssh --include=unsigned/ --include=*irefox* --include=win32 --include=*en-US* --include=update/ --exclude=* cltbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build1/ unsigned-build${BUILD} rsync -av -e ssh --exclude=*partial* --exclude=*asc --include=win32 --include=*irefox* --include=*en-US* --include update/ --exclude=* cltbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build1/ signed-build${BUILD} make stubs PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 ./sign-release.py -p --keydir ${KEYDIR} -j1 --product firefox -o signed-build${BUILD} unsigned-build${BUILD} >> win32_signing_build${BUILD}.log 2>&1 rsync -av --include=*.mar --include=update --include=*SUMS --include=*dmg --include=*.tar.bz2 --include=*xpi* --include=*fr* --include=*linux* --include=*mac* --exclude=* unsigned-build1/ signed-build1/ rsync -av ./unsigned-build${BUILD}/unsigned/win32/xpi ./signed-build${BUILD}/win32 # manually updated the SUMS files with new md5/sha1 sums cd signed-build1 ../signing/sign-files . rm win32/en-US/*.asc cd ../ # push builds back rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' signed-build1/ ffxbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.6b4-candidates/build1/
updates
- done on moz2-linux-slave01
- used patcher config
cd /builds/slave/updates/build perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg find temp/firefox/3.6b3-3.6b4/ftp/firefox/nightly/3.6b4-candidates/build1 -type f -exec chmod 644 {} \; find temp/firefox/3.6b3-3.6b4/ftp/firefox/nightly/3.6b4-candidates/build1 -type d -exec chmod 755 {} \; cd /builds/slave/updates/build/temp/firefox/3.6b3-3.6b4/ftp/firefox/nightly/3.6b4-candidates/build1 rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' --exclude=*complete.mar update ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6b4-candidates/build1/ cd /builds/slave/updates/build/temp/firefox/3.6b3-3.6b4 rsync -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091125-Firefox-3.6b4-fr-respin rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091125-Firefox-3.6b4-fr-respin-test
# cltbld @ aus2-staging cd /opt/aus2/snippets/staging ~/bin/pushsnip 20091125-Firefox-3.6b4-fr-respin-test
This only regenerated the snippets for b3 and we also need to do b1 and b2 (because the complete mar hash changed). Redo using this config
# cltbld@moz2-linux-slave01 cd /builds/slave/updates/build/temp/firefox/3.6b3-3.6b4 mkdir old-aus mv aus2 old-aus cd ../../.. perl patcher2.pl --create-patchinfo --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg cd temp/firefox/3.6b3-3.6b4/ diff -rq {old-aus/,}aus2.test Only in aus2.test/Firefox: 3.6b1 Only in aus2.test/Firefox: 3.6b2 diff -rq {old-aus/,}aus2 Only in aus2/Firefox: 3.6b1 Only in aus2/Firefox: 3.6b2 rsync -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091125-Firefox-3.6b4-fr-respin-2 rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091125-Firefox-3.6b4-fr-respin-2-test
# cltbld @ aus2-staging ~/bin/pushsnip 20091125-Firefox-3.6b4-fr-respin-2-test
Update Bouncer
Added the usual 3 products and 9 file locations, plus the three locations for WinCE.
Added Firefox-3.6b4-Partial-3.6b1 product & 3 file locations for straggler locales. Sentry is looking for en-US copies of these files, so fake this out with zero byte files
# ffxbld@stage touch /pub/mozilla.org/firefox/nightly/3.6b4-candidates/build1/update/{linux-i686,mac,win32}/en-US/firefox-3.6b1-3.6b4.partial.mar
Final checks before push
cd /pub/mozilla.org/firefox/nightly/3.6b4-candidates/build1/ find . ! -user ffxbld find . ! -group firefox find . -type f ! -perm 644 find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*' find . -maxdepth 1 -type d ! -perm 2775 -name 'contrib*'
Only returns Solaris builds (different user) for the user check. Had to fix the mode on the respun fr mar files and 2775 mode on the contrib dirs.
Push to mirrors
# ffxbld@stage rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*.crashreporter-symbols.zip /pub/mozilla.org/firefox/nightly/3.6b4-candidates/build1/ /pub/mozilla.org/firefox/releases/3.6b4/
Removed 3.6b3 from /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude as we don't need lots of mirror coverage and the biggest mirror syncs very quickly from the mozilla-releases module.
Final Verification
- slave: moz2-linux-slave13
- fails to find updates for 3.6b1 ka, as expected
Push updates to release channel
Well ahead of time
time ~/bin/backupsnip 20091124-Firefox-3.6b4 real 61m52.014s user 0m55.967s sys 1m39.784s
~/bin/pushsnip 20091124-Firefox-3.6b4 ~/bin/pushsnip 20091125-Firefox-3.6b4-fr-respin-2 ~/bin/pushsnip 20091125-Firefox-3.6b1-3.6b4-locales
This part happened the following monday.
~/bin/pushsnip 20091125-Firefox-3.6b4-WinCE
Release
No symlink until final
XULRunner build
Builds are forced manually, using revision of FIREFOX_3_6b4_RELEASE, name and reason. When they complete
# xrbld@stage cd /pub/mozilla.org/xulrunner/nightly/ mkdir -p 1.9.2b4-candidates/build1/ mv -v latest-mozilla-1.9.2/*-1.9.2b4.* 1.9.2b4-candidates/build1/
Contacted mfinkle and dtownsend for smoketest QA.
Staging and push to mirrors pending.