Releases/Firefox 3.6.3/BuildNotes
Contents
- 1 Build Engineers
- 2 Signed-off Revision(s)
- 3 Tags
- 4 Build data
- 5 Notes
- 5.1 Build 1
- 5.1.1 Tag
- 5.1.2 Source
- 5.1.3 Build 1 (redux)
- 5.1.4 Tag
- 5.1.5 Source
- 5.1.6 Build/Repack
- 5.1.7 Partner Repacks
- 5.1.8 Unit tests
- 5.1.9 Sign
- 5.1.10 L10nVerify
- 5.1.11 Generate updates and push betatest snippets
- 5.1.12 Update verify
- 5.1.13 Socorro product data updates
- 5.1.14 Update Bouncer
- 5.1.15 Final checks before push
- 5.1.16 Virus Scan
- 5.1.17 Push to mirrors
- 5.1.18 Final Verification
- 5.1.19 Push updates to beta & release channel
- 5.1.20 Release
- 5.1.21 Manually redo missing source tarball and bundle
- 5.1.22 3.5.9->3.6.3 MU
- 5.1.23 EU Ballot Build
- 5.1.24 XULRunner build
- 5.1.25 Push beta channel snippets
- 5.1 Build 1
Build Engineers
bhearsum
Signed-off Revision(s)
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/3093de37377d
Tags
On releases/mozilla-1.9.2:
Build # | Tag | Changeset |
1 | GECKO1922_20100315_RELBRANCH | 3093de37377d |
FIREFOX_3_6_3_BUILD1 FIREFOX_3_6_3_RELEASE | d6e028dc1b68 |
Build data
Build # | Type | Build ID | Build machine |
1 | Linux | 20100401074458 | mv-moz2-linux-ix-slave12 |
Mac | 20100401064631 | bm-xserve21 | |
Windows | 20100401080539 | mw32-ix-slave17 |
Notes
Build 1
- Clobbered 'mozilla-1.9.2' on pm02 with clobberer.
- landed backout of changes to l10n-changesets_mozilla-1.9.2 since 3.6.2.
- Landed automation config
- Reconfiged pm02
- Started the automation:
buildbot sendchange --username=bhearsum --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.3 build1" Go
- Manually tagged configs, buildbotcustom, and tools with FIREFOX_3_6_3_BUILD1 and FIREFOX_3_6_3_RELEASE.
Tag
No problems.
Source
No problems
HALT. Had to restart after this because tagging was done improperly (baseTag was still set to FIREFOX_3_6_2, which made us move the 3.6.2 tags rather than create 3.6.3 ones).]
Build 1 (redux)
Don't need to do a build2 since the build1 tags were never created. Cleanup from the screwup:
- Set clobberer to clobber mozilla-1.9.2 on pm02 again
- Deleted 3.6.3-candidates from stage (only had source tarball/bundle in it)
- Reverted tag commits on mozilla-1.9.2 repo:
hg up -C GECKO1922_20100315_RELBRANCH hg tag -f -r f14062c981ba -m "Revert accidental movement of FIREFOX_3_6_2_RELEASE tag." FIREFOX_3_6_2_RELEASE hg tag -f -r f14062c981ba -m "Revert accidental movement of FIREFOX_3_6_2_BUILD1 tag." FIREFOX_3_6_2_BUILD1 hg push ssh://hg.mozilla.org/releases/mozilla-1.9.2
- Landed fixed config
- Reconfiged pm02
- Started automation:
buildbot sendchange --username=bhearsum --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.3 build1" Go
Tag
No problems. The tag ended up on http://hg.mozilla.org/releases/mozilla-1.9.2/rev/28ef231a65a3 instead of http://hg.mozilla.org/releases/mozilla-1.9.2/rev/d6e028dc1b68, but that makes no difference as far as the builds are concerned.
Source
Dep scheduler didn't fire. No problems
Build/Repack
Dep scheduler didn't fire. Forced builds manually until fast slaves (ix/xserve) were building, then killed the other ones (per release-drivers request).
Forced l10n builds with helper scripts:
python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -t FIREFOX_3_6_3_RELEASE -v -b releases/mozilla-1.9.2 -p macosx -n bhearsum python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -t FIREFOX_3_6_3_RELEASE -v -b releases/mozilla-1.9.2 -p linux -n bhearsum python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -t FIREFOX_3_6_3_RELEASE -v -b releases/mozilla-1.9.2 -p win32 -n bhearsum
- Mac: no en-US problems, no l10n problems
- Linux: no en-US build problems, very slow uploads, no l10n problems
- Win32: same as Linux
Deleted en-US.xpi from stage.
Partner Repacks
Can't do these from the automation until bug 554321 is resolved.
Unit tests
- Mac: PASS
- Win32: PASS
- Linux: first run of mochitest-browser-chrome was orange, with an intermittent failure. Next run was green.
Sign
Started downloading builds early to speed up end to end time. Signing happened like this:
export PRODUCT=firefox export VERSION=3.6.3 export BUILD=1 export TAG=FIREFOX_3_6_3_RELEASE export REPO='releases/mozilla-1.9.2' export EMAIL=bhearsum@mozilla.com export KEYDIR=d:/2009-keys cd ~/hg-tools hg pull && hg up -r ${TAG} mkdir ~/signing-work/${PRODUCT}-${VERSION} cd ~/signing-work/${PRODUCT}-${VERSION} cp ~/hg-tools/release/signing/* .
make setup PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 # This step was run a few times, every 10 minutes or so until all builds finished make download PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 make stubs sign2 stage verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}
L10nVerify
The post_signing Scheduler didn't look like it was going to fire so I forced a build of this. The Scheduler fired later, and I cancelled that build.
No differences, this is a full PASS.
Generate updates and push betatest snippets
The post_signing Scheduler didn't like it was going to fire so I forced a build of this. The Scheduler fired later, and I cancelled that build.
- Snippet uploads failed because of changes in bug 511967. Need to fix permissions on aus2-staging, or upload with a different user for releases. bug 557195 file bug to track
- Kicked through manually with:
# cltbld @ moz2-linux-slave43 cd /builds/slave/updates/temp/firefox/3.6.2-3.6.3/ rsync -av -e 'ssh -oIdentityFile=~/.ssh/id_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20100401-Firefox-3.6.3 rsync -av -e 'ssh -oIdentityFile=~/.ssh/id_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20100401-Firefox-3.6.3-test rsync -av -e 'ssh -oIdentityFile=~/.ssh/id_dsa' aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20100401-Firefox-3.6.3-beta
And then,
# cltbld @ aus2-staging cd /opt/aus2/snippets/staging ~/bin/backupsnip 20100401-Firefox-3.6.3-test ~/bin/pushsnip 20100401-Firefox-3.6.3-test
Update verify
- Linux: PASS
- Win32: PASS
- Mac: PASS
Socorro product data updates
We can do this ourselves (a subset of relengs, ask nthomas or bhearsum or joduinn). No need to file an IT bug for this.
joduinn added 3.6.3 and 3.6.4pre with the crash-stats admin page.
Update Bouncer
Added with BouncerBuddy.
Final checks before push
# ffxbld@stage cd /pub/mozilla.org/firefox/nightly/3.6.3-candidates/build1/ find . ! -user ffxbld #nothing showed up, as expected. find . ! -group firefox #nothing showed up, as expected. find . -type f ! -perm 644 # unsigned/partner-repacks have 664 permission find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*' # unsigned/partner-repacks have 664 permission find . -maxdepth 1 -type d ! -perm 2775 -name 'contrib*' # nothing showed up, as expected.
Virus Scan
# ffxbld@stage $ cd /pub/mozilla.org/firefox/nightly/3.6.3-candidates/build1 $ clamdscan -m /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.3-candidates/build1 /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.3-candidates/build1: OK ----------- SCAN SUMMARY ----------- Infected files: 0 Time: 3257.886 sec (54 m 17 s)
Push to mirrors
# ffxbld@stage rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*.crashreporter-symbols.zip --exclude=*.tests.tar.bz2* --exclude=*partner-repacks* /pub/mozilla.org/firefox/nightly/3.6.3-candidates/build1/ /pub/mozilla.org/firefox/releases/3.6.3/
Updated cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to include 3.6.3 instead of 3.6.2
Final Verification
green. there were a few 302 temporary moved responses, but all looked ok.
Push updates to beta & release channel
# cltbld@aus2-staging cd /opt/aus2/snippets/staging/ ~/bin/backupsnip 20100401-Firefox-3.6.3 ... real 3m45.063s user 0m0.389s sys 0m3.667s ... ~/bin/pushsnip 20100401-Firefox-3.6.3
Release
#ffxbld @ stage cd /pub/mozilla.org/firefox/releases rm latest-3.6; ln -s 3.6.3 latest-3.6
Manually redo missing source tarball and bundle
- Normal tarball as part of automation got halted because of config error, see above.
- lsblakk triggered the source step in automation while she was prep-ing for a downtime early this morning.
- joduinn signed tarball and bundle as follows:
# new keymaster mkdir -p ~/signing-work/firefox-3.6.3/unsigned-build1/source mkdir -p ~/signing-work/firefox-3.6.3/signed-build1/source cd ~/signing-work/firefox-3.6.3/unsigned-build1/source scp -i ~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.6.3-candidates/build1/source . cd ~/signing-work/firefox-3.6.3/signed-build1/source scp -i ~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.6.3-candidates/build1/source . # modified MD5SUMS and SHA1SUMS gpg -v -u releases -sba firefox-3.6.3.bundle gpg -v -u releases -sba firefox-3.6.3.source.tar.bz2 gpg --verify firefox-3.6.3.bundle.asc gpg --verify firefox-3.6.3.source.tar.bz2.asc scp -i ~/.ssh/ffxbld_dsa * ffxbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.6.3-candidates/build1/source
3.5.9->3.6.3 MU
- Lots of fiddling around with master (thanks bhearsum). kicked off snippets by clicking on "major snippet" on waterfall. Posted to betatest channel
- abillings spotted that detailsURL was wrong. Seems this happened in 3.6.2 also.
- bhearsum showed me how he fixed it for 3.6.2:
# cltbld @ aus2-staging cd /opt/aus2/snippets/staging rsync -av 20100402-Firefox-3.5.9-3.6.3-MU-test/ 20100402-Firefox-3.5.9-3.6.3-MU-test-url-fixed/ cd 20100402-Firefox-3.5.9-3.6.3-MU-test-url-fixed/ find . -type f -exec sed -i -e s,/3.6.3/,/3.6/, {} \; cd .. # verification - confirms that only the detailsUrl has changed diff -Naur 20100402-Firefox-3.5.9-3.6.3-MU-test/ 20100402-Firefox-3.5.9-3.6.3-MU-test-url-fixed/ | grep -v '[+-][+-]' | grep '^[+-]' | grep -v detailsUrl # # now do the beta and release snippets, too rsync -av 20100402-Firefox-3.5.9-3.6.3-MU-beta/ 20100402-Firefox-3.5.9-3.6.3-MU-beta-url-fixed/ cd 20100402-Firefox-3.5.9-3.6.3-MU-beta-url-fixed/ find . -type f -exec sed -i -e s,/3.6.3/,/3.6/, {} \; # # verification - confirms that only the detailsUrl has changed cd .. diff -Naur 20100402-Firefox-3.5.9-3.6.3-MU-beta/ 20100402-Firefox-3.5.9-3.6.3-MU-beta-url-fixed/ | grep -v '[+-][+-]' | grep '^[+-]' | grep -v detailsUrl rsync -av 20100402-Firefox-3.5.9-3.6.3-MU/ 20100402-Firefox-3.5.9-3.6.3-MU-url-fixed/ cd 20100402-Firefox-3.5.9-3.6.3-MU-url-fixed/ find . -type f -exec sed -i -e s,/3.6.3/,/3.6/, {} \; cd .. # verification - confirms that only the detailsUrl has changed diff -Naur 20100402-Firefox-3.5.9-3.6.3-MU/ 20100402-Firefox-3.5.9-3.6.3-MU-url-fixed/ | grep -v '[+-][+-]' | grep '^[+-]' | grep -v detailsUrl # # push the new test snippets ~/bin/backupsnip 20100402-Firefox-3.5.9-3.6.3-MU-test-url-fixed ~/bin/pushsnip 20100402-Firefox-3.5.9-3.6.3-MU-test-url-fixed # # move old, unwanted directories mv 20100402-Firefox-3.5.9-3.6.3-MU-beta WRONG-URL-20100402-Firefox-3.5.9-3.6.3-MU-beta mv 20100402-Firefox-3.5.9-3.6.3-MU WRONG-URL-20100402-Firefox-3.5.9-3.6.3-MU
EU Ballot Build
- Fixed typo in bouncer: Firefox-3.6.3-EUballot win /firefox/releases/3.6.3/win32-EUballot/en-US/Firefox Setup 3.6.3.exe
- signed builds using following: https://intranet.mozilla.org/Build:CombinedSigning#Partner_Repack_Signing
XULRunner build
- This step doesn't have to be the last one
- Triggered the builds by going to the buildbot waterfall for mozilla-1.9.2 and forcing builds for linux (not x64), mac and windows xulrunner. Set the field "Revision to build" (not properties) to 'FIREFOX_3_6_3_RELEASE'. Email the builds to release-drivers.
- When waterfall showed builds done, looked at logs to find location of files on stage, and gather them into one place.
# as xrbld@stage cd /pub/mozilla.org/xulrunner/nightly mkdir -p 1.9.2.3-candidates/build1 cd -p 1.9.2.3-candidates/build1 scp /pub/mozilla.org/xulrunner/tinderbox-builds/mozilla-1.9.2-win32/1270238019/xulrunner-1.9.2.3.en-US.win32* . scp /pub/mozilla.org/xulrunner/tinderbox-builds/mozilla-1.9.2-linux/1270237392/* . scp /pub/mozilla.org/xulrunner/tinderbox-builds/mozilla-1.9.2-macosx/1270237393/* .
- Waiting on reply from mfinkle and/or dtownsend to publish these builds
Push beta channel snippets
These were not pushed at the same time as the release snippets.
# cltbld @ aus2-staging cd /opt/aus2/snippets/staging screen # push the new test snippets ~/bin/backupsnip 20100402-Firefox-3.5.9-3.6.3-MU-beta-url-fixed ... real 0m26.269s user 0m0.038s sys 0m0.537s ~/bin/pushsnip 20100402-Firefox-3.5.9-3.6.3-MU-beta-url-fixed ... sent 217198 bytes received 9788 bytes 7442.16 bytes/sec total size is 180234 speedup is 0.79