Releases/Firefox 3.5 Preview/BuildNotes
Contents
- 1 Build Engineers
- 2 Signed-off Revision(s)
- 3 Tags
- 4 Build data
- 5 Notes
- 5.1 Build 1
- 5.2 Build 2
- 5.2.1 Tag
- 5.2.2 Source
- 5.2.3 Build/Repack
- 5.2.4 Sign
- 5.2.5 L10nVerify
- 5.2.6 Generate updates and push betatest snippets
- 5.2.7 Update verify
- 5.2.8 Update Bouncer
- 5.2.9 Create working 3.5b4 snippets
- 5.2.10 Final checks before push
- 5.2.11 Push to mirrors
- 5.2.12 Final verification
- 5.2.13 Publish updates to beta channel
Build Engineers
nthomas/bhearsum
release tracking bug
Signed-off Revision(s)
Build1: We started from rev:010761f9d36b, at that time the tip. Then landed the version bumps and titlebar/installer tweaks to get rev:46a0538a7ade.
Tags
On release/mozilla-1.9.1:
Build # | Tag | Changeset |
1 | GECKO191b99_20090604_RELBRANCH | 46a0538a7ade |
FIREFOX_3_5b99_BUILD1 | 0d4036dfff71 | |
2 | FIREFOX_3_5b99_BUILD2 FIREFOX_3_5b99_RELEASE | 3f775c3bb24a |
Build data
Type | Build ID | Build machine |
[Windows installer/zip] | Build1:20090605010243 Build2:20090605162636 | |
[Mac compressed] | Build1:20090604233133 Build2:20090605144508 | |
[Linux compressed] | Build1:20090604233331 Build2:20090605144815 |
Notes
Build 1
- clean up directories on slaves, except {linux,macosx,win32}_repack (this would prove to be a bad idea)
- patch buildbot-configs/mozilla2/release_config.py, get review, land
- checkin latest copy of l10n-changesets from tracking bug
- reconfig production-master
- do sendchange
buildbot sendchange --username=nthomas --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5 Preview (b99) build1" goforit
Tag
No problems
Source
No problems
Build/Repack
No problems with en-US, then any slave which did a 3.5b4 locale failed to build this release. In short, they end up with a mozilla-1.9.1 repo on the GECKO191b4_20090423_RELBRANCH, then we hg up -r default, but then can't resolve the FIREFOX_3_5b99_RELEASE tag.
The *_repack dirs were then removed from every slave, the automation patched to make tag, source, en-US builds a no-op by doing a 5 second wait. FIXME: Need an efficient way to clean up the slaves, particularly windows
Get review, land, reconfig master, sendchange:
buildbot sendchange --username=nthomas --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5 Preview (b99) build1 - l10n restart" goforit
Removed .../build1/en-US.xpi, which is not handled properly until bug 485860 is fixed.'
7 locales failed out with ssh errors:
ssh_exchange_identification: Connection closed by remote host
Uploaded fy-NL, si, fi, zh-TW, de, is, gu-IN by hand with:
cd /e/builds/moz2_slave/win32_repack/build/mozilla-1.9.1/browser/locales make upload AB_CD=$locale MOZ_MAKE_COMPLETE_MAR=1 MOZ_PKG_PRETTYNAMES=1 MOZ_PKG_VERSION=3.5b99 UPLOAD_HOST=stage.mozilla.org UPLOAD_SSH_KEY="~/.ssh/ffxbld_dsa" UPLOAD_TO_TEMP=1 UPLOAD_USER=ffxbld POST_UPLOAD_CMD="post_upload.py -p firefox -v 3.5b99 -n 1 --release-to-candidates-dir"
Sign
instructions While signing was running a blocker was found that caused a respin, bug 496515
Build 2
- Merged all the code changes to the relbranch (changeset)
- Cleaned up $platform_repack builddirs on all production slaves. Also cleaned up 'tag' dir on moz2-linux-slave23 and 'source' dir on slave22.
- Landed a config bump for build2 that also backed out the dummy factories that were landed when we had to kick the automation to get l10n to work.
- Kicked off automation with:
buildbot sendchange --username=bhearsum --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5 Preview (b99) build2" mfbt
- forgot to reconfig before doing the sendchange, had to kill every single l10n build, reclean the directories, and start again
- Kicked off automation again with:
buildbot sendchange --username=bhearsum --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5 Preview (b99) build2" mfbt
Tag
No problems
Source
No problems
Build/Repack
- No problems on Linux
- No problems on en-US Mac.
- One Mac locale failed to unpack the en-US DMG file, and then a subsequent one failed beacuse of my mucking around. To fix, had to build and upload them both manually, like so:
# moz2-darwin9-slave07 cd /builds/slave/macosx_repack/build/mozilla-1.9.1/browser/locales make installers-el MOZ_MAKE_COMPLETE_MAR=1 MOZ_PKG_PRETTYNAMES=1 MOZ_PKG_VERSION=3.5b99 ZIP_IN=/builds/slave/macosx_repack/build/mozilla-1.9.1/firefox.dmg make l10n-upload-el MOZ_MAKE_COMPLETE_MAR=1 MOZ_PKG_PRETTYNAMES=1 MOZ_PKG_VERSION=3.5b99 POST_UPLOAD_CMD="post_upload.py -p firefox -v 3.5b99 -n 2 --release-to-candidates-dir" UPLOAD_HOST=stage.mozilla.org UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa UPLOAD_TO_TEMP=1 UPLOAD_USER=ffxbld make installers-sk MOZ_MAKE_COMPLETE_MAR=1 MOZ_PKG_PRETTYNAMES=1 MOZ_PKG_VERSION=3.5b99 ZIP_IN=/builds/slave/macosx_repack/build/mozilla-1.9.1/firefox.dmg make l10n-upload-sk MOZ_MAKE_COMPLETE_MAR=1 MOZ_PKG_PRETTYNAMES=1 MOZ_PKG_VERSION=3.5b99 POST_UPLOAD_CMD="post_upload.py -p firefox -v 3.5b99 -n 2 --release-to-candidates-dir" UPLOAD_HOST=stage.mozilla.org UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa UPLOAD_TO_TEMP=1 UPLOAD_USER=ffxbld
- No problems with win32.
Sign
- Blat failed because of the office move. Had to s/mail.mozilla.com/smtp.mozilla.org/ in the registry. Restarted with:
make sign stage verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} TAG=${TAG} REPO=${REPO} EMAIL=${EMAIL}
- Removed en-US.xpi, and the entry for that in the two SUMS files
- Uploaded
make upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} TAG=${TAG} REPO=${REPO} EMAIL=${EMAIL}
- Uploaded signing log by writing upload-log target (bug 496841) and doing
make upload-log PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} TAG=${TAG} REPO=${REPO} EMAIL=${EMAIL}
- bug 497041: FtpPoller is looking in the wrong place, from twistd.log:
2009-06-07 17:31:08-0700 [-] Polling dir http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.5b4-candidates/build1/
- Manually triggered off l10n_verify and updates
L10nVerify
Failed due to usage of 3.1b4 in config instead of 3.5b4, committed a fix and restarted.
Second attempt looks ok. 'Only in' output matches shipped-locales changes. The general classes of differences are
- bug 488936 moved the locale info from updater.ini to a new file called update.locale, eg
diff -r firefox-3.5b99-build2/diffs/linux-i686.af.diff firefox-3.5b4-build1/diffs/linux-i686.af.diff 193,197d192 < diff -r source/firefox/update.locale target/firefox/update.locale < 1c1 < < en-US < --- < > af 204a200,203 > 9c9 > < Locale=en-US > --- > > Locale=af
- changes to updater, crashreporter.ini and so on show up directly in this log
- some locales changed the searchplugins they shipped
- Path to buildbot slaves differs across pool, recorded in js comment
< > //@line 36 "/builds/moz2_slave/linux_repack/build/releases/l10n-mozilla-1.9.1/oc/browser/firefox-l10n.js" --- > > //@line 36 "/builds/slave/linux_repack/build/releases/l10n-mozilla-1.9.1/oc/browser/firefox-l10n.js"
Generate updates and push betatest snippets
Failed due to usage of 3.1b4 in config instead of 3.5b4, committed a fix and restarted. No problems with second run.
Update verify
Essentially a pass mark, with the following FAIL's detected
- mn and tr missed this release, so there is no update path from 3.5b4 and older
- ml missed 3.5b4, and patcher doesn't know how to generate a 3.1b3 -> 3.5b99 path
- win32 has the usual reports about freebl3.chk & softokn3.chk
Update Bouncer
Done
Create working 3.5b4 snippets
bug 496917 has the details on this. Basically, 'appv=3.5 Beta 99' in snippets broke in 3.5b4. To fix, we need to switch it to 'appv=3.5b99'. Here's how:
# cltbld@aus2-staging.mozilla.org cd ~/ rsync -av /opt/aus2/snippets/staging/20090607-Firefox-3.5b99/ 20090607-Firefox-3.5b99-3.5b4-fix cd 20090608-Firefox-3.5b99-3.5b4-fix/Firefox rm -rf 3.1a1 3.1a2 3.1b1 3.1b2 3.1b3 find . -type f -exec sed -i -e 's/3.5 Beta 99/3.5b99/' {} \; # this next command should print out a bunch of 'appv=3.5b99' lines find . -type f -exec grep appv {} \; cd ../../
# now the test channel rsync -av /opt/aus2/snippets/staging/20090607-Firefox-3.5b99-test/ 20090608-Firefox-3.5b99-test-3.5b4-fix cd 20090608-Firefox-3.5b99-test-3.5b4-fix/Firefox rm -rf 3.1a1 3.1a2 3.1b1 3.1b2 3.1b3 find . -type f -exec sed -i -e 's/3.5 Beta 99/3.5b99/' {} \; find . -type f -exec sed -i -e 's/3.5b99 (build 2)/3.5b99/' {} \; # this next command should print out a bunch of 'appv=3.5b99' lines find . -type f -exec grep appv {} \;
# now rsync these snippets over to the staging area cd ~ rsync -av 20090608-Firefox-3.5b99-3.5b4-fix /opt/aus2/snippets/staging/ rsync -av 20090608-Firefox-3.5b99-test-3.5b4-fix /opt/aus2/snippets/staging/
# backupsnip before repushing the test snippets ~/bin/backupsnip 20090608-Firefox-3.5b99-test-3.5b4-fix ~/bin/pushsnip 20090608-Firefox-3.5b99-test-3.5b4-fix
Final checks before push
Each of the find calls should return no output.
cd /pub/mozilla.org/firefox/nightly/3.5b99-candidates/build2/ find . ! -user ffxbld find . ! -group firefox find . -type f ! -perm 644 find . -type d -mindepth 1 ! -perm 755 ! -name 'contrib*' find . -type d -maxdepth 1 ! -perm 2775 -name 'contrib*'
Got some Solaris builds in contrib/ for the user check, which are OK.
Push to mirrors
# root@stage mkdir /home/ftp/pub/firefox/releases/3.5b99/ chown ffxbld /home/ftp/pub/firefox/releases/3.5b99/
FIXME: need to figure out a better solution than this directory creation.
# ffxbld@stage rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* \ /pub/mozilla.org/firefox/nightly/3.5b99-candidates/build2/ \ /pub/mozilla.org/firefox/releases/3.5b99/
Update the mozilla-current rsync module and latest-3.5 symlink (created this time)
# cltbld@stage nano /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude # replace 3.5b5 with 3.5b99. cd /pub/mozilla.org/firefox/releases/ ln -s 3.5b99 latest-3.5
Final verification
Use 'Force Build to start the final_verification builder, chase down any 503 or 404 errors to find bad mirrors. Get the same warnings as update verify (mn & tr missing 3.5b4 update, ml getting nothing from 3.1b3).
Publish updates to beta channel
~/bin/backupsnip 20090607-Firefox-3.5b99 ~/bin/pushsnip 20090607-Firefox-3.5b99 ~/bin/pushsnip 20090608-Firefox-3.5b99-3.5b4-fix
The additional pushsnip is for "Create working 3.5b4 snippets" above.