Releases/Firefox 3.5.4/BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

coop https://bugzilla.mozilla.org/show_bug.cgi?id=519365

Signed-off Revision(s)

Build1: 09522dd1776e
Build2: e178dbd2a389

Tags

On releases/mozilla-1.9.1:

Build # Tag Changeset
1 GECKO1914_20091006_RELBRANCH 09522dd1776e
FIREFOX_3_5_4_BUILD1 FIREFOX_3_5_4_RELEASE f2eb72702e5b
2 GECKO1914_20091006_RELBRANCH 4a35b0f000c7
FIREFOX_3_5_4_BUILD2 FIREFOX_3_5_4_RELEASE e178dbd2a389
3 GECKO1914_20091006_RELBRANCH 49342a1d9d93
FIREFOX_3_5_4_BUILD3 FIREFOX_3_5_4_RELEASE 2fe392234f8c

Build data

Build # Type Build ID Build machine
1 [Windows installer/zip] 20091007001339 moz2-win32-slave44
[Mac compressed] 20091006224018 moz2-darwin9-slave13
[Linux compressed] 20091006225203 moz2-linux-slave38
2 [Windows installer/zip] 20091014223740 moz2-win32-slave47
[Mac compressed] 20091014210210 moz2-darwin9-slave14
[Linux compressed] 20091014211016 moz2-linux-slave35
3 [Windows installer/zip] 20091016092926 moz2-win32-slave52
[Mac compressed] 20091016081620 moz2-darwin9-slave16
[Linux compressed] 20091016081727 moz2-linux-slave36

Notes

  • Useful docs and FAQs here.
  • Done on production-master02

Build 1

  • Cleaned up previous releases' build dirs
    • Went through logs on the master to gather hostnames for linux/win32. Just cleared all possible buildslaves on mac. Used csshX on linux/mac, RDP as Administrator on win32.
      • Linux (moz2-linux-slave[18-23,25])
        • /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
      • Mac (bm-xserve22, moz2-darwin9-slave[13-18])
        • /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
      • Win32 (moz2-win32-slave[32-39]
        • /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*
  • l10n-changesets updated in http://hg.mozilla.org/build/buildbot-configs/rev/61377630502c
  • Land version bump, reconfig production-master02
  • Kicked-off automation:
buildbot sendchange --username=coop --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.4 build1" DOOOOEEEET

Socorro product data updates

  • Pinged IT to add the following entries to the Socorro db:
insert into productdims (product, version, branch, release) values ('Firefox','3.5.4', '1.9.1', 'major');
insert into productdims (product, version, branch, release) values ('Firefox','3.5.5pre', '1.9.1', 'development');

Tag

  • (slave: moz2-linux-slave35) No problems

Source

  • (slave: moz2-linux-slave35) No problems

Build/Repack

  • Linux: No problems
  • Mac: No problems
  • Win32: No problems

Sign

  • Cleaned up Firefox 3.0.12 signing dirs for a little more space (22GB total)
  • First attempt failed, exiting with a make error.
    • Removed the signed-build1/ dir, and tried again successfully using -j1
  • instructions

L10nVerify

  • (slave: moz2-darwin9-slave16) No problems

Generate updates and push betatest snippets

  • 1st attempt: (slave: moz2-linux-slave36) patcher_create_patches failed: failure log
Exception: cmd failed mar -C /builds/slave/updates/build/tmpsVvBD6-fastmode/223 ...
  • 2nd attempt: (slave: moz2-linux-slave30) diff_patcher_config failed (didn't remove bump step) and patcher_create_patches failed: failure log
Exception: cmd failed mar -x /builds/slave/updates/build/temp/firefox/3.5.4/ftp/firefox-3.5.4.sv-SE.win32.complete.mar
  • 3rd attempt: (slave: moz2-linux-slave49) checkout_patcher failed. This slave had never connected to the CVS server before. Updated the linux slave migration doc to include the CVS server.
  • 4th attempt: (slave: moz2-linux-slave47) backupsnip timed out (timeout is 7200s)
  • at this point, I gave up with automation, took moz2-linux-slave47 out of rotation, checked the updates steps for the 3.5.3 run, and ran the remaining commands (backupsnip/pushsnip/wait/upload_snippets) by hand (updated for 3.5.4, of course)
[cltbld@moz2-linux-slave47 build]$ ssh -l cltbld aus2-staging.mozilla.org ~/bin/backupsnip 20091007-Firefox-3.5.4-test
  • OK, that failed too. We hit a hard ssh timeout of (2 hours). joduinn filed bug 521139 to track the slowdown on aus2-staging. The next step was running the command directly on aus2-staging in a screen session:
bash-3.2$ time ~/bin/backupsnip 20091007-Firefox3.5.4-test
Running /bin/tar cfvj /opt/aus2/snippets/backup/20091007-4-pre-20091007-Firefox-3.5.4-test.tar.bz2 .
bash-3.2$ time ~/bin/pushsnip 20091007-Firefox-3.5.4-test
  • The above commands succeeded. backupsnip took 60 minutes, pushsnip took almost 3 minutes.
  • then re-ran the rsync command (upload_snippets) on the original build slave (moz2-linux-slave47):
[cltbld@moz2-linux-slave47 ~]$ cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4
[cltbld@moz2-linux-slave47 3.5.3-3.5.4]$ rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091007-Firefox-3.5.4-beta

Notes: I'm unsure as to why the early mar failures occurred and why the patch generation magically worked on the 4th attempt. I *think* the slaves in question might have been running out of space, but the slaves were being re-used by buildbot so quickly after the failed step that it's hard to tell. The clean_old_builds step passed in all cases, so maybe we need to up the required space?

Update verify

  • Linux: (slave: moz2-linux-slave46) No problems
  • Mac: (slave: bm-xserve22) No problems
  • Win32: (slave: moz2-win32-slave34) No problems

Publish updates to beta channel

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20091007-Firefox-3.5.4-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20091009-1-pre-20091007-Firefox-3.5.4-beta.tar.bz2 .
real    67m23.979s
user    0m55.489s
sys     2m2.785s
~/bin/pushsnip 20091007-Firefox-3.5.4-beta

XULRunner build

Create the builds by going to the buildbot waterfall for mozilla-1.9.1 and forcing builds for linux (not x64), mac and windows xulrunner. Set the branch to 'releases/mozilla-1.9.1' and revision to 'FIREFOX_3_5_4_RELEASE'.

When they finish, as xrbld@stage move the dir from /pub/mozilla.org/xulrunner/nightly/2009/10/2009-10-xx-xx-mozilla-1.9.1/ to ..../nightly/1.9.1.4-candidates/build1. Remove the broken symlink at xulrunner/nightly/2009-10-xx-xx-mozilla-1.9.1. Contact mfinkle and dtownsend for smoketest QA.

RESPIN

Build 2

  • Cleaned up build dirs from build 1
    • Went through builders on the master to gather hostnames for linux/win32/mac. Used cssh on linux/mac, RDP as Administrator on win32.
      • Linux (moz2-linux-slave[30,35,36,38,40,43-50])
        • /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
      • Mac (bm-xserve22, moz2-darwin9-slave[13-18])
        • /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
      • Win32 (moz2-win32-slave[34,44,52-59])
        • /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*

Tag

  • attempt 1 (slave: moz2-linux-slave44): initial reconfig didn't take due to a local typo I introduced while checking that the config was updated. I didn't notice because the reconfig usually times out anyway :/
  • attempt 2 (slave: moz2-linux-slave44): failed because I hadn't cleaned out the release dirs after the previous attempt.
  • attempt 3 (slave: moz2-linux-slave33): third time's the charm.

Source

  • (slave: moz2-linux-slave37) No problems

Build/Repack

  • Linux: No problems
  • Mac: No problems
  • Win32: No problems

Sign

L10nVerify

  • (slave: bm-xserve22) No problems

Generate updates and push betatest snippets

  • first backupsnip completed properly before respin was called, which is better than can be said for build 1.

RESPIN

Build 3

  • Cleaned up build dirs from build 2
    • Went through builders on the master to gather hostnames for linux/win32/mac. Used cssh on linux/mac, RDP as Administrator on win32.
      • Linux (moz2-linux-slave[20,22,33,35,37,44])
        • /builds/slave/{tag,source,linux_build,linux_repack,updates,linux_update_verify,final_verification}/*
      • Mac (bm-xserve[11,19,22], moz2-darwin9-slave14)
        • /builds/slave/{macosx_build,macosx_repack,l10n_verification,macosx_update_verify}/*
      • Win32 (moz2-win32-slave[32,33,39,47])
        • /e/builds/moz2_slave/{win32_build,win32_repack,win32_update_verify}/*

Tag

  • (slave: moz2-linux-slave39) KaiRo re-tagged mozilla-1.9.1 in the middle of our tagging step, so the tagging step failed on mozilla-1.9.1. All locales were tagged fine though
  • to recover, I logged into the slave (moz2-linux-slave39), enabled the hg rebase extension in the .hgrc file, and then ran the following commands:
cd /builds/slave/tag/mozilla-1.9.1
hg pull --rebase
hg push -e 'ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa' -f ssh://hg.mozilla.org/releases/mozilla-1.9.1
  • I then disabled the rebase extension in the .hgrc file again

Source

  • (slave: moz2-linux-slave37) forced from the builder page, no problems

Build/Repack

  • Linux: en-US build forced from the builder page
  • Mac: en-US build forced from the builder page
  • Win32: en-US build forced from the builder page
  • to force the l10n repacks, I had to use the same process as bhearsum used for 3.6b1:
# on coop's machine:
hg clone http://hg.mozilla.org/users/bhearsum_mozilla.com/tools tools-bhearsum
cd tools-bhearsum/buildbot-helpers
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-linux
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-macosx
wget http://people.mozilla.org/~coop/files/3.5.4/shipped-locales-win32
# after linux en-US finished:
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-linux -n coop 2>&1 | tee linux_repack_trigger.log
real	3m11.262s
user	0m0.606s
sys	0m0.274s
# after mac en-US finished
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-macosx -n coop 2>&1 | tee macosx_repack_trigger.log
real	3m9.023s
user	0m0.609s
sys	0m0.293s
# after win32 en-US finished
time python force_release_l10n.py -m http://production-master02.build.mozilla.org:8010 -v -t FIREFOX_3_5_4_RELEASE -s shipped-locales-win32 -n coop 2>&1 | tee win32_repack_trigger.log
real	3m34.634s
user	0m0.612s
sys	0m0.300s
  • NOTE: I sent the initial batch of linux l10n re-triggers to the wrong master (production-master), so I had to kill off a bunch of linux_repack builds there.
  • bug 522989: decided to do a single-locale rebuild on build3 for the Turkish (tr) locale
    • re-tagged changeset fdb83f4b08fc as FIREFOX_3_5_4_BUILD3 and FIREFOX_3_5_4_RELEASE
    • forced builds from the buildbot waterfall: linux_repack, macosx_repack, win32_repack
param value
en_REVISION FIREFOX_3_5_4_RELEASE
l10n_REVISION FIREFOX_3_5_4_RELEASE
locale tr
    • post_upload.py failed while trying to chmod /home/ftp/pub/firefox/nightly/3.5.4-candidates/build3/contrib/solaris_pkgadd,
      • I modified the post_upload.py command on stage.m.o to ignore the contrib dirs
      • I then had to login to the linux/macosx repack slaves and run the following commands:
cd /builds/slave/{linux|macosx}_repack/build/mozilla-1.9.1/browser/locales
export AB_CD=tr
export MOZ_MAKE_COMPLETE_MAR=1
export MOZ_PKG_PRETTYNAMES=1
export MOZ_PKG_VERSION=3.5.4
export POST_UPLOAD_CMD='post_upload.py -p firefox -v 3.5.4 -n 3 --release-to-candidates-dir'
export UPLOAD_HOST=stage.mozilla.org
export UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa
export UPLOAD_TO_TEMP=1
export UPLOAD_USER=ffxbld
make l10n-upload-tr
      • I simply re-ran the win32 repack again as that was faster (once the script on staging was fixed) than trying to get all the env settings right to do it by hand

Sign

  • ran with CONCURRENCY=1 for bug 521080
  • removed signed-builds[1-2] and unsigned-builds[1-2] for 3.5.4 to free up space (19G)
  • instructions
  • signing of build3 single-locale rebuild (tr):
    • set variables
    • moved old unsigned-build3 dir out of the way
    • ran following commands:
cd ~/signing-work/firefox-3.5.4
make download PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
# Removed all non-tr builds/xpis from unsigned-build3
./sign-release.py --first-locale tr --keydir d:/2008-keys -j4 --product firefox -o signed-build3 unsigned-build3 >> win32_signing_build3.log  2>&1
cat ./unsigned-build3/unsigned/win32_info.txt >> ./signed-build3/win32_info.txt
rsync -av ./unsigned-build3/unsigned/win32/xpi ./signed-build3/win32
make checksum-files PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
# Remove existing .asc files for .tr builds from signed-build3, and also for the *SUM files.
make create-sigs PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
# contrib. dirs. already exist, so we just run 'make fix-permissions' rather than 'make stage'
make fix-permissions PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
make verify PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
make fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

L10nVerify

  • (slave: bm-xserve22) No problems

Generate updates and push betatest snippets

  • attempt 1: (slave: moz2-linux-slave30) failed while extracting a mar (log link):
Exception: cmd failed mar -x /builds/slave/updates/build/temp/firefox/3.5.4/ftp/firefox-3.5.4.ta.win32.complete.mar
  • cleaned up slave and forced the step again from the builder page
  • attempt 2: (slave: moz2-linux-slave32) No problems
# connect as cltbld@moz2-linux-slave32
cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4
for dir in aus2.test aus2.beta; do
  # copy complete snippets from 3.5.3 into a new dir for 3.5.4
  mkdir -p b1-b3-${dir}/Firefox
  rsync -a --exclude 'partial.txt' ${dir}/Firefox/3.5.3/ b1-b3-${dir}/Firefox/3.5.4/
  # use the buildid's from build1
  mv b1-b3-${dir}/Firefox/3.5.4/Darwin_Universal-gcc3/20090824085414 b1-b3-${dir}/Firefox/3.5.4/Darwin_Universal-gcc3/20091006224018 
  mv b1-b3-${dir}/Firefox/3.5.4/Linux_x86-gcc3/20090824085743 b1-b3-${dir}/Firefox/3.5.4/Linux_x86-gcc3/20091006225203
  mv b1-b3-${dir}/Firefox/3.5.4/WINNT_x86-msvc/20090824101458 b1-b3-${dir}/Firefox/3.5.4/WINNT_x86-msvc/20091007001339
done
rsync -a -e 'ssh -i ~/.ssh/aus' b1-b3-aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4build1-build3-test
rsync -a -e 'ssh -i ~/.ssh/aus' b1-b3-aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4build1-build3-beta

Then go to aus2-staging and push 20091019-Firefox-3.5.4build1-build3-test live.

  • Updates for tr re-build required some manual work. Logged on to the slave that generated the updates (moz2-linux-slave32), and
cd /builds/slave/updates/build/patcher-configs.

Narrowed the locale list in moz191-branch-patcher2.cfg to include only 'tr' for the 3.5.3 and 3.5.4 blocks, and then ran the following commands:

cd /builds/slave/updates/build
perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-patcher2.cfg
find temp/firefox/3.5.3-3.5.4/ftp/firefox/nightly/3.5.4-candidates/build3 -type f -exec chmod 644 {} \;
find temp/firefox/3.5.3-3.5.4/ftp/firefox/nightly/3.5.4-candidates/build3 -type d -exec chmod 755 {} \;
cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4/ftp/firefox/nightly/3.5.4-candidates/build3
rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' --exclude=*complete.mar update ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/3.5.4-candidates/build3/
cd /builds/slave/updates/build/temp/firefox/3.5.3-3.5.4
rsync -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4-from-3.5.3
rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4-from-3.5.3-test
rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091019-Firefox-3.5.4-from-3.5.3-beta
  • pushed tr updates live:
# cltbld@aus2-staging
~/bin/pushsnip 20091019-Firefox-3.5.4-from-3.5.3-test

Update verify

  • Linux: (slave: moz2-linux-slave37) No problems.
  • Mac: (slave: moz2-darwin9-slave13) No problems.
  • Win32: (slave: moz2-win32-slave45) No problems.

Publish updates to beta channel

backupsnip was run once see here and encompasses both 3.0.15 and 3.5.4

# cltbld@aus2-staging
# Regular beta snippets
~/bin/pushsnip 20091016-Firefox-3.5.4-beta
# build1->build3 snippets
~/bin/pushsnip 20091019-Firefox-3.5.4build1-build3-beta
# tr build3 rebuild snippets
~/bin/pushsnip 20091019-Firefox-3.5.4-from-3.5.3-beta

Update Bouncer

No problems.

Final checks before push

cd /pub/mozilla.org/firefox/nightly/3.5.4-candidates/build3/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -type d -mindepth 1 ! -perm 755 ! -name 'contrib*'
find . -type d -maxdepth 1 ! -perm 2775  -name 'contrib*'

Only returned Solaris builds (different user). This was expected.

Push to mirrors

# ffxbld@stage
rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/mozilla.org/firefox/nightly/3.5.4-candidates/build3/ /pub/mozilla.org/firefox/releases/3.5.4/
# cltbld@stage
vim /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.5.3 with 3.5.4

Final Verification

Kicked off from pm02 waterfall - no problems with any links.

Push updates to release channel

Backupsnip had been run for 3.0.15.

# cltbld @ aus2-staging
time ~/bin/pushsnip 20091016-Firefox-3.5.4
real    1m54.901s
user    0m0.244s
sys     0m7.906s
~/bin/pushsnip 20091019-Firefox-3.5.4-from-3.5.3
real    0m1.187s
user    0m0.005s
sys     0m0.069s

Release

cd /home/ftp/pub/firefox/releases 
rm latest-3.5 && ln -s 3.5.4 latest-3.5

XULRunner build

Created the builds by going to the buildbot waterfall for mozilla-1.9.1 and forcing builds for linux (not x64), mac and windows xulrunner. Set the branch to 'releases/mozilla-1.9.1' and revision to 'FIREFOX_3_5_4_RELEASE'.

When they finished, as xrbld@stage moved the dir from /pub/mozilla.org/xulrunner/nightly/2009/10/2009-10-28-07-mozilla-1.9.1/ to ..../nightly/1.9.1.4-candidates/build3. Removed the broken symlink at xulrunner/nightly/2009-10-28-07-mozilla-1.9.1. Contacted mfinkle and dtownsend for smoketest QA.

Publishing is done in a similar (but not identical!) way as 3.0.13:

# cltbld@stage
VERSION=1.9.1.4
FX_VERSION=3.5.4
BUILD=1

mkdir -p ~/xulrunner-${VERSION}/batch1/stage-unsigned/{runtimes,source,sdk}
cd ~/xulrunner-${VERSION}/batch1/stage-unsigned
rsync -av --exclude=*sdk* --exclude=*info* \
  /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ runtimes/
rsync -av --include=*sdk* --exclude=* \
  /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ sdk/
cp -pv /pub/mozilla.org/firefox/releases/${FX_VERSION}/source/firefox-${FX_VERSION}.source.tar.bz2 \
  source/xulrunner-${VERSION}.source.tar.bz2
cd ~/xulrunner-${VERSION}/batch1
rsync -av stage-unsigned/ stage-signed/

Then create detached signatures per usual process (with PRODUCT=xulrunner, VERSION=1.9.1.4, don't need BUILD or TAG), skip the win32 signing section, do pgp, verify and upload. Back on stage:

cd ~/xulrunner-${VERSION}/batch1/stage-signed
cp /pub/mozilla.org/firefox/releases/${FX_VERSION}/KEY .
~/bin/checksum-files .
chown -R cltbld:xulrunner .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ~/xulrunner-${VERSION}/batch1
rsync -nav stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/

If all is fine then

rsync -av stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/

Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!