Releases/Firefox 3.0.15/BuildNotes

Note: Check with coop/armenzg/bhearsum to make sure nothing has changed before beginning this release

Build Engineer(s)

lsblakk

Tracking release bug

Bonsai queries

cvsroot

l10n

Notes

Build 1

on fx-linux-1.9-slave2 deleted:
- /builds/verify/firefox-3.0.14
- /builds/verify/firefox-3014-353-major
- /builds/updates/firefox-3.0.14
- /builds/source/firefox-3.0.14
- /builds/tags/FIREFOX_3_0_12_BUILD1
- /builds/tags/FIREFOX_3_0_13_BUILD1
- /builds/tags/FIREFOX_3_1a1_RELEASE
- /builds/data/cltbld/firefox-3.0.14
- /builds/tinderbox/Xr-Mozilla1.9-Release/Linux_2.6.18-53.1.13.el5_Depend/

On fx-win32-1.9-slave2 deleted:
- /e/xr19rel/WINNT_5.2_Depend
- /e/fx19l10nrel/WINNT_5.2_Depend
- /e/fx19rel/WINNT_5.2_Depend

On fx-mac-1.9-slave2 deleted:
- /builds/verify/firefox-3.0.14
- /builds/verify/firefox-3014-353-major

Available space on slaves:
- fx-mac-1.9-slave2 19 GB on /
- fx-win32-1.9-slave2 15.9 GB on D: / 14.1 GB on E:
- fx-linux-1.9-slave2 23 GB on /builds

Kicked off automation:

buildbot sendchange --username=lsblakk --master=localhost:9989 -m"Firefox 3.0.15build1 release" go

Tag

No problems on the second run. I had forgotten to tag the bootstrap.cfg files with RELEASE_AUTOMATION_M15 prior to kicking of automation so the first tag attempt failed.

Source

No problems

Build & Repack

Linux - no problems
Mac - no problems
Win32 - no problems

Sign

Signing doc
when signing the files, the first run stopped early, and only signed 41/63 builds so a second attempt was made (after clobbering signed-build1 dir) and on this second run it only got to hi-IN before stopping. Then a third run (again clobbering signed-build1) with a -j1 flag instead of -j4 went through all the way and 63/63 got signed.
first run had both mk error and also said that zw-TW failed at being signed even though it seems to never have gotten past the N's pastebin

L10nVerify

no problems - no l10n changes

Generate Updates

no problems

Publish Updates to Test Channels (betatest & releasetest)

This is now automated in the Update step (see logs for push)

Update Verify

linux - full PASS
win32 - .chk files differ, as always, otherwise find.
mac - full PASS

Stage

No problems

At this point a respin was declared due to bug 521969

Build 2

on fx-linux-1.9-slave2 deleted:
- /builds/verify/firefox-3.0.15
- /builds/updates/firefox-3.0.15
- /builds/source/firefox-3.0.15
- /builds/tags/FIREFOX_3_0_14_BUILD1
- /builds/tags/FIREFOX_3_0_14_BUILD2
- /builds/data/cltbld/firefox-3.0.15
- /builds/tinderbox/Xr-Mozilla1.9-Release/Linux_2.6.18-53.1.13.el5_Depend/

On fx-win32-1.9-slave2 deleted:
- /e/xr19rel/WINNT_5.2_Depend
- /e/fx19l10nrel/WINNT_5.2_Depend
- /e/fx19rel/WINNT_5.2_Depend

On fx-mac-1.9-slave2 deleted:
- /builds/verify/firefox-3.0.15

Available space on slaves:
- fx-mac-1.9-slave2 18 GB on /
- fx-win32-1.9-slave2 15.8 GB on D: / 14.2 GB on E:
- fx-linux-1.9-slave2 22 GB on /builds

Kick off automation

buildbot sendchange --username=lsblakk --master=localhost:9989 -m"Firefox 3.0.15build2 release" go

Tag

No Problems

Source

No Problems

Build & Repack

Mac - no problems
Linux - no problems
Win - no problems

Note: at this point a respin was called due to bug 519589

Build 3

on fx-linux-1.9-slave2 deleted:
- /builds/source/firefox-3.0.15
- /builds/tags/FIREFOX_3_0_15_BUILD*

On fx-win32-1.9-slave2 deleted:
- /e/fx19l10nrel/WINNT_5.2_Depend
- /e/fx19rel/WINNT_5.2_Depend

On fx-mac-1.9-slave2 deleted:
- nothing

Available space on slaves:
- fx-mac-1.9-slave2 18 GB on /
- fx-win32-1.9-slave2 15.8 GB on D: / 14.1 GB on E:
- fx-linux-1.9-slave2 23 GB on /builds

Kick off automation

buildbot sendchange --username=lsblakk --master=localhost:9989 -m"Firefox 3.0.15build3 release" go

Tag

No problems

Source

No problems

Build & Repack

Mac no problems
Linux no problems
Win32 no problems

Sign

Signing Doc
ran with -j1 instead of -j4 to be safe and all were signed the first time run

L10n Verify

No problems

Generate Updates

backupsnip timed out so I had to run it manually on aus2-staging, as well as pushsnip. Triggered update_verify through the waterfall force build

Publish Updates to Test Channels (betatest & releasetest)

Other than having to do manually because of the timeout, no problems

Update Verify

Linux - full pass
Win32 - .chk fail as usual
Mac - full pass

Stage

No problems

XULRunner

No problems

Push updates to beta channel

About an hour before the official go-to-beta time do backupsnip - because 3.5.4 is launching at the same time this backupsnip is run only once for both.

 # cltbld@aus2-staging.m.o
 # make sure scripts are up to date
 cd bin
 cvs update
 cd /opt/aus2/snippets/staging/
 ~/bin/backupsnip 20091016-Firefox-3.0.15-beta

Then on official "go" email do pushsnip:

  ~/bin/pushsnip 20091016-Firefox-3.0.15-beta

We also need to unthrottle the beta channel - patch, review by morgamic (should be done well before going to beta). Pushed live by IT once the MU snippets are overwritten by the 3.0.14 -> 3.0.15 update - bug 523488.

3.0.15 -> 3.5.x Major Update Refresh

version bumps for patcher config and release/update

Snippet Generation

On fx-linux-1.9-slave2:

mkdir -p /builds/3.0.15-3.5.4-major-update/snippets
cd /builds/3.0.15-3.5.4-major-update/snippets

cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log

# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.5.4 3.0.15-3.5.4
cd ../..

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Quick Verify

Check that releasetest = beta = release. New in 3.0.15 we now use a new dir for beta

cd temp/firefox/3.0.15-3.5.4
# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

Push to AUS

New in 3.0.15 - 3 dirs to sync now instead of 2

cd /builds/3.0.15-3.5.4-major-update/snippets/patcher/temp/firefox/3.0.15-3.5.4
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091020-Firefox-3.0.15-3.5.4-MU-test
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091020-Firefox-3.0.15-3.5.4-MU-beta
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091020-Firefox-3.0.15-3.5.4-MU

Enable test snippets

ssh -i ~/.ssh/aus2 cltbld@aus2-staging
cd /opt/aus2/snippets/staging

~/bin/backupsnip 20091020-Firefox-3.0.15-3.5.4-MU-test
~/bin/pushsnip   20091020-Firefox-3.0.15-3.5.4-MU-test

Note backupsnip connection timed out (use SCREEN) and so the backup was tagged as -MAYBE-INCOMPLETE just in case, but nthomas and I looked at the test dir and the snippets looked good so I pushed.

Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-3015-354-major/
cd /builds/verify/firefox-3015-354-major/
hg clone http://hg.mozilla.org/build/tools

# get patch to trawl "Only in" directories
cd tools/release/common
# on mac & linux
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=367544 | patch -p3
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=367544§
patch -p3 < patch

# everyone ...
cd ../updates
platform=linux   # or mac or win32
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Check logs for results - whatever is after "succeeded", known issues are here

Ran into some kind of strange issue on Mac update verify where the the disk was not mounting and there were lots of "file vanished" errors

rsync: opendir "//Library/Logs/Console/501" failed: Permission denied (13)
rsync: opendir "//Library/Logs/Console/503" failed: Permission denied (13)
file has vanished: "//Network/Buildnet/Win32-Slave02"
file has vanished: "//Network/Buildnet/Production-Opsi"

Rebooting fx-mac-1.9-slave2 and running verify.sh again worked as expected. Not sure what the cause of this error was.

Verify succeeded if in the results everything is the same as 3.0.14 -> 3.5.4 MU. aka, everything is OK

Sign Installers

A day or two before the scheduled release day

Done manually using these installer-signing-instructions here. NB: Doc location change, content unmodified.

On Stage complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0.15/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0.15/stage-merged/
~/bin/checksum-files .

Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

Done.

Wait for official "push to mirrors" email...

Push to mirrors

Make sure that you use ffxbld for 'push to mirrors' , rather than cltbld (this is a change, but a desired one).

push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-3.0.15/stage-merged/ /home/ftp/pub/firefox/releases/3.0.15/

edit the exclude file (as cltbld) /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.15) and remove the previous release (3.0.14).

Final Verification

hg clone http://hg.mozilla.org/build/tools
cd tools/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

- No Problems

Before pushing final updates, verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20091016-Firefox-3.0.15
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20091016-Firefox-3.0.15-test/$a");'

Publish Updates to Release Channel

Start doing backupsnip a few hours at least before the release time

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20091016-Firefox-3.0.15
real    62m7.035s
user    0m50.558s
sys     1m16.894s

Wait for "go" to do pushupsnip for Firefox 3.0.15

time ~/bin/pushsnip 20091016-Firefox-3.0.15
real    4m20.110s
user    0m0.495s
sys     0m17.358s

Then again, waiting for go on MU

time ~/bin/pushsnip 20091020-Firefox-3.0.15-3.5.4-MU
real	0m6.366s
user	0m0.016s
sys	0m0.711s

time ~/bin/pushsnip 20091020-Firefox-3.0.15-3.5.4-MU-beta
real	0m1.801s
user	0m0.018s
sys	0m0.557s

We need to unthrottle 3.0.14 on the release channel - patch, should get review from morgamic before release day (rather than last minute!). Pushed live by IT once the MU snippets on 3.0.14 release are overwritten by 3.0.15 - bug 524876.

Release

# ffxbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.0 && ln -s 3.0.15 latest-3.0

Release matching XULRunner build

XULRunner builds are produced by the automation, this is to groom them and push to the mirrors.

# cltbld@stage
VERSION=1.9.0.15
FX_VERSION=3.0.15
BUILD=3

mkdir -p ~/xulrunner-${VERSION}/batch1/stage-unsigned/{runtimes,source,sdk}
cd ~/xulrunner-${VERSION}/batch1/stage-unsigned
rsync -av --exclude=*sdk* --exclude=*info* \
  /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ runtimes/
rsync -av --include=*sdk* --exclude=* \
  /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ sdk/
cp -pv ~/firefox-${FX_VERSION}/stage-merged/source/firefox-${FX_VERSION}-source.tar.bz2 \
  source/xulrunner-${VERSION}-source.tar.bz2
cd ~/xulrunner-${VERSION}/batch1
rsync -av stage-unsigned/ stage-signed/

Then create detached signatures per usual process (with PRODUCT=xulrunner, VERSION=1.9.0.13, don't need BUILD or TAG), skip the win32 signing section, do pgp, verify and upload. Back on stage:

cd ~/xulrunner-${VERSION}/batch1/stage-signed
cp ~/firefox-${FX_VERSION}/stage-merged/KEY .
~/bin/checksum-files .
chown -R cltbld:xulrunner .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ~/xulrunner-${VERSION}/batch1
rsync -nav stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/

If all is fine then

rsync -av stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/

(Mirror updating can take a bit, 10-15 mins this time)

Ping mfinkle or Mossop to update the links in these two wiki pages for the version change

Module	Branch	Tag	Pull date
cvsroot/mozilla	HEAD	GECKO190_20091006_RELBRANCH FIREFOX_3_0_15_BUILD1 FIREFOX_3_0_15_RELEASE	2009-10-05 18:20 PDT
l10n/l10n	HEAD	GECKO190_20091006_RELBRANCH FIREFOX_3_0_15_BUILD1 FIREFOX_3_0_15_RELEASE	2009-09-22 05:34 PDT

Module	Branch	Tag	Pull date
cvsroot/mozilla	HEAD	GECKO190_20091006_RELBRANCH FIREFOX_3_0_15_BUILD2 FIREFOX_3_0_15_RELEASE
l10n/l10n	HEAD	GECKO190_20091006_RELBRANCH FIREFOX_3_0_15_BUILD2 FIREFOX_3_0_15_RELEASE

Releases/Firefox 3.0.15/BuildNotes

Contents

Build Engineer(s)

Bonsai queries

Tags

Notes

Build 1

Tag

Source

Build & Repack

Sign

L10nVerify

Generate Updates

Publish Updates to Test Channels (betatest & releasetest)

Update Verify

Stage

Build 2

Tag

Source

Build & Repack

Build 3

Tag

Source

Build & Repack

Sign

L10n Verify

Generate Updates

Publish Updates to Test Channels (betatest & releasetest)

Update Verify

Stage

XULRunner

Push updates to beta channel

3.0.15 -> 3.5.x Major Update Refresh

Snippet Generation

Quick Verify

Push to AUS

Enable test snippets

Update verify

Sign Installers

Update Bouncer

Push to mirrors

Final Verification

Publish Updates to Release Channel

Release

Release matching XULRunner build

Navigation menu

Search