Until we're about to ship this page will serve as a spot to put down a rough checklist of what needs to be done to ship.

Release Engineers

bhearsum,nthomas

Tracking Bug

bug 500442

Notes

Generate major update

Updated the patcher and update verify configs.

# ssh cltbld@prometheus-vm.build.mozilla.org
mkdir -p /builds/3.0.11-3.5rc3build2-major-update-test/snippets
cd /builds/3.0.11-3.5rc3build2-major-update-test/snippets

cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R8 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R8 -d MozBuild mozilla/tools/release/MozBuild

export CVSROOT=:ext:ffxbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R8 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log

# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.5rc3 3.0.11-3.5rc3
cd ../..

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Separate out beta snippets

Skipped over this because beta and release will be pushed together, done like this.

Quick verify

Check that releasetest = beta = release.

cd temp/firefox/3.0.11-3.5rc3
# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2 -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; system("diff -r -u $_ $a");'

Push to AUS

cd /builds/3.0.11-3.5rc3build2-major-update-test/snippets/patcher/temp/firefox/3.0.11-3.5rc3
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20090625-Firefox-3.0.11-3.5rc3build2-MU-test
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20090625-Firefox-3.0.11-3.5rc3build2-MU

Enable test snippets

# cltbld@aus2-staging
~/bin/backupsnip 20090625-Firefox-3.0.11-3.5rc3build2-MU-test
~/bin/pushsnip   20090625-Firefox-3.0.11-3.5rc3build2-MU-test

Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-3011-35rc3build2-major-test/
cd /builds/verify/firefox-3011-35rc3build2-major-test/
hg clone http://hg.mozilla.org/build/tools

# get patch to trawl "Only in" directories
cd tools/release/common
# on mac & linux
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=367544 | patch -p3
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=367544
patch -p3 < patch

# everyone ...
cd ../updates
platform=linux   # or mac or win32
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Results Everything the same as the latest test run with one addition:

• mn removed amazondotcom.xml, answers.xml, eBay.xml and yahoo.xml bug 477989

Use empty snippets for mn and tr snippets

bug 500642 was filed after the MU update generation was done. We can't just regenerate the snippets because test snippets were already pushed, so we'll modify the MU snippets to serve nulls to these locales:

# cltbld @ aus2-staging
cd ~/
# test snippets
rsync -av /opt/aus2/snippets/staging/20090625-Firefox-3.0.11-3.5rc3build2-MU-test/ 20090625-Firefox-3.0.11-3.5rc3build2-MU-test/
cd 20090625-Firefox-3.0.11-3.5rc3build2-MU-test
for i in `find . -name mn -or -name tr`; do for file in `find $i -type f`; do rm $file && touch $file; done; done
# verification - should have no output
cd ..
diff --brief -Nar /opt/aus2/snippets/staging/20090625-Firefox-3.0.11-3.5rc3build2-MU-test/  20090625-Firefox-3.0.11-3.5rc3build2-MU-test/ | grep -v '/\(tr\|mn\)/'
# push them back to snippets/staging
rsync -av 20090625-Firefox-3.0.11-3.5rc3build2-MU-test/ /opt/aus2/snippets/staging/20090625-Firefox-3.0.11-3.5rc3build2-MU-test-no-mn-or-tr/
# real snippets
cd ~/
rsync -av /opt/aus2/snippets/staging/20090625-Firefox-3.0.11-3.5rc3build2-MU/ 20090625-Firefox-3.0.11-3.5rc3build2-MU/
cd 20090625-Firefox-3.0.11-3.5rc3build2-MU
for i in `find . -name mn -or -name tr`; do for file in `find $i -type f`; do rm $file && touch $file; done; done
# verification - should have no output
cd ..
diff --brief -Nar /opt/aus2/snippets/staging/20090625-Firefox-3.0.11-3.5rc3build2-MU/  20090625-Firefox-3.0.11-3.5rc3build2-MU/ | grep -v '/\(tr\|mn\)/'
# push them back to snippets/staging
rsync -av 20090625-Firefox-3.0.11-3.5rc3build2-MU/ /opt/aus2/snippets/staging/20090625-Firefox-3.0.11-3.5rc3build2-MU-no-mn-or-tr/

Re-verify that releasetest = beta = release

cd /opt/aus2/snippets/staging
# releasetest == beta
find 20090625-Firefox-3.0.11-3.5rc3build2-MU-test-no-mn-or-tr -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/20090625-Firefox-3.0.11-3.5rc3build2-MU-test-no-mn-or-tr/20090625-Firefox-3\.0\.11-3\.5rc3build2-MU-no-mn-or-tr/; system("diff -r -u $_ $a");'
find 20090625-Firefox-3.0.11-3.5rc3build2-MU-no-mn-or-tr -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/20090625-Firefox-3\.0\.11-3\.5rc3build2-MU-no-mn-or-tr/20090625-Firefox-3.0.11-3.5rc3build2-MU-test-no-mn-or-tr/; system("diff -r -u $_ $a");'

# beta == release
find 20090625-Firefox-3.0.11-3.5rc3build2-MU-no-mn-or-tr -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find 20090625-Firefox-3.0.11-3.5rc3build2-MU-no-mn-or-tr -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; system("diff -r -u $_ $a");'

Push updated test snippets

# cltbld@aus2-staging
~/bin/backupsnip 20090625-Firefox-3.0.11-3.5rc3build2-MU-test-no-mn-or-tr
~/bin/pushsnip   20090625-Firefox-3.0.11-3.5rc3build2-MU-test-no-mn-or-tr

Stage and Rename files

We'll have to do something similar to Firefox 3.0 with at least the following caveats:

• Don't trim out contrib builds if they exist
• Work is done as ffxbld

# ffxbld@stage
mkdir firefox-3.5
cd firefox-3.5
rsync -av --exclude='*partial.mar' --exclude=source --exclude=*SUMS /pub/mozilla.org/firefox/releases/3.5rc3/ .
find contrib linux-i686 update -type f | perl -nle '$nf = $_; if ( $nf =~ s/rc3// ) { system("mv","-v",$_,$nf)}' 
find mac win32 -type f | perl -nle '$nf = $_; if ( $nf =~ s/\ RC\ 3// ) { system("mv","-v",$_,$nf)};'
wget -O../firefox-3.5-shipped-locales http://hg.mozilla.org/releases/mozilla-1.9.1/raw-file/a625a31a0ad1/browser/locales/shipped-locales
~cltbld/bin/verify-locales.pl -m ~/firefox-3.5-shipped-locales -l bz2
~cltbld/bin/checksum-files .
chown -R ffxbld:firefox linux-i686 mac update win32
chmod 644 *SUMS

Content verification

find /pub/mozilla.org/firefox/releases/3.5rc3/ -type f | perl -nle '$a = substr($_,41); $a =~ s/rc3//; $a =~ s/\ RC\ 3//; system("diff","-q",$_,$a);'
# Output:
diff: contrib/solaris_tarball/firefox-3.5b99-3.5.en-US.opensolaris-i386.partial.mar: No such file or directory
diff: source/firefox-3.5-source.tar.bz2: No such file or directory
diff: source/firefox-3.5-source.tar.bz2.asc: No such file or directory
diff: source/firefox-3.5.bundle: No such file or directory
diff: source/firefox-3.5.bundle.asc: No such file or directory
diff: update/linux-i686/af/firefox-3.5rc2-3.5.partial.mar: No such file or directory
diff: update/linux-i686/ar/firefox-3.5rc2-3.5.partial.mar: No such file or directory
...
diff: update/win32/zh-CN/firefox-3.5rc2-3.5.partial.mar: No such file or directory
diff: update/win32/zh-TW/firefox-3.5rc2-3.5.partial.mar: No such file or directory
Files /pub/mozilla.org/firefox/releases/3.5rc3/MD5SUMS and MD5SUMS differ
Files /pub/mozilla.org/firefox/releases/3.5rc3/SHA1SUMS and SHA1SUMS differ

Everything above is expected: We don't want the partial, we regenerated *SUMS because of filename changes, and we haven't yet regenerated source packages

Ownernship/permission verification

find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -type d -mindepth 1 ! -perm 755 ! -name 'contrib*'
find . -type d -maxdepth 1 ! -perm 2775  -name 'contrib*'

contrib/solaris_* dirs are 2755, which is OK.

Add index files

for dir in `find . -type d -mindepth 1`; do cp -p index.html $dir/; done

Tag and Source package regeneration

Because we have so many l10n repositories to deal with it will be best to do this with the automation. We'll need to patch both release_config.py. [This patch] was used to do so.

We have to call it build2 due to the way relbranchoverride works (bug 500471 bug 500473). You don't need to touch the old* vars since Tag and Source don't use them, but you can if you want. The Source packages will be pushed to 3.5-candidates/build2. Those files should be quickly moved to ~ffxbld and disposed of or moved.

After landing the patch, manually kicked off the tag factory. Once that completed, kicked off source. Once that completed, ran the following to grab and then wipe out the files:

# ffxbld @ stage
rsync -av /home/ftp/pub/firefox/nightly/3.5-candidates/ 3.5-candidates/ && rm -rf /home/ftp/pub/firefox/nightly/3.5-candidates

Then, to stage:

cd ~/firefox-3.5
rsync -av ~/3.5-candidates/build2/source/ source/

Then, pull the source/ dir in on the signing machine and generate a sig for it:

mkdir -p ~/signing-work/firefox-3.5
cd ~/signing-work/firefox-3.5
rsync -av -e 'ssh -i /home/cltsign/.ssh/ffxbld_dsa' ffxbld@stage.mozilla.org:~/firefox-3.5/source/ stage-unsigned/
rsync -av stage-unsigned/ stage-signed/
cd stage-signed/
sign-files .
cd ..
# should only push the asc here
rsync -nav -e 'ssh -i /home/cltsign/.ssh/ffxbld_dsa' stage-signed/ ffxbld@stage.mozilla.org:~/firefox-3.5/source/ 
rsync -av -e 'ssh -i /home/cltsign/.ssh/ffxbld_dsa' stage-signed/  ffxbld@stage.mozilla.org:~/firefox-3.5/source/ 

Then back on stage, regenerate SUMS files:

cd ~/firefox-3.5
rm *SUMS
~cltbld/bin/checksum-files .

Bouncer Setup

Done

Push to Mirrors

• push the covering index.html in the top level (Monday 2240 EDT/1940 PDT)

# ffxbld@stage
rsync -av /home/ffxbld/firefox-3.5/index.html /pub/mozilla.org/firefox/releases/3.5/
# 0045 EDT/2145 PDT
rsync -av --exclude 'contrib*' --exclude '*.exe' --exclude '*.dmg' \
  --exclude '*.bz2' --exclude '*.asc' --exclude '*.mar' --exclude '*.xpi' \
  --exclude '*.bundle' \
  /home/ffxbld/firefox-3.5/ /pub/mozilla.org/firefox/releases/3.5/

• push the stage-merged directory to the releases area (Tuesday 0100 EDT/Monday 2200 PDT)

 rsync -av /home/ffxbld/firefox-3.5/ /pub/mozilla.org/firefox/releases/3.5/

Update rsync modules

As cltbld, replaced two references to 3.5rc3 with 3.5 in /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude

Trim mk builds out

mk builds had a horrible bug that prevents updating so we decided not to ship it. bug 501344. First, trim the builds out of the releases dir

# ffxbld @ stage
# start in the staging dir
cd ~/firefox-3.5
find . -type d -iname mk -exec rm -r {} \;
# update SUMS files
sed -i -e '/^.*mk.*$/d' SHA1SUMS
sed -i -e '/^.*mk.*$/d' MD5SUMS
# then rsync over to releases
rsync -av --delete ./ /pub/mozilla.org/firefox/releases/3.5/

Now, trim out the MU snippets

# cltbld @ aus2-staging
# get rid of the live test snippets first
cd /opt/aus2/incoming/3/Firefox/3.0.11
find . -maxdepth 3 -type d -wholename "*200906021*/mk*" -exec rm -r {} \;
# and now get rid of the beta+release channel snippets in the staging dir 
rsync -av 20090625-Firefox-3.0.11-3.5rc3build2-MU-no-mn-or-tr/ 20090625-Firefox-3.0.11-3.5rc3build2-MU-no-mn-or-tr-or-mk/
cd 20090625-Firefox-3.0.11-3.5rc3build2-MU-no-mn-or-tr-or-mk/Firefox/3.0.11
find . -maxdepth 3 -type d -wholename "*200906021*/mk*" -exec rm -r {} \;

Push Major Update

Prep

time ~/bin/backupsnip 20090625-Firefox-3.0.11-3.5rc3build2-MU-no-mn-or-tr
real    50m34.052s
user    0m48.859s
sys     2m17.473s

Had to re-run backupsnip because it was invalidated after we trimmed 3.0.11 mk snippets:

time ~/bin/backupsnip 20090625-Firefox-3.0.11-3.5rc3build2-MU-no-mn-or-tr-or-mk
real	52m38.078s
user	0m47.961s
sys	1m35.855s

~/bin/pushsnip 20090625-Firefox-3.0.11-3.5rc3build2-MU-no-mn-or-tr-or-mk

Symlink update

Update the 'latest' symlink.

cd /pub/mozilla.org/firefox/releases
rm latest-3.5 && ln -s 3.5 latest-3.5
rm latest     && ln -s latest-3.5 latest

Remove index.html files

Once we are fully announced we can remove the index.html files:

cd /pub/mozilla.org/firefox/releases/3.5/
find . -name index.html | less
find . -name index.html | xargs rm -v