Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

Signed-off Revision(s)

L10N changesets

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20110517192056	mv-moz2-linux-ix-slave05	1 hrs, 5 mins
	Linux64		moz2-linux64-slave03	1 hrs, 41 mins
	Mac		moz2-darwin10-slave26	4 hrs, 27 mins
	Windows		mw32-ix-slave14	3 hrs, 6 mins

Notes

Build 1

Setup

set clobber for pm03:8010 for all builders on mozilla-beta
reserved slaves set to 6
Hit 'ship' button for Firefox 5 Beta Build 2 (fx5_beta_b2) at l10n milestones page
landed configs, merged to production
manually tagged buildbot-configs, buildbotcustom, tools with FIREFOX_5_0b2_RELEASE, FIREFOX_5_0b2_BUILD1
updated on pm03 & pm01 - buildbot-configs, buildbotcustom, tools

Dry run:

PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u nthomas -V 5.0b2 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py --dryrun localhost:9010

Ran without --dry-run to reconfig & start the release:

PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u nthomas -V 5.0b2 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py localhost:9010

Tag

No problems.

Update Bouncer

Buildbot step succeeded. Had to fix locations for mac and win32 'installers', filed bug 658227 to fix the automation. Couldn't remove unused mac partial location because of bug 629404.

Source

No problems.

Send mail to metrics

Let them know the buildID is 20110517192056.

Build/Repack

Linux en-US was fine, but repacks failed out because the l10n-mozconfig's were out of date. The mozconfigs got patched, see the bug comment for an explanation. Manually clobbered the working dirs and used rebuild to redo the linux jobs. Forced the repack_complete builder.

Linux64 OK.

Mac and possibly Windows were affected by bug 657882, mac fatally. After patching the code, and manually clobbering the 6 mac repack slaves, used rebuild to redo. Left the already completed windows builds alone, they must just be using the zip file to create all localized files. The af installer seems to work fine.

Unittests & Talos

All the talos results failed to post results to graphs - reopened bug 653328.

Unittest results didn't show up on the Mozilla-Beta-Release tree. Turns out we're not doing a sendchange for them, filed bug 659254.

Signing

Auto-sign was used.

L10N Verify

Can only verify binary differences here since 5.0b1 was en-US only. None were found looking at the win32 diffs on the build slave.

Updates

This section is the automated 5.0b1 -> 5.0b2 updates. Since this is the first set of updates we setup the patcher and update verify configs before hand.

Update Verify

Linux and Linux64 were green. Checked with Rob Strong that the non-fatal error at the end of the updater log,

...
FINISH REMOVEDIR searchplugins/
non-fatal error removing directory: searchplugins/, rv: 0, err: 39
succeeded
calling QuitProgressUI

is expected.

Mac and Windows failed because they tried to download 'installers' with the old-style pretty names - bug 658083. Fixed the config.

Mac would have also failed because of the broken partial discovered by QA - bug 658094. Pulled the mac partial to resolve this:

# cltbld@aus2-staging
# staging snippets
cd /opt/aus2/snippets/staging/
rsync -a Firefox-5.0b2-build1-test/ Firefox-5.0b2-build1-withMacPartial-test/
rsync -a Firefox-5.0b2-build1/ Firefox-5.0b2-build1-withMacPartial/
find Firefox-5.0b2-build1{-test,}/Firefox/5.0/Darwin* -name partial.txt -exec rm {} \;
# live snippets
cd /opt/aus2/incoming/3/Firefox/5.0/
find Darwin* -name 'partial.txt' -exec rm -v {} \;
# and repeat in PHX

# ffxbld@stage - move the partial away
mkdir -p ~/5.0b2-broken-partial-bug658094/update/mac/en-US
mv /pub/mozilla.org/firefox/nightly/5.0b2-candidates/build1/update/mac/en-US/firefox-5.0b1-5.0b2.partial.mar ~/5.0b2-broken-partial-bug658094/update/mac/en-US

Force building the mac and windows update verify gives mac failure on missing partial (expected) but complete OK. Windows was green. So all OK now.

Updates from 4.0b8-4.0.1

A manually crafted patcher config + update verify configs, plus manual snippet generation (FIXME automate to death).

Generation was done on mv-moz2-linux-ix-slave05, using the new UPDATE_PACKAGING_R14 from bug 657597 except for the en-US pull for the mar tools.

cd /builds/tmp/

# checkout patcher
cvs -d :ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R14 -d bug654507-4.0-5.0b2-update mozilla/tools/patcher

cd bug654507-4.0-5.0b2-update
# get utility libraries
cvs -d :ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R14 -d MozBuild mozilla/tools/release/MozBuild
cvs -d :ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R14 -d Bootstrap mozilla/tools/release/Bootstrap/Util.pm
hg clone http://hg.mozilla.org/build/tools/

cvs -d :ext:cltbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs

export HGROOT=http://hg.mozilla.org/releases/mozilla-2.0 
perl patcher2.pl --build-tools-hg --tools-revision=UPDATE_PACKAGING_R13 --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg

perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg

cd temp/firefox
ln -s 5.0b2 4.0.1-5.0b2
cd ../..

perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg

# remove partials for 4.0.1
cd temp/firefox/4.0.1-5.0b2
find aus2{,.test} -name partial.txt -exec rm {} \;

# check releasetest == beta
bash /builds/tmp/bug654507-4.0-5.0b2-update/tools/release/compare-channel-snippets.sh aus2 beta aus2.test releasetest

# push up to aus
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b2-build1-4.0.1-build1-test
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b2-build1-4.0.1-build1

# push test snippets live
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/backupsnip Firefox-5.0b2-build1-4.0.1-build1-test'
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip Firefox-5.0b2-build1-4.0.1-build1-test'

Update Verify for 4.0b8-4.0.1

All four platforms failed, but in ways which are only an artifact of how we are testing and are effectively green.

no updates found for 4.0.1 locales ak, bg, bn-BD, cy, es-AR, id, ja, ja-JP-mac, ku, mk, nn-NO, or, sr, ta, ta-LK, th, vi - these locales weren't built for 5.0b2 but we have all 4.0.1 locales in the update verify config (those locales do find updates from 4.0rc2 and older, but get 4.0.1)
no partials found for any 4.0.1 locale - fake partials were explicitly removed so that all the 4.0 versions have only a complete but patch_types="partial complete" was left in the update verify config
all the updates leave a diff like this, which is expected:

diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js
1,2c1,2
< //@line 2 "/builds/slave/rel-2.0-lnx-bld/build/browser/app/profile/channel-prefs.js"
< pref("app.update.channel", "release");
---
> //@line 2 "/builds/slave/rel-beta-lnx-bld/build/browser/app/profile/channel-prefs.js"
> pref("app.update.channel", "beta");
WARN: non-binary files found in diff

a couple also leave some searchplugins/ behind (Only in source/firefox/...), which look normal productization changes
- en-ZA: answers.xml, creativecommons.xml
- ro: creativecommons.xml

Free the reserved slaves

# production-master03:/builds/buildbot/builder_master
echo 0 > reserved_slaves_pm03

Filed bug 659256 to get a nagios alert on this so we don't forget. Again.

Generate channel switching snippets

For aurora -> beta switches we need a set of snippets for all the available 5.0b2 locales, which is pretty much what the 4.0.1 ones are. We put them in a bogus buildID of 30000000000000 to keep the channel switching snippets separate from more specific ones.

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
mkdir -p Firefox-5.0b2-build1-channel-switching/Firefox/
rsync -a Firefox-5.0b2-build1-4.0.1-build1/Firefox/4.0.1 Firefox-5.0b2-build1-channel-switching/Firefox/
cd Firefox-5.0b2-build1-channel-switching/Firefox/
mv 4.0.1 5.0
for d in 5.0/*; do 
  mv -v $d/* $d/30000000000000
done

Don't need to remove updateType=major like 5.0b1 from these because they're already minor updates.

Socorro product data updates

5.0 was already added for 5.0b1, nothing to do here.

Permissions and virus check

Both green

Notify Mirrors

N/A, see Push files for explanation.

Notify A/V vendors

No problems with automation.

Push files

We're switching to using bouncer rather than ftp.m.o for beta builds, and will cover the load with dm-download02, pv-mirror01, and pv-mirror02. The main set of mirrors were excluded by adding

- firefox/releases/5.0b2

to stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude before the pushed. Consequently the highest uptake we should see will be 3.

Force the "release-mozilla-beta-push_to_mirrors" builder with the following properties:

   release_config: mozilla/release-firefox-mozilla-beta.py
   script_repo_revision: FIREFOX_5_0b2_RELEASE

Final verify

Wasn't expecting the uptake check to fire emails for enough coverage for releasetest because the expected uptake would only be 3, and to need to watch this instead. But actually the two 3crowd CDNs came online (along with a couple of other random mirrors) so we got to nearly 5k uptake.

Final verification failed because the partial snippets couldn't be found. Created fake zh-TW partials so sentry would pick them up:

cd /pub/mozilla.org/firefox/releases/5.0b2/update
for p in linux-i686 linux-x86_64 win32/; do touch $p/zh-TW/firefox-5.0b1-5.0b2.partial.mar; done

Push snippets

Snippets directories:

5.0b1 -> 5.0b2:    Firefox-5.0b2-build1
4.0   -> 5.0b2:    Firefox-5.0b2-build1-4.0.1-build1
Channel switching: Firefox-5.0b2-build1-channel-switching

Backups before hand (fast & slower respectively)

# cltbld@aus2-staging
~/bin/backupsnip Firefox-5.0b2-build1
~/bin/backupsnip Firefox-5.0b2-build1-4.0.1-build1
~/bin/backupsnip Firefox-5.0b2-build1-channel-switching

Drivers wished to throttle before enabling updates to 5.0b2, so landed, tagged and asked for deployment of the AUS patch.

At request of driver

~/bin/pushsnip Firefox-5.0b2-build1
~/bin/pushsnip Firefox-5.0b2-build1-4.0.1-build1
~/bin/pushsnip Firefox-5.0b2-build1-channel-switching

Unthrottle updates

AUS patch got backed out, tagged, and deployed.

Build #	Tag	Changeset
1	GECKO50_2011051718_RELBRANCH	2b3275216413
1	FIREFOX_5_0b2_RELEASE FIREFOX_5_0b2_BUILD1	40ea1355db3f

Releases/Firefox 5.0b2/BuildNotes

Contents