Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

Signed-off Revision(s)

Build 1: cf3d01c081fd Build 2: 612ad84a4004

L10N changesets

Updates l10n-changesets obtained from the l10n shipping dashboard.

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20101122054930	linux-ix-slave13	39 mins, 6 secs
	Mac	20101122054656	bm-xserve19	1 hrs, 24 mins, 31 secs
	Windows	20101122063257	mw32-ix-slave10	1 hrs, 26 mins, 43 secs
2	Linux	20101130074614	linux-ix-slave13	36 mins, 39 secs
	Mac		bm-xserve19
	Windows		mw32-ix-slave10

Notes

using pm01

Build 1

first buildbot 0.8.x release
Clobbered some release builders via clobberer, mostly xulrunner ones
No slave movement prior to kicking off automation because pm01 already has Linux slaves
Landed updated automation config
Updated configs and buildbotcustom
Removed *.pyc
Manually tagged buildbot-configs, buildbotcustom, tools, compare-locales with FIREFOX_3_5_16_RELEASE and FIREFOX_3_5_16_BUILD1
ran the following command to kick-off the release:

buildbot sendchange --username=coop --master=localhost:9010 --branch=releases/mozilla-1.9.1 --revision=FIREFOX_3_5_16_RELEASE -p release_config:mozilla/release-firefox-mozilla-1.9.1.py DOOOEEET

Tag

No problems.

Source

No problems.

Build/Repack

Linux en-US: no problems
Linux repacks:
- bug 613944 caused some repack jobs to fail initially.
- rebuilt failures using the button on the builder page in the waterfall
- builders 7, 9, 11, 12, and 14 all had upload errors for some locales
  - ran the following script on the affected slaves to re-upload the missing locales:

#!/bin/bash
set -x
export MOZ_MAKE_COMPLETE_MAR=1
export UPLOAD_USER=ffxbld
export UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa
export UPLOAD_HOST=stage.mozilla.org
export POST_UPLOAD_CMD='post_upload.py -p firefox -n 1 -v 3.5.16 --release- to-candidates-dir'
export UPLOAD_TO_TEMP=1
export MOZ_PKG_PRETTYNAMES=1
export AB_CD=$1
export BUILDER=$2

cd /builds/slave/release-mozilla-1.9.1-linux_repack_${BUILDER}/build/mozilla-1.9.1/obj-l10n/browser/locales
make l10n-upload-${AB_CD}

- - then ran the following command on stage to verify that we had 75 locales:

cd /home/ftp/pub/firefox/nightly/3.5.16-candidates/build1 &&ls linux-i686 | grep -v xpi | wc -l

Mac en-US: No problems.
Mac repacks: same upload issues as for linux
- used script on various slaves to re-upload
- verified # of locales on stage
Win32 en-US: No problems.
Win32 repacks: same upload issues as for linux
- used script on various slaves to re-upload
- verified # of locales on stage

Sign

Removed en-US.xpi from candidates directory

PRODUCT=firefox
VERSION=3.5.16
BUILD=1
TAG=FIREFOX_3_5_16_RELEASE
REPO='releases/mozilla-1.9.1'
EMAIL=coop@mozilla.com
KEYDIR=d:/2010-keys
cd ~/hg-tools
hg pull && hg up -r ${TAG}
mkdir ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
make PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

And then to upload:

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

L10N Verify

All three platforms "failed", but for legitimate string changes and absolute path changes due to the builder rename, e.g.

19331c19333
< < //@line 36 "/builds/slave/release-mozilla-1.9.1-linux_build/build/browser/locales/en-US/firefox-l10n.js"
---
> < //@line 36 "/builds/slave/linux_build/build/browser/locales/en-US/firefox-l10n.js"

Updates

No problems

Update Verify

Linux: No problems
Mac: No problems
Win32: No problems

Socorro product data updates

Added '3.5.16' and '3.5.17pre' to crash-stats with this admin page.

Update Bouncer

Added by forcing bouncer_submitter builder

Major Update Generation (-> 3.6.13)

Used 'force build' to start the major_update builder.

Major Update Verification

mn fails for all platforms, normal and expected.
sl blurts the following, normal and expected:

Only in source/firefox/searchplugins: iskalnik-gov-si.xml
Only in source/firefox/searchplugins: najdi-si.xml
Only in source/firefox/searchplugins: yahoo.xml

Linux:
some path differences:
- /build/slave vs /build/moz2_slave (damn naming inconsistency, expected)
- builder name change (expected)
- ru:

Only in source/firefox/searchplugins: rambler.xml

Mac: the usual mac differences:

diff -r source/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js target/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js
1c1
 < //@line 2 "/builds/moz2_slave/release-mozilla-1.9.1-macosx_build/build/browser/app/profile/channel-prefs.js"
---
 > //@line 2 "/builds/slave/release-mozilla-1.9.2-macosx_build/build/browser/app/profile/channel-prefs.js"
Only in source/Firefox.app/Contents: Plug-Ins
Contents of source/Firefox.app/Contents/Plug-Ins dir only in source or target
231980194        0 drwxr-xr-x    3 cltbld   staff         102 Nov 22 07:00 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin
231980195        0 drwxr-xr-x    4 cltbld   staff         136 Nov 23 10:06 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents
231980199        0 drwxr-xr-x    2 cltbld   staff          68 Nov 23 10:06 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/MacOS
231980200        0 drwxr-xr-x    3 cltbld   staff         102 Nov 22 07:00 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources
231980202        0 drwxr-xr-x    3 cltbld   staff         102 Nov 23 10:06 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj
231980206        0 drwxr-xr-x    2 cltbld   staff          68 Nov 23 10:06 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib

Windows: the usual .chk file differences:

Only in source/bin: .autoreg
diff -r source/bin/defaults/pref/channel-prefs.js target/bin/defaults/pref/channel-prefs.js
1c1
< //@line 2 "e:\builds\moz2_slave\release-mozilla-1.9.1-win32_build\build\browser\app\profile\channel-prefs.js"
---
> //@line 2 "e:\builds\moz2_slave\release-mozilla-1.9.2-win32_build\build\browser\app\profile\channel-prefs.js"
Files source/bin/freebl3.chk and target/bin/freebl3.chk differ
Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
Only in source/bin: removed-files
Files source/bin/softokn3.chk and target/bin/softokn3.chk differ

Publish updates to beta channel

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20101122-Firefox-3.5.16-beta
 real    36m39.797s
 user    0m1.617s
 sys     0m16.608s
time ~/bin/pushsnip 20101122-Firefox-3.5.16-beta
 real    11m27.557s
 user    0m0.711s
 sys     0m12.860s

XULRunner

Builds

Linux: No problems.
Windows: No problems.
Mac: No problems.

Signing

Signed with:

PRODUCT=xulrunner
VERSION=1.9.1.16
BUILD=1
TAG=FIREFOX_3_5_16_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=coop@mozilla.com
KEYDIR=d:/2010-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Build 2

Clobbered all builders labelled release-mozilla-1.9.1-* via clobberer
Landed updated automation config
Manually tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_3_5_16_RELEASE and FIREFOX_3_5_16_BUILD2
Updated configs and buildbotcustom
ran the following command to kick-off the release:
release_sanity.py dry run

PYTHONPATH=/tools/buildbotcustom:/builds/buildbot/builder_master1/tools/lib/python:. python release_sanity.py -d -u coop -B mozilla-1.9.1 -V 3.5.16 -N 2 localhost:9010

kick off build 2:

PYTHONPATH=/tools/buildbotcustom:/builds/buildbot/builder_master1/tools/lib/python:. python release_sanity.py -u coop -B mozilla-1.9.1 -V 3.5.16 -N 2 localhost:9010

Tag

No problems.

Source

No problems.

Build/Repack

Linux en-US: No problems
Linux repacks: some repacks failed in hg update:

abort: unknown revision '26016fb6587015a1e0d8017ced5f43b3977ada5b'!

- rebuilds went fine
Mac en-US: No problems
Mac repacks: same as for Linux, some repacks failed in hg update
Win32 en-US: No problems
Win32 repacks: same as for Linux, some repacks failed in hg update

Sign

Removed en-US.xpi from candidates directory

PRODUCT=firefox
VERSION=3.5.16
BUILD=2
TAG=FIREFOX_3_5_16_RELEASE
REPO='releases/mozilla-1.9.1'
EMAIL=coop@mozilla.com
KEYDIR=d:/2010-keys
cd ~/hg-tools
hg pull && hg up -r ${TAG}
mkdir ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
make PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

And then to upload:

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

L10N Verify

same as for build 1, nothing unexpected

Updates

Update generation failed with the following error:

TypeError: getShippedLocales() takes at most 6 arguments (7 given)

bhearsum checked-in a fix in bug 615668, after which the update builder was forced via the web interface

Update Verify

Linux: No problems
Mac: No problems
Win32: No problems

Socorro product data updates

no need

Update Bouncer

no need

Major Update Generation (-> 3.6.13)

Used 'force build' to start the major_update builder.
first pass failed in create_buildN_snippets (should not have been running this step):

urllib2.HTTPError: HTTP Error 404: Not Found

- patch was missing in hg on default branch. bhearsum landed patch, and I triggered a rebuild
subsequent builds failed because version configs had already been bumped
- I commented out the version config bumps and the builder completed successfully.
Firefox 3.6.13 required a build 3, so we need to regenerate the major update once the updates were available
- no problems with the builder this time

Major Update Verification

nothing unexpected, same output as for Build 1

Publish updates to beta channel

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20101130-Firefox-3.5.16-beta
 real    27m44.909s
 user    0m1.718s
 sys     0m15.275s
time ~/bin/pushsnip 20101130-Firefox-3.5.16-beta
 real    11m27.557s
 user    0m0.711s
 sys     0m12.860s

ashughes reported that he was not receiving updates for 3.5.16build1 on Mac. bhearsum discovered that this was because the 3.5.16 Mac builds use "Darwin_x86-gcc3-u-ppc-i386" instead of "Darwin_Universal-gcc3" in the update URL. To workaround the issue for now, I created a symlink in the live snippet directory:

# cltbld@aus2-staging
cd /opt/aus2/incoming/3/Firefox/3.5.16
ln -s Darwin_Universal-gcc3 Darwin_x86-gcc3-u-ppc-i386

This is bug 583671.

Final checks before push

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.5.16-candidates/build2/
find . ! -user ffxbld # (anything in contrib/ is ok)
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

contrib/solaris_pkgadd and solaris_tarball were 755 (g+s), which is OK too.

Virus check

# cltbld@stage
clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.5.16-candidates/build2/
----------- SCAN SUMMARY -----------
Infected files: 0
Time: 3393.680 sec (56 m 33 s)

real    56m33.707s
user    0m0.001s
sys     0m0.002s

Push to mirrors

#ffxbld@stage
rsync -av --exclude="*reporter*" --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/mozilla.org/firefox/nightly/3.5.16-candidates/build2/ /pub/mozilla.org/firefox/releases/3.5.16/

# cltbld@stage
vim -n /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replaced 3.5.15 with 3.5.16

Final verify

A few 404s, but nothing serious.

Push updates to release channel

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20101130-Firefox-3.5.16
 real    23m52.552s
 user    0m1.576s
 sys     0m13.876s
time ~/bin/pushsnip 20101130-Firefox-3.5.16
 real    14m37.916s
 user    0m0.719s
 sys     0m12.444s

Release

# ffxbld@stage
cd /home/ftp/pub/firefox/releases/
rm latest-3.5 && ln -s 3.5.16 latest-3.5

Sent the following to metrics@mozilla.com

Firefox,firefox,firefox,3.5.16,3.5
Firefox,firefox,firefox,3.5.17pre,3.5

Push 3.5.16 -> 3.6.13 MU

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20101203-Firefox-3.5.16-3.6.13-MU
 real    2m21.475s
 user    0m0.103s
 sys     0m1.071s
~/bin/pushsnip 20101203-Firefox-3.5.16-3.6.13-MU
 real    0m34.206s
 user    0m0.101s
 sys     0m1.064s

coop: I had a typo in the date I used for my initial MU push to beta and I didn't notice :(, so lsblakk had to re-push again later on request in IRC from ashughes/LegNeato.

~/bin/backupsnip 20101203-Firefox-3.5.16-3.6.13-MU-beta
 real	0m42.563s
 user	0m0.100s
 sys	0m1.015s
~/bin/pushsnip 20101203-Firefox-3.5.16-3.6.13-MU-beta
 real	0m23.569s 
 user	0m0.031s
 sys	0m0.574s

FIREFOX DONE TO HERE

XULRunner

Builds

Linux: No problems.
Windows: No problems.
Mac: No problems.

Signing

Signed with:

PRODUCT=xulrunner
VERSION=1.9.1.16
BUILD=2
TAG=FIREFOX_3_5_16_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=coop@mozilla.com
KEYDIR=d:/2010-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Push XULRunner to mirrors

xrbld@stage

rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*symbols.zip \
/pub/mozilla.org/xulrunner/nightly/1.9.1.16-candidates/build2/  \
/pub/mozilla.org/xulrunner/releases/1.9.1.16/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added

- xulrunner/releases/1.9.1.15

...to exclude the previous release.

Update XULRunner wiki

Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!

XR DONE TO HERE

Build #	Tag	Changeset
1	FIREFOX_3_5_16_RELEASE FIREFOX_3_5_16_BUILD1	cf3d01c081fd
2	FIREFOX_3_5_16_RELEASE FIREFOX_3_5_16_BUILD2	612ad84a4004

Releases/Firefox 3.5.16/BuildNotes

Contents

Notes About Releasing

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Notes

Build 1

Tag

Source

Build/Repack

Sign

L10N Verify

Updates

Update Verify

Socorro product data updates

Update Bouncer

Major Update Generation (-> 3.6.13)

Major Update Verification

Publish updates to beta channel

XULRunner

Builds

Signing

Build 2

Tag

Source

Build/Repack

Sign

L10N Verify

Updates

Update Verify

Socorro product data updates

Update Bouncer

Major Update Generation (-> 3.6.13)

Major Update Verification

Publish updates to beta channel

Final checks before push

Virus check

Push to mirrors

Final verify

Push updates to release channel

Release

Push 3.5.16 -> 3.6.13 MU

XULRunner

Builds

Signing

Push XULRunner to mirrors

Update XULRunner wiki

Navigation menu

Search