Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

Signed-off Revision(s)

L10N changesets

Signed off on the shipped-locales and l10n-changesets changes in the l10n shipping milestones dashboard.

NOTE: If the "Ship It" button is greyed out for you even after logging in (with LDAP), Axel will have to grant you access in the db. bug 566468 is on file for having this access happen automatically.

l10n-changesets update landed as part of revision 56d65e04955d to buildbot-configs.

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20100930122135	mv-moz2-linux-ix-slave09	29 mins, 53 secs
	Mac	20100930123633	bm-xserve22	1 hrs, 44 mins
	Windows	20100930131606	mw32-ix-slave20	1 hrs, 28 mins

Notes

using pm02

Build 1

Used clobberer to clean up all previous releases from pm02:8010
Landed automation bumps
Already had slaves attached
Updated config-default checkout, also checked /tools/buildbotcustom/buildbotcustom (no changes in hg in)
removed *.pyc
notified in #developers that tree was closing for tagging
tagged tip of buildbot-configs, buildbotcustom, tools with FIREFOX_3_5_14_RELEASE, FIREFOX_3_5_14_BUILD1
Kicked off automation

buildbot sendchange --username=lsblakk --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.14 build1" Go

Automation halted due to missing reconfig on pm02

clobbered pm02:8010 again
reconfig'd moz2-master
Kicked off automation again:

buildbot sendchange --username=lsblakk --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.14 build1" Go

Tag

Since the first run through re-used the 3.5.13 configs, bhearsum had to go fix the relbranch tag for GECKO19112_20100824_RELBRANCH and the automation had to be stopped immediately (no builds even finished compile so damage was minimal). L10n changesets/tags were fine. Bhearsum did the tag fixes because I don't have permission on releases/mozilla-1.9.1

On the second tagging run (after reconfig) there were no problems.

Bhearsum landed version bump for me

Source / XULRunner source

No problems

Build/Repack

Linux, Mac, Win32 - no problems on build or repack

Socorro product data updates

Use crash-stats admin page
"Branch Data Sources"
If not present in the 'Missing Entries' table (no crashes yet), use the 'add new product version' link
Added Firefox 3.5.14 and 3.5.15pre

Signing

Instructions

Removed en-US.xpi from stage.

[ffxbld@surf build1]$ rm /pub/mozilla.org/firefox/nightly/3.5.14-candidates/build1/en-US.xpi

#!/bin/bash
set -x
set -e

PRODUCT=firefox
VERSION=3.5.14
BUILD=1
TAG=FIREFOX_3_5_14_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=lsblakk@mozilla.com
KEYDIR=d:/2009-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}
cd -

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
  KEYDIR=${KEYDIR} USE_NEW=1

Got "Signing of firefox 3.5.14 done, ready for inspection" email. Reviewed stdout, win32_signing_build1.log.

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Did XULRunner Signing immediately afterwards.

L10N Verify

No problems

Generate updates and push betatest snippets

No problems

Update Verify

No problems.

At this point a respin for build2 was called because of bug 601108

Build 2

Used clobberer to clean up all previous releases from pm02:8010
Landed automation bumps for build2
Already had slaves attached
Updated config-default checkout, also checked /tools/buildbotcustom/buildbotcustom (no changes in hg in)
removed *.pyc
notified in #developers that tree was closing for tagging
reconfig pm02
tagged tip of buildbot-configs, buildbotcustom, tools with FIREFOX_3_5_14_RELEASE, FIREFOX_3_5_14_BUILD2
Kicked off automation

buildbot sendchange --username=lsblakk --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.14 build2" Go

Tag

First run of tagging failed out early because of a space in the relbranch_override in configs. Took out the space, landed, clobbered again, updated configs-default on pm02, reconfig, kicked off automation again. I also re-tagged the buildbot-configs with FIREFOX_3_5_14_RELEASE and FIREFOX_3_5_14_BUILD2 for the updated config.

Second run on tagging completed without problems.

Source / XULRunner source

Source no problems Linux build no problems

Build/Repack

Linux build/repack no problem At this point it was discovered by QA that the UA string said "3.5.14pre" and so the sourceRepoRevision had not been updated in build2 configs to include the version bumps -- respin needed

Build 3

Used clobberer to clean up all previous releases from pm02:8010
Landed automation bumps for build2
Already had slaves attached
Updated config-default checkout, also checked /tools/buildbotcustom/buildbotcustom (no changes in hg in)
removed *.pyc
notified in #developers that tree was closing for tagging
reconfig pm02
tagged tip of buildbot-configs, buildbotcustom, tools with FIREFOX_3_5_14_RELEASE, FIREFOX_3_5_14_BUILD3
Kicked off automation

buildbot sendchange --username=lsblakk --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.14 build3" Go

Tag

No problems

Source / XULRunner source

No problems

Build/Repack

No problems on linux,mac,win32 for build or repack steps.

Socorro product data updates

done in build 1

Signing

Instructions

Removed en-US.xpi from stage.

[ffxbld@surf build3]$ rm /pub/mozilla.org/firefox/nightly/3.5.14-candidates/build3/en-US.xpi

#!/bin/bash
set -x
set -e

PRODUCT=firefox
VERSION=3.5.14
BUILD=3
TAG=FIREFOX_3_5_14_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=lsblakk@mozilla.com
KEYDIR=d:/2009-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}
cd -

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
  KEYDIR=${KEYDIR} USE_NEW=1

Got "Signing of firefox 3.5.14 done, ready for inspection" email. Reviewed stdout, win32_signing_build1.log.

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Did XULRunner Signing immediately afterwards.

L10N Verify

Linux, Win32 no problems. Mac died out in downloading binary, force rebuild ran successfully second time through.

Generate updates and push betatest snippets

Had to run 3 times to get updates generated properly. On the first try it died downloading a .mar (187/450). On the second attempt it failed when trying to create an update for that .mar because of cache holding a 0 byte sized file for that .mar. I cleaned out /builds/slave/updates/build/temp/firefox/ftp/* to clear cache and also /builds/slace/updates/build/mozilla to re-do the updates step one more time. On the next attempt it was successful.

Update Verify

Linux: FAIL (ro partial.mar only) with a strange bz2.1 issue:

Using  https://aus2.mozilla.org/update/1/Firefox/3.5.13/20100914121211/Linux_x86-gcc3/ro/betatest/update.xml?force=1
14:45:58 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.5.14-candidates/build3/update/linux-i686/ro/firefox- 3.5.13-3.5.14.partial.mar [714985/714985] -> "update/partial.mar" [1]
15:01:00 URL:http://stage-old.mozilla.org/pub/mozilla.org//firefox/releases/3.5.13/linux-i686/ro/firefox-3.5.13.tar.bz2 [10263849/10263849] -> "firefox-3.5.13.tar.bz2.1" [2]
15:01:02 URL:http://stage-old.mozilla.org/pub/mozilla.org//firefox/nightly/3.5.14-candidates/build3/linux-i686/ro/firefox-3.5.14.tar.bz2 [10276405/10276405] -> "firefox-3.5.14.tar.bz2" [1]
tar: Unexpected EOF in archive
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-3.5.13.tar.bz2 vs. downloads/firefox-3.5.14.tar.bz2: 1

Mac: PASS Win32: PASS

Major Update Generation (-> 3.6.11build2)

hit force build on waterfall
first run failed because of Mercurial upgrade which took away the ability to fallback to a default username and bug 601996 fixed that
second run failed in download step, had couple of false starts because the configs had already been bumped. I backed out 20fb3b32e1cc changeset from tools, clobbered major_update/* on the slave and restarted
fifth run - failed on createNsnippets - bug 602037 filed, patch landed, updated buildbotcustom on pm02, clobbered major_update on the slave, backed out the config bump in hg.m.o/build/tools and started again.
sixth run generated MU, but turned red due to already bumped patcher config. bug 596568 is already filed from 3.5.13. Red status can be ignored for this run.

Verification

All red due to:

mn locale was removed in 3.6.x:

grep ^FAIL: *_stdio.html
# all output is mn releated

The following entries are expected and can be ignored:

$ grep "^Only in" *_stdio.html |sort -u     
linux_stdio.html:Only in source/firefox/searchplugins: iskalnik-gov-si.xml
linux_stdio.html:Only in source/firefox/searchplugins: najdi-si.xml
linux_stdio.html:Only in source/firefox/searchplugins: yahoo.xml
mac_stdio.html:Only in source/Firefox.app/Contents/MacOS/searchplugins: iskalnik-gov-si.xml
mac_stdio.html:Only in source/Firefox.app/Contents/MacOS/searchplugins: najdi-si.xml
mac_stdio.html:Only in source/Firefox.app/Contents/MacOS/searchplugins: yahoo.xml
mac_stdio.html:Only in source/Firefox.app/Contents: Plug-Ins
win32_stdio.html:Only in source/bin: .autoreg
win32_stdio.html:Only in source/bin: removed-files
win32_stdio.html:Only in source/bin/searchplugins: iskalnik-gov-si.xml
win32_stdio.html:Only in source/bin/searchplugins: najdi-si.xml
win32_stdio.html:Only in source/bin/searchplugins: yahoo.xml

$ grep -E "^File.*differ" *_stdio.html |sort -u
win32_stdio.html:Files source/bin/freebl3.chk and target/bin/freebl3.chk differ
win32_stdio.html:Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
win32_stdio.html:Files source/bin/softokn3.chk and target/bin/softokn3.chk differ

Major Update Generation (3.5.14 build3 -> 3.6.11build3)

landed config bump

for build3, updated configs-default on moz2-master, reconfig'd pm02

hit force build on waterfall MU builder

Verification

All three failed with same issues (known, and acceptable) as previous MU verification

Publish updates to beta channel

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging/
time ~/bin/backupsnip 20101004-Firefox-3.5.14-beta
time ~/bin/pushsnip 20101004-Firefox-3.5.14-beta

Update Bouncer

Triggered the bouncer_submitter on pm02 - ran sucessfully, bouncer updated.

Final checks before push

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.5.14-candidates/build3/
find . ! -user ffxbld # (anything in contrib/ is ok)
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

Virus check

# cltbld@stage
clamdscan -r /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.5.14-candidates/build3/
WARNING: Ignoring unsupported option --recursive (-r)
/mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.5.14-candidates/build3: OK
.....
----------- SCAN SUMMARY -----------
Infected files: 0
Time: 3829.341 sec (63 m 49 s)

Push to mirrors

rsync -av --exclude="*reporter*" --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/mozilla.org/firefox/nightly/3.5.14-candidates/build3/ /pub/mozilla.org/firefox/releases/3.5.14/

# cltbld@stage
vim -n /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.5.13 with 3.5.14

Final verify

Used "force build" to start the final_verification builder on pm02. Final verify failed because www.mirrorservice.org wasn't picking up the win32 builds and that happened to be the mirror sentry was hitting over and over. After discussion in IRC with justdave it was decided that www.mirrorservice.org would catch up eventually and this isn't a huge concern.

Push release channel snippets

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20101004-Firefox-3.5.14
~/bin/backupsnip 20101013-Firefox-3.5.14-3.6.11-MU

On "Go" in release-drivers email:

~/bin/pushsnip 20101004-Firefox-3.5.14

Since we waited on the MU due to late time on release day, the MU went out two days later:

~/bin/pushsnip 20101013-Firefox-3.5.14-3.6.11-MU

Release

# ffxbld@stage
cd /home/ftp/pub/firefox/releases/
rm latest-3.5 && ln -s 3.5.14 latest-3.5

Sent the following to metrics@mozilla.com

Firefox,firefox,firefox,3.5.14,3.5
Firefox,firefox,firefox,3.5.15pre,3.5

XULRunner

No problems

Signing

Used the following script

#!/bin/bash
set -x
set -e

PRODUCT=xulrunner
VERSION=1.9.1.14
BUILD=3 
TAG=FIREFOX_3_5_14_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=lsblakk@mozilla.com
KEYDIR=d:/2009-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}
cd -

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Push XULRunner to mirrors

xrbld@stage

rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*symbols.zip \
/pub/mozilla.org/xulrunner/nightly/1.9.1.14-candidates/build3/  \
/pub/mozilla.org/xulrunner/releases/1.9.1.14/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added

- xulrunner/releases/1.9.1.13
# you want to exclude the previous release, not the current as I initially did

Update XULRunner wiki

Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!

Ignore me - never completed - Build 4

Build4 happened because the release tag *still* wasn't on the right revision, and was only discovered very close to pushing to beta. The build2 tags happened on the default branch, but the build3 ones happened on the relbranch. For some unknown reason, Mercurial still believes that the release tag is the default branch one -- which is incorrect. To fix, I removed the _RELEASE tag from default and the relbranch manually before starting the automation. This was tested in my user repo, and worked fine. (- bhearsum). Other prep:

Used clobberer to clean up all previous releases from pm02:8010
Landed automation bumps for build4
No slave movement necessary prior to tagging
Updated /tools/buildbotcustom/buildbotcustom and /builds/buildbot/configs-default on pm02
Removed *.pyc in the master directory
Closed the tree
Reconfiged pm02
Started automation with:

buildbot sendchange --username=bhearsum --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.14 build4" Go

Tag

Manually pushed FIREFOX_3_5_14_BUILD4 and updated FIREFOX_3_5_14_RELEASE tag to buildbotcustom, buildbot-configs, and tools. While tagging ran, move 4 additional fast Windows machines and 4 additional xserves to pm02.

No problems with tagging. Verified the tags afterwards by looking at hgweb, noticed that dc338f6c00de was correctly tagged with _BUILD4 and _RELEASE.

Source

No problems

Build/Repack

Verified that builds got the right revision by noticing that got_revision was set to dc338f6c00de.

Linux: No problems
Mac: No problems
Win32: No problems

Moved 4 mac and 4 ix slaves back to pm01

Socorro product data updates

already done in build1

Signing

Instructions

Removed en-US.xpi from stage.

[ffxbld@surf build4]$ rm /pub/mozilla.org/firefox/nightly/3.5.14-candidates/build4/en-US.xpi

#!/bin/bash
set -x
set -e

PRODUCT=firefox
VERSION=3.5.14
BUILD=4
TAG=FIREFOX_3_5_14_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=lsblakk@mozilla.com
KEYDIR=d:/2009-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}
cd -

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
  KEYDIR=${KEYDIR} USE_NEW=1

Got "Signing of firefox 3.5.14 done, ready for inspection" email. Reviewed stdout, win32_signing_build1.log.

At this point it became apparent that build4 was the same as build3 and so we stopped build4 (verified by diffing build3 & build4 application.ini - only diff was build id)

Build #	Tag	Changeset
1	GECKO19114_20100930_RELBRANCH	d3c9cf653951
1	FIREFOX_3_5_14_BUILD1 FIREFOX_3_5_14_RELEASE	dc338f6c00de
2	FIREFOX_3_5_14_BUILD2 FIREFOX_3_5_14_RELEASE	06ad1b12aef9
3	FIREFOX_3_5_14_BUILD3 FIREFOX_3_5_14_RELEASE	dc338f6c00de
4	FIREFOX_3_5_14_BUILD4 FIREFOX_3_5_14_RELEASE	dc338f6c00de

Releases/Firefox 3.5.14/BuildNotes

Contents

Notes About Releasing

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Notes

Build 1

Tag

Source / XULRunner source

Build/Repack

Socorro product data updates

Signing

L10N Verify

Generate updates and push betatest snippets

Update Verify

Build 2

Tag

Source / XULRunner source

Build/Repack

Build 3

Tag

Source / XULRunner source

Build/Repack

Socorro product data updates

Signing

L10N Verify

Generate updates and push betatest snippets

Update Verify

Major Update Generation (-> 3.6.11build2)

Verification

Major Update Generation (3.5.14 build3 -> 3.6.11build3)

Verification

Publish updates to beta channel

Update Bouncer

Final checks before push

Virus check

Push to mirrors

Final verify

Push release channel snippets

Release

XULRunner

Signing

Push XULRunner to mirrors

Update XULRunner wiki

Ignore me - never completed - Build 4

Tag

Source

Build/Repack

Socorro product data updates

Signing

Navigation menu

Search