Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

Signed-off Revision(s)

L10N changesets

Same as 3.5.12

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20100914121211	mv-moz2-linux-ix-slave15	34 mins, 27 secs
	Mac	20100914122142	bm-xserve22	1 hrs, 46 mins, 6 secs
	Windows	20100914130356	mw32-ix-slave20	1 hrs, 30 mins, 3 secs

Notes

Using pm02.

Build 1

Used clobberer to clean up all previous releases
Landed automation version bumps. Tagged tip of buildbot-configs, buildbotcustom, and tools with FIREFOX_3_5_13_RELEASE and FIREFOX_3_5_13_BUILD1
updated master checkouts, removed *.pyc from master dir, reconfigured production-master02
closed Firefox3.5 tree
Kicked off automation

buildbot sendchange --username=catlee --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.5.13 build1" Go

Tag

No problems. Landed version bump and reopened mozilla-1.9.1.

Source / XULRunner source

No problems

Build/Repack

Linux all green
Mac all green
Windows. eo and mr failed. Rebuilt.

python force_release_l10n.py -m http://localhost:8010 -t FIREFOX_3_5_13_RELEASE -v -s my-locales -n rail
Forcing http://localhost:8010/builders/win32_repack/force with params: {'username': 'rail', 'prop3value': 'FIREFOX_3_5_13_RELEASE', 'prop2value': 'eo', 'comments': 'Unknown', 'prop2name': 'locale', 'prop1value': 'FIREFOX_3_5_13_RELEASE', 'prop3name': 'l10n_revision', 'branch': 'eo', 'prop1name': 'en_revision'}
Forcing http://localhost:8010/builders/win32_repack/force with params: {'username': 'rail', 'prop3value': 'FIREFOX_3_5_13_RELEASE', 'prop2value': 'mr', 'comments': 'Unknown', 'prop2name': 'locale', 'prop1value': 'FIREFOX_3_5_13_RELEASE', 'prop3name': 'l10n_revision', 'branch': 'mr', 'prop1name': 'en_revision'}
[cltbld@production-master02 buildbot-helpers]$ cat my-locales 
eo win32
mr win32

Socorro product data updates

Asked joduinn to add 3.5.13 and 3.5.14pre using the following steps:

Use crash-stats admin page
"Branch Data Sources"
If not present in the 'Missing Entries' table (no crashes yet), use the 'add new product version' link
Added Firefox 3.5.13 and 3.5.14pre

Signing

Instructions

en-US.xpi removed from stage.m.o:

[ffxbld@surf build1]$ rm /pub/mozilla.org/firefox/nightly/3.5.13-candidates/build1/en-US.xpi

Script used for signing

#!/bin/bash
set -x
set -e

PRODUCT=firefox
VERSION=3.5.13
BUILD=1
TAG=FIREFOX_3_5_13_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=rail@mozilla.com
KEYDIR=d:/2009-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}
cd -

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
  KEYDIR=${KEYDIR} USE_NEW=1

Got "Signature verification succeeded" email. Reviewed stdout, win32_signing_build1.log.

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Did XULRunner Signing immediately afterwards.

L10N Verify

No problems

Generate updates and push betatest snippets

No problems with automation.

Recreate beta snippets to use bouncer (reducing QA load)

# cltbld@aus-staging
cd /opt/aus2/snippets/staging
rsync -a 20100914-Firefox-3.5.13/ 20100914-Firefox-3.5.13-bouncer-beta/
cd 20100914-Firefox-3.5.13-bouncer-beta/
find . -maxdepth 5 -mindepth 5 -type d -exec mv -v {}/release {}/beta \;
find . | grep release

Update Verify

All green
Linux and Mac logs don't contain "^Only in"
Windows log contains some expected entries:

$ grep "^Only in" stdio.html|sort -u
Only in source/bin: .autoreg
Only in source/bin: removed-files

$ grep -E "^Files .* differ$" stdio.html|sort -u
Files source/bin/freebl3.chk and target/bin/freebl3.chk differ
Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
Files source/bin/softokn3.chk and target/bin/softokn3.chk differ

Major Update Generation (-> 3.6.10build1)

Hit Force Build on major update builder
The first build failed due to connectivity
The second one failed due to not clobbered build directory
The third run generated MU, but turned red due to already bumped patcher config. bug 596568 with a patch filed. Red status can be ignored for this run.

Verification

All red due to:

mn locale was removed in 3.6.x:

grep ^FAIL: *_stdio.html
# all output is mn releated

The following entries are expected and can be ignored:

$ grep "^Only in" *_stdio.html |sort -u     
linux_stdio.html:Only in source/firefox/searchplugins: iskalnik-gov-si.xml
linux_stdio.html:Only in source/firefox/searchplugins: najdi-si.xml
linux_stdio.html:Only in source/firefox/searchplugins: yahoo.xml
mac_stdio.html:Only in source/Firefox.app/Contents/MacOS/searchplugins: iskalnik-gov-si.xml
mac_stdio.html:Only in source/Firefox.app/Contents/MacOS/searchplugins: najdi-si.xml
mac_stdio.html:Only in source/Firefox.app/Contents/MacOS/searchplugins: yahoo.xml
mac_stdio.html:Only in source/Firefox.app/Contents: Plug-Ins
win32_stdio.html:Only in source/bin: .autoreg
win32_stdio.html:Only in source/bin: removed-files
win32_stdio.html:Only in source/bin/searchplugins: iskalnik-gov-si.xml
win32_stdio.html:Only in source/bin/searchplugins: najdi-si.xml
win32_stdio.html:Only in source/bin/searchplugins: yahoo.xml

$ grep -E "^File.*differ" *_stdio.html |sort -u
win32_stdio.html:Files source/bin/freebl3.chk and target/bin/freebl3.chk differ
win32_stdio.html:Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
win32_stdio.html:Files source/bin/softokn3.chk and target/bin/softokn3.chk differ

Update Bouncer

Forced the builder from the web interface. The first run failed, because BuildSlaves.py on pm02 didn't contain Tuxedo credentials.
Added Tuxedo credentials to BuildSlaves.py on pm02 and reran the builder.

Final checks before push

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.5.13-candidates/build1/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

Virus check

# cltbld@stage
clamdscan -r /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.5.13-candidates/build1
......
----------- SCAN SUMMARY -----------
Infected files: 150
Time: 3055.695 sec (50 m 55 s)

Found Trojan.Generic.Bredolab-2, which also seen in in 4.0b6 (false positive).

Ben filed bug 596671. The second run passed.

Push to mirrors

# ffxbld@stage 
rsync -av --exclude="*reporter*" --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/mozilla.org/firefox/nightly/3.5.13-candidates/build1/ /pub/mozilla.org/firefox/releases/3.5.13/

# cltbld@stage
vim -n /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.5.12 with 3.5.13

Final verify

Green

Push updates to release channel

Filed bug 596686 for throttling changes.

# cltbld@aus2-staging
~/bin/backupsnip 20100914-Firefox-3.5.13
~/bin/backupsnip 20100914-Firefox-3.5.13-3.6.10-MU

~/bin/pushsnip 20100914-Firefox-3.5.13
~/bin/pushsnip 20100914-Firefox-3.5.13-bouncer-beta

We'll do the MU a later day.

Release

#ffxbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.5 && ln -s 3.5.13 latest-3.5

Sent the following mail to metrics@mozilla.com:

Firefox,firefox,firefox,3.5.13,3.5
Firefox,firefox,firefox,3.5.14pre,3.5

Push 3.5.13 -> 3.6.10 MU (beta+release)

As cltbld@aus2-staging:

~/bin/backupsnip 20100914-Firefox-3.5.13-3.6.10-MU
~/bin/pushsnip 20100914-Firefox-3.5.13-3.6.10-MU

After QA finished testing the release channel:

~/bin/pushsnip 20100914-Firefox-3.5.13-3.6.10-MU-beta

XULRunner

No problems.

Signing

Used the following script:

#!/bin/bash
set -x
set -e

PRODUCT=xulrunner
VERSION=1.9.1.13
BUILD=1
TAG=FIREFOX_3_5_13_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=rail@mozilla.com
KEYDIR=d:/2009-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}
cd -

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Push XULRunner to mirrors

# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip \
  /pub/mozilla.org/xulrunner/nightly/1.9.1.13-candidates/build1/ \
  /pub/mozilla.org/xulrunner/releases/1.9.1.13/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added

- xulrunner/releases/1.9.1.12

Update XULRunner wiki

Updated the links in these two wiki pages for the version change:

Build #	Tag	Changeset
1	GECKO19112_20100824_RELBRANCH	9cabb92310de
1	FIREFOX_3_5_13_BUILD1 FIREFOX_3_5_13_RELEASE	bea84ea1dadd

Releases/Firefox 3.5.13/BuildNotes

Contents

Notes About Releasing

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Notes

Build 1

Tag

Source / XULRunner source

Build/Repack

Socorro product data updates

Signing

L10N Verify

Generate updates and push betatest snippets

Update Verify

Major Update Generation (-> 3.6.10build1)

Verification

Update Bouncer

Final checks before push

Virus check

Push to mirrors

Final verify

Push updates to release channel

Release

Push 3.5.13 -> 3.6.10 MU (beta+release)

XULRunner

Signing

Push XULRunner to mirrors

Update XULRunner wiki

Navigation menu

Search