Releases/Firefox 3.5.19/BuildNotes

From MozillaWiki
Jump to: navigation, search

Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

coop - bug 649514

Notifications

  • Notify AV vendors ASAP
  • Notify metrics once we release

Bugs hit

  • starting autosign failed the first time because RELCONFIG was not defined
  • Notified AV vendors manually due to issues described in https://bugzilla.mozilla.org/show_bug.cgi?id=568884#c25
  • whimboo reported seeing a recurrence of bug 628829, but only on the test channels.
    • Armen followed the example here to fix snippets for some locales for both 3.6.17 and 3.5.19.
  • hg server went down: bug 651780. This caused the updates step for build 2 to fail.
  • NB: AUS config bump didn't land before beta, so beta users weren't getting updated to 3.5.19
    • bug 653676 - Forgot to unthrottle the current version and throttle 4.0.1. 3.5.18 people are not getting 3.5.19 because it is still throttled. Anyone who does get 3.5.19 is getting prompted with a 4.0.1 major update offer because it is not throttled.
    • This happened for 3.6.17 too. Sections for updating AUS have been EXPLICITLY ADDED to both sets of build notes.
  • due to copy-n-paste from build1 notes -> build2 notes, XR builds for build2 weren't actually signed before being rsync-ed for release. Had to be signed later and then rsync-ed again: bug 654103
    • avoid this by not copying build notes between builds

Signed-off Revision(s)

Build 1: 1c37949defbe

L10N changesets

Tags

Build # Tag Changeset
1 FIREFOX_3_5_19_BUILD1 GECKO19119_2011041408_RELBRANCH 5341366b7692
2 FIREFOX_3_5_19_RELEASE FIREFOX_3_5_19_BUILD2 GECKO19119_2011041408_RELBRANCH 6a8fcd90b366

Build data

Build # Type Build ID Build machine Time to build
1 Linux 20110414083440 linux-ix-slave14 32 mins, 35 secs
Mac 20110414083440 bm-xserve15 1 hrs, 56 mins, 3 secs
Windows 20110414083440 mw32-ix-slave02 1 hrs, 19 mins, 22 secs
2 Linux 20110420144310 mv-moz2-linux-ix-slave02 30 mins, 5 secs
Mac 20110420144310 bm-xserve11 1 hrs, 47 mins, 13 secs
Windows 20110420144310 mw32-ix-slave06 1 hrs, 20 mins, 26 secs

Notes

Done on pm01

Build 1

  • set release clobberer for mozilla-1.9.1, release-mozilla-1.9.1-post_signing, and release-mozilla-1.9.1-major_update on pm01
  • set reserved slaves to 6 on pm01
  • Landed updated automation config
  • Manually tagged buildbot-configs, buildbotcustom, tools with FIREFOX_3_5_19_RELEASE FIREFOX_3_5_19_BUILD1
  • Updated buildbot-configs/buildbotcustom/tools on pm01
  • Ran release_sanity.py w/ the dry run option:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u coop -V 3.5.19 --branch mozilla-1.9.1 --build-number 1 -c release-firefox-mozilla-1.9.1.py --dryrun localhost:9010
  • After it passed, ran it without --dry-run to start the release process:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u coop -V 3.5.19 --branch mozilla-1.9.1 --build-number 1 -c release-firefox-mozilla-1.9.1.py  localhost:9010

Tag

No problems

Source

No problems

Build

  • Linux: no problems
  • Win32: no problems
  • Mac: no problems

L10n Repack

  • Linux: no problems
  • Win32:
    • repack builder 2/6 timed out when calling clobberer (1200s). Triggered a rebuild from the waterfall.
  • Mac: no problems

Sign

Set up signing for 3.5.19. It was done concurrently with 4.0.1(and later, 3.6.17) by using two RDP sessions, one for each autosign instance.

Removed en-US.xpi from candidates directory. 

PRODUCT=firefox
VERSION=3.5.19
BUILD=1
TAG=FIREFOX_3_5_19_RELEASE
REPO='releases/mozilla-1.9.1'
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys
RELCONFIG=mozilla/release-firefox-mozilla-1.9.1.py
cd ~/hg-tools
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1

Signature verification succeeded.

L10n Verify

All platforms failed 'l10n metadiff' with what looks to be search engine ordering changes e.g. 

diff -r firefox-3.5.19-build1/diffs/linux-i686.af.diff firefox-3.5.18-build1/diffs/linux-i686.af.diff
12669c12669
< 1,49d0
---
> 1,47d0
12679,12680c12679,12680
< < browser.contentHandlers.types.0.title=Bloglines
< < browser.contentHandlers.types.0.uri=http://www.bloglines.com/login?r=/sub/%s
---
> < browser.contentHandlers.types.0.title=Google
> < browser.contentHandlers.types.0.uri=http://fusion.google.com/add?feedurl=%s
12683,12684d12682
< < browser.contentHandlers.types.2.title=Google
< < browser.contentHandlers.types.2.uri=http://fusion.google.com/add?feedurl=%s

Updates

No problems

Update Verify

whimboo reported seeing a recurrence of bug 628829, but only on the test channels.

Armen followed the example here to fix snippets for some locales for both 3.6.17 and 3.5.19.

Socorro product data updates

crash-stat admin: added Firefox 3.5.19 and Firefox 3.5.20pre .

Update Bouncer

Added by automation, no problems.

Major Update Generation

Needed to force a rebuild because the MU config was still pointing at 4.0rc2. Commented the following 2 lines in process/factory.py, reconfigured and rebuilt: 

#self.bumpPatcherConfig()
#self.bumpVerifyConfigs()

Local changes discarded, the master reconfigured after MU updates builder was triggered.

Major Update Verification

Failed for the following dropped locales: as, ka, mn, oc. Expected.

Unreserve slaves

Reset reserved slaves to 0

Publish updates to beta channel

cd /opt/aus2/snippets/staging/
time ~/bin/backupsnip Firefox-3.5.19-build1-beta
 real	56m38.988s
 user	0m7.943s
 sys	0m40.626s
time ~/bin/pushsnip Firefox-3.5.19-build1-beta
 real	29m48.032s
 user	0m9.151s
 sys	0m48.627s

Add beta to AUS throttling exceptions

CVS diff. Done after pushing to beta channel.

Build 2

  • set release clobberer for mozilla-1.9.1, release-mozilla-1.9.1-post_signing, and release-mozilla-1.9.1-major_update on pm01
  • set reserved slaves to 6 on pm01
  • Landed updated automation config
  • Manually tagged buildbot-configs, buildbotcustom, tools with FIREFOX_3_5_19_RELEASE FIREFOX_3_5_19_BUILD2
  • Updated buildbot-configs/buildbotcustom/tools on pm01
  • Ran release_sanity.py w/ the dry run option:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u coop -V 3.5.19 --branch mozilla-1.9.1 --build-number 2 -c release-firefox-mozilla-1.9.1.py --dryrun localhost:9010
  • After it passed, ran it without --dry-run to start the release process:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u coop -V 3.5.19 --branch mozilla-1.9.1 --build-number 2 -c release-firefox-mozilla-1.9.1.py  localhost:9010

Tag

No problems

Source

No problems

Build

  • Linux: no problems
  • Win32: no problems
  • Mac: no problems

L10n Repack

  • Linux: no problems
  • Win32:
    • repack builder 2/6 timed out when calling clobberer (1200s). Triggered a rebuild from the waterfall.
  • Mac: no problems

Sign

Set up signing for 3.5.19. It was done concurrently with 3.6.17 build3 by using two RDP sessions, one for each autosign instance. 

PRODUCT=firefox
VERSION=3.5.19
BUILD=2
TAG=FIREFOX_3_5_19_RELEASE
REPO='releases/mozilla-1.9.1'
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys
RELCONFIG=mozilla/release-firefox-mozilla-1.9.1.py
cd ~/hg-tools
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1

Signature verification succeeded.

L10n Verify

Same "failures" as Build 1.

Updates

  • 1st attempt: hg server went down, causing step to fail: bug 651780
  • 2nd attempt: failed to find snippets for kn, ml, and te for 3.5.18: bug 629256, bug 651802

Update Verify

  • Linux:
    • transient snippet errors:
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.5/20090615155223/Linux_x86-gcc3/fi/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5b4/20090423192432/Linux_x86-gcc3/es-MX/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
  • Mac:
    • kn, ml, te:
Binary files source/Firefox.app/Contents/MacOS/plugins/DefaultPlugin.plugin/Contents/MacOS/DefaultPlugin and target/Firefox.app/Contents/MacOS/plugins/DefaultPlugin.plugin/Contents/MacOS/DefaultPlugin differ
Binary files source/Firefox.app/Contents/MacOS/updater.app/Contents/MacOS/updater and target/Firefox.app/Contents/MacOS/updater.app/Contents/MacOS/updater differ
FAIL: binary files found in diff
FAIL: check_updates returned failure for Darwin_Universal-gcc3 downloads/Firefox 3.5.18.dmg vs. downloads/Firefox 3.5.19.dmg: 1
    • transient snippet errors:
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5b4/20090423191946/Darwin_Universal-gcc3/de/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
  • Win:
    • transient snippet errors:
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.1b1/20081007144708/WINNT_x86-msvc/hu/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

Major Update Generation

No problems

Major Update Verification

  • No snippets for as, ka, mn, oc on betatest

Unreserve slaves

Reset reserved slaves to 0

Going to beta day!

Do this in the morning before the rel-driver gives you the GO. 

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
time ~/bin/backupsnip Firefox-3.5.19-build2-beta

Push to Beta Channel

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of PHX datacenter
cd /opt/aus2/snippets/staging/
time ~/bin/pushsnip Firefox-3.5.19-build2-beta
 real	32m30.777s
 user	0m9.713s
 sys	0m49.973s

Email mirrors about release day date

joduinn sent an email a day before for all versions.

Final checks before push

Virus check

re-built AV scan, no problems

Pre-push Checks

re-ran check permissions builder, no problems

Push to mirrors

Forced push_to_mirrors builder with the following properties set: 

release_config: mozilla/release-firefox-mozilla-1.9.1.py
script_repo_revision: FIREFOX_3_5_19_RELEASE

No problems.

Final verify

Was triggered automatically when there is enough uptake.

AUS Throttling

See bug 653676 for the details and patch needed to unthrottle 3.5.18 -> 3.5.19, and throttle 3.5.19 -> 4.0.1.

Push updates to release channel

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of slow I/O in MPT
cd /opt/aus2/snippets/staging/
~/bin/backupsnip Firefox-3.5.19-build2

time ~/bin/pushsnip   Firefox-3.5.19-build2
real    26m7.217s
user    0m8.129s
sys     0m42.729s

time ~/bin/backupsnip Firefox-3.5.19-build2-4.0.1-build1-MU
real    1m41.181s
user    0m0.266s
sys     0m1.517s

time ~/bin/pushsnip Firefox-3.5.19-build2-4.0.1-build1-MU
real    0m28.118s
user    0m0.163s
sys     0m0.979s

Release

# ffxbld@stage
cd /home/ftp/pub/firefox/releases/
rm latest-3.5 && ln -s 3.5.19 latest-3.5

Sent the following to metrics@mozilla.com 

Firefox,firefox,firefox,3.5.19,3.5
Firefox,firefox,firefox,3.5.20pre,3.5

FIREFOX DONE TO HERE

XULRunner

XULRunner Source & Build

  • Source: no problems
  • Linux: no problems
  • Win32: no problems
  • Mac: no problems

Signing

Signed with: 

PRODUCT=xulrunner
VERSION=1.9.1.19
BUILD=2
TAG=FIREFOX_3_5_19_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=coop@mozilla.com
KEYDIR=d:/2010-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Push XULRunner to mirrors

  1. xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*symbols.zip \
/pub/mozilla.org/xulrunner/nightly/1.9.1.19-candidates/build2/  \
/pub/mozilla.org/xulrunner/releases/1.9.1.19/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added 

- xulrunner/releases/1.9.1.18

...to exclude the previous release.

Update XULRunner wiki

Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!

XULRUNNER DONE TO HERE

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_3.5.19/BuildNotes&oldid=311612"