Releases/Firefox 3.6.18/BuildNotes

From MozillaWiki
Jump to: navigation, search

Contents

Build Engineers

rail - bug 663579

Signed-off Revision(s)

Tags

Build # Tag Changeset
1 FIREFOX_3_6_18_BUILD1 d020c6aeb86c
2 FIREFOX_3_6_18_BUILD2 cc94987d6b34

Notifications

  • We have to give mirrors as much notice as possible before pushing to mirrors. This has been 24-72 hours, but has been up to a week in advance. (for email address and template example, see bug 633657 )
  • We have to notify infra-all@ before "go to build" and before we ship. The more advance notice the better.
  • We have to notify metrics once we release

Bugs hit

NOTE: DON'T FORGET TO UNTHROTTLE BETA CHANNEL ON BETA DAY AND REMOVE IT ON THE RELEASE DAY ADDING THROTTLING FOR THE LAST VERSION ON THE RELEASE CHANNEL TO PREVENT UNPROMPTED MAJOR UPDATES!

Build data

Build # Build ID Type Build machine Time to build
2 20110613024313 Linux mv-moz2-linux-ix-slave18 46 mins, 25 secs
Mac bm-xserve16 1 hrs, 43 mins, 44 secs
Windows mw32-ix-slave07 1 hrs, 40 mins, 41 secs
2 20110614230723 Linux mv-moz2-linux-ix-slave06 44 mins, 42 secs
Mac bm-xserve16 1 hrs, 40 mins, 52 secs
Windows mw32-ix-slave07 1 hrs, 46 mins, 51 secs

Build 1

  • Clobbered "Any master/1.9.2/Any builder"
  • Commited buildbot-configsand merged to production branch
  • Tagged using FIREFOX_3_6_18_BUILD1 and FIREFOX_3_6_18_RELEASE: buildbotcustom (tip of production-0.8), tools (tip of default), buildbot-configs (tip of production)
  • Updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
  • Reserved 6 slaves:
echo 6 >reserved_slaves_pm03
  • Ran release sanity:
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python \
/builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py \
-u rail -V 3.6.18 --branch mozilla-1.9.2 --build-number 1 \
-c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010

which failed with the following message

2011-06-13 02:05:58,779 : ERROR : 	locale ast revisions not matching: 2eaad86320c8 (config) vs. 8ad733347436 (dashboard)
2011-06-13 02:05:58,779 : ERROR : 	locale ru revisions not matching: 8e74eb072189 (config) vs. 915987d5da2b (dashboard)
2011-06-13 02:05:58,780 : ERROR : Error verifying l10n dashboard changesets
  • Updated l10n-changesets_mozilla-1.9.2:
wget -O l10n-changesets_mozilla-1.9.2 https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx3.6.18
  • Commited to default, merged to production, retagged and pushed buildbot-configs.
  • Kicked off automation with:
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python \
/builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py \
-u rail -V 3.6.18 --branch mozilla-1.9.2 --build-number 1 \
-c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010

Metrics Email

Sent the following mail to: metrics@mozilla.com, cc:release@mozilla.com:

Firefox,firefox,firefox,3.6.18,3.6
Firefox,firefox,firefox,3.6.19pre,3.6

Tag

No problems

Start autosigning

Set up autosinging using instructions

Update Bouncer

No problems.

Source / XULRunner Source

No problems

Build

No problems.

L10n Repack

No problems.

XULRunner Build

No problems.

Unittests

Signing

Followed CombinedSigning doc.

L10n Verify

Nothing unusual in the logs.

Updates

3.6* -> 3.6.18

Generation

Quote from https://bugzilla.mozilla.org/show_bug.cgi?id=663579#c8: The update verify bumper removed the wrong lines, because there was no comment lines. Because of this, we lost the 3.6.15 quick checks, and didn't get 3.6.16 checks at all. This patch should fix it for all platforms.

Fixed by http://hg.mozilla.org/build/tools/rev/e6f43e646c8d

Update Verify

Verify configs were bumped wrong, because bug 408453 didn't add in the proper comments to them. They were first fixed by hand in this patch, and then subsequently, the Mac ones were fixed again in this patch.

2 of 10 mac update verify builders failed for ml and te locales (expected, these locales' updates were disabled in 3.6.17}:

FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110420140830/Darwin_Universal-gcc3/te/betatest/update.xml?force=1
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110420140830/Darwin_Universal-gcc3/ml/betatest/update.xml?force=1

3.6.x -> 4.0.1 Major Updates

MU Generation
  • Per bug 650933 we need to disable as, ka & oc. Landed MU config bump patch (CVS), landed MU update verify bump, retagged tools repo with FIREFOX_3_6_18_BUILD1_RUNTIME FIREFOX_3_6_18_RELEASE_RUNTIME.
  • Waited until Firefox 5b6 updates finish.
  • Reconfigured pm03 with locally applied patch
  • Hit "Force build" for 1.9.2 MU
MU Verify

No problems.

Updates (3.5.x -> 3.6.18)

Close to release day we found out that we needed to generate these updates. Didn't land a patcher config, because no one was around to review it.

Created patcher config.

Did work on mv-moz2-linux-ix-slave13.build.mozilla.org.

cd /builds/slave/rel-192-mu/build/patcher-configs
wget -O moz191-branch-major-update-patcher2.cfg --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=540644

perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz191-branch-major-update-patcher2.cfg

cd temp/firefox
ln -s 3.6.18 3.5.19-3.6.18
cd ../..

perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz191-branch-major-update-patcher2.cfg

cd temp/firefox/3.5.19-3.6.18
# check releasetest == release
bash /builds/slave/rel-192-mu/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest

# push up to aus
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.18-build2-3.5.19-build2-test
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.18-build2-3.5.19-build2-beta
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.18-build2-3.5.19-build2

bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip Firefox-3.6.18-build2-3.5.19-build2-test'

Had to fix a few things after the fact:

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
# details URL
find Firefox-3.6.18-build2-3.5.19-build2-test Firefox-3.6.18-build2-3.5.19-build2-beta Firefox-3.6.18-build2-3.5.19-build2 -type f -exec sed -i -e 's,/firefox/4.0/details/,/firefox/3.6/details/,' {} \;
# Build targets
for i in Firefox-3.6.18-build2-3.5.19-build2{-test,-beta,}; do for j in 3.5.{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}; do mv $i/Firefox/$j/{Darwin_x86-gcc3-u-ppc-i386,Darwin_Universal-gcc3}; done; done
for i in Firefox-3.6.18-build2-3.5.19-build2{-test,-beta,}; do for j in 3.5.{16,17,18,19}; do rsync -av $i/Firefox/$j/Darwin_{x86,ppc}-gcc3-u-ppc-i386/; done; done
Update Verify

Landed update verify configs, retagged with FIREFOX_3_5_19_{RELEASE,BUILD1}_RUNTIME. To start update verify:

for p in linux macosx win32; do
  for i in 1 2 3 4 5 6 7 8 9 10; do
    curl "http://production-master1.build.sjc1.mozilla.com:8010/builders/release-mozilla-1.9.1-${p}_major_update_verify_${i}%2F10/force" >/dev/null
  done
done

Socorro product data updates

crash-stat admin: added Firefox 3.6.18 and Firefox 3.6.19pre.

Going to beta day!

Push to Beta Channel

#From a build slave you can reach aus2-staging
# ssh cltbld@aus2-staging.mozilla.org

~/bin/backupsnip Firefox-3.6.18-build1-beta
~/bin/pushsnip Firefox-3.6.18-build1-beta

Unthrottle beta channel

Turns out we didn't need this, as 3.6.17 was already unthrottled. Re-throttled 3.6.18 on beta (below) so we could push the major update without prompting people.

Email mirrors about release day date

Build 2

Preparation

  • Clobbered "Any master/1.9.2/Any builder"
  • Clobbered MU directory on mv-moz2-linux-ix-slave10 (build1) manually
  • Commited buildbot-configs and merged to production branch
  • Tagged using FIREFOX_3_6_18_BUILD2 and FIREFOX_3_6_18_RELEASE: buildbotcustom (tip of production-0.8), tools (tip of default), buildbot-configs (tip of production)
  • Updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
  • Reserved 12 slaves (Firefox 5b7 was in process on the same master):
echo 12 >reserved_slaves_pm03
  • Kicked off automation with:
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python \
/builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py \
-u rail -V 3.6.18 --branch mozilla-1.9.2 --build-number 2 \
-c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010

Metrics Email

Not needed for build2

Tag

No problems.

Start autosigning

Set up autosinging using instructions

Update Bouncer

N/A for build2

Source / XULRunner Source

No problems

Build

No problems

L10n Repack

No problems

XULRunner Build

No problems

Unittests

Signing

Followed CombinedSigning doc.

L10n Verify

Nothing unusual in the logs.

Updates

3.6* -> 3.6.18

Generation

  • create_buildN step turned red with the following errors:
Error reading from /builds/slave/rel-192-updates/build/temp/firefox/3.6.17-3.6.18/aus2.test/Firefox/3.6.17/Darwin_Universal-gcc3/20110420140830/ml/betatest/complete.txt
[Errno 2] No such file or directory: '/builds/slave/rel-192-updates/build/temp/firefox/3.6.17-3.6.18/aus2.test/Firefox/3.6.17/Darwin_Universal-gcc3/20110420140830/ml/betatest/complete.txt'
...
Error reading from /builds/slave/rel-192-updates/build/temp/firefox/3.6.17-3.6.18/aus2/Firefox/3.6.17/Darwin_ppc-gcc3-u-ppc-i386/20110420140830/te/release/complete.txt
[Errno 2] No such file or directory: '/builds/slave/rel-192-updates/build/temp/firefox/3.6.17-3.6.18/aus2/Firefox/3.6.17/Darwin_ppc-gcc3-u-ppc-i386/20110420140830/te/release/complete.txt'

which are normal, because ml and te updates are disabled for some versions.

  • pushsnip timed out. Ran the following:
# on mv-moz2-linux-ix-slave06
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.17-3.6.18
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.18-build2-beta
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.18-build2
bash /builds/slave/rel-192-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest
echo $?
ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip Firefox-3.6.18-build2-test

Update Verify

  • rebuilt build1 builders (they use the same properties)

ml te problems

bug 666082 - Firefox 3.6.17 ml, te gets updated to 4.0.1 instead of 3.6.18

# gather the information
$ cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.17-3.6.18
$ stat -c "%s" ../3.6.18/ftp/firefox-3.6.18.ml.mac.complete.mar 
18922733
$ stat -c "%s" ../3.6.18/ftp/firefox-3.6.18.te.mac.complete.mar 
18923797
$ openssl dgst -sha512 ../3.6.18/ftp/firefox-3.6.18.ml.mac.complete.mar 
SHA512(../3.6.18/ftp/firefox-3.6.18.ml.mac.complete.mar)= f4a48b54b478049f493d4718936d9b8f115daad069e98b5d4551b029710ba520f1a70ebd3aa6e022f1443058246f11f5cdca9260dbf0116da20e06a62f7591d6
$ openssl dgst -sha512 ../3.6.18/ftp/firefox-3.6.18.te.mac.complete.mar 
SHA512(../3.6.18/ftp/firefox-3.6.18.te.mac.complete.mar)= 28d14c6e291bacde8f7bf3b43eb571b7323c97bf2852d8bf27de48807b1c838d90534e6a5b021e3d8a4affee65deeeeb6b38245ef74adabd9f614c406be6a2fe
#!/bin/bash
# on mv-moz2-linux-ix-slave06 in /builds/slave/rel-192-updates/build/temp/firefox/3.6.17-3.6.18
set -x
set -e

mkdir aus2.test.ml-te
cd aus2.test.ml-te

mkdir -p Firefox/3.6.17/Darwin_ppc-gcc3-u-ppc-i386/20110420140830/{ml,te}/betatest/

cat > Firefox/3.6.17/Darwin_ppc-gcc3-u-ppc-i386/20110420140830/ml/betatest/complete.txt << EOS
version=1
type=complete
url=http://download.mozilla.org/?product=firefox-3.6.18-complete&os=osx&lang=ml
hashFunction=SHA512
hashValue=f4a48b54b478049f493d4718936d9b8f115daad069e98b5d4551b029710ba520f1a70ebd3aa6e022f1443058246f11f5cdca9260dbf0116da20e06a62f7591d6
size=18922733
build=20110614230723
appv=3.6.18
extv=3.6.18
detailsUrl=https://www.mozilla.com/ml/firefox/3.6.18/releasenotes/
EOS

cat > Firefox/3.6.17/Darwin_ppc-gcc3-u-ppc-i386/20110420140830/te/betatest/complete.txt << EOS
version=1
type=complete
url=http://download.mozilla.org/?product=firefox-3.6.18-complete&os=osx&lang=te
hashFunction=SHA512
hashValue=28d14c6e291bacde8f7bf3b43eb571b7323c97bf2852d8bf27de48807b1c838d90534e6a5b021e3d8a4affee65deeeeb6b38245ef74adabd9f614c406be6a2fe
size=18923797
build=20110614230723
appv=3.6.18
extv=3.6.18
detailsUrl=https://www.mozilla.com/te/firefox/3.6.18/releasenotes/
EOS

for l in ml te; do
    sed 's/type=complete/type=partial/' Firefox/3.6.17/Darwin_ppc-gcc3-u-ppc-i386/20110420140830/$l/betatest/complete.txt > Firefox/3.6.17/Darwin_ppc-gcc3-u-ppc-i386/20110420140830/$l/betatest/partial.txt
    rsync -a Firefox/3.6.17/Darwin_ppc-gcc3-u-ppc-i386/20110420140830/$l/{betatest,releasetest}/

    for p in Darwin_Universal-gcc3 Darwin_x86-gcc3-u-ppc-i386; do
      mkdir -p Firefox/3.6.17/$p/20110420140830/$l/
      rsync -a Firefox/3.6.17/{Darwin_ppc-gcc3-u-ppc-i386,$p}/20110420140830/$l/
    done

done

cd -
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test.ml-te/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.18-build2-test-ml-te
# on aus2-staging:
~/bin/backupsnip Firefox-3.6.18-build2-test-ml-te
~/bin/pushsnip Firefox-3.6.18-build2-test-ml-te


# beta and release channels
rsync -a aus2.test.ml-te/ aus2.beta.ml-te/
find aus2.beta.ml-te -type d -name releasetest -exec rm -r {} \;

for l in ml te; do
  for p in Darwin_ppc-gcc3-u-ppc-i386 Darwin_Universal-gcc3 Darwin_x86-gcc3-u-ppc-i386; do
    mv aus2.beta.ml-te/Firefox/3.6.17/$p/20110420140830/$l/beta{test,}
  done
done


rsync -a aus2.test.ml-te/ aus2.ml-te/
find aus2.ml-te -type d -name betatest -exec rm -r {} \;

for l in ml te; do
  for p in Darwin_ppc-gcc3-u-ppc-i386 Darwin_Universal-gcc3 Darwin_x86-gcc3-u-ppc-i386; do
    mv aus2.ml-te/Firefox/3.6.17/$p/20110420140830/$l/release{test,}
  done
done

rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.beta.ml-te/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.18-build2-beta-ml-te
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.ml-te/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.18-build2-ml-te


# on aus2-staging:
~/bin/backupsnip Firefox-3.6.18-build2-beta-ml-te
~/bin/backupsnip Firefox-3.6.18-build2-ml-te

3.6.x -> 4.0.1 Major Updates

MU Generation

MU Verify

No problems.

Socorro product data updates

N/A for build2

Going to beta day!

Push to Beta Channel

As cltbld@aus2-staging:

cd /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-3.6.18-build2-beta
~/bin/pushsnip Firefox-3.6.18-build2-beta

Unthrottle beta channel

3.6.17 -> 3.6.18 on beta was already unthrottled, nothing to do there. But LegNeato also wants a MU from 3.6.18 -> 4.0.1 to be available but not advertised, so we need to rethrottle beta 3.6.18.

Push the MU to the beta channel

~/bin/backupsnip Firefox-3.6.18-build2-4.0.1-build1-MU-beta
~/bin/pushsnip   Firefox-3.6.18-build2-4.0.1-build1-MU-beta

Email mirrors about release day date

Done after the fact.

Going live day!

Final checks before push & virus scan

No viruses, no incorrect permissions.

Push to Mirrors

Used "force build" to start the release-mozilla-beta-push_to_mirrors builder with the following properties:

  • release_config: mozilla/release-firefox-mozilla-1.9.2.py
  • script_repo_revision: FIREFOX_3_6_18_RELEASE

Note that I incorrectly used mozilla-beta's builder, but because the logic is all client side, based on the properties, the right files were pushed. However, the wrong e-mails were sent.

Manually updated stage:/pub/mozilla.org/mozilla.org/zz/rsyncd-mozilla-current.exclude:

  • Removed 3.6.17
  • Added 3.6.18

Final Verification split by platform

Nothing unusual, macosx turned red because of ml, te locales without partial updates:

FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110420140830/Darwin_Universal-gcc3/te/releasetest/update.xml?force=1

Push release channel snippets

In advance, we need to backupsnip for the first two (the third is covered by the second one):

# cltbld@aus2-staging
~/bin/backupsnip Firefox-3.6.18-build2
~/bin/backupsnip Firefox-3.6.18-build2-3.5.19-build2

And the pushsnips:

# cltbld@aus2-staging
~/bin/pushsnip Firefox-3.6.18-build2
~/bin/pushsnip Firefox-3.6.18-build2-3.5.19-build2
~/bin/pushsnip Firefox-3.6.18-build2-3.5.19-build2-beta

And 3.6.18 -> 4.0.1 MU

# cltbld@aus2-staging
~/bin/pushsnip Firefox-3.6.18-build2-4.0.1-build1-MU

AUS Throttling

This patch was landed to update AUS throttling.

Release

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases/
rm latest-3.6 && ln -s 3.6.18 latest-3.6

Sent mail to metrics@mozilla.com:

Firefox,firefox,firefox,3.6.18,3.6
Firefox,firefox,firefox,3.6.19pre,3.6

XULRunner Signing

Signed with:

#!/bin/bash
set -x
set -e

PRODUCT=xulrunner
VERSION=1.9.2.18
BUILD=2
TAG=FIREFOX_3_6_18_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Push XULRunner to mirrors

# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip \
  /pub/mozilla.org/xulrunner/nightly/1.9.2.18-candidates/build2/ \
  /pub/mozilla.org/xulrunner/releases/1.9.2.18/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added

- xulrunner/releases/1.9.2.17

...to exclude the previous release.

Update XULRunner entries on wiki

Updated XULRunner wiki pages: https://developer.mozilla.org/en/XULRunner_1.9.2_Release_Notes
https://developer.mozilla.org/en/Gecko_SDK