Releases/Firefox 3.6.14/BuildNotes

From MozillaWiki
Jump to: navigation, search

Contents

Build Engineers

armenzg - bug 624496

Signed-off Revision(s)

28ce03bbd37c

Tags

Build # Tag Changeset
1 FIREFOX_3_6_14_BUILD1 releases/mozilla-1.9.2/69bfc730b3e9
2 FIREFOX_3_6_14_BUILD2 releases/mozilla-1.9.2/77abb6fcd2b7
3 FIREFOX_3_6_14_BUILD3 releases/mozilla-1.9.2/a73120a6aa93

Notifications

  • We have to notify AV vendors once the signed bits are on candidates directory (for email address and template example, see bug 568884#c15 )
  • We have to give mirrors as much notice as possible before pushing to mirrors. This has been 24-72 hours, but has been up to a week in advance. (for email address and template example, see bug 633657 )
  • We have to notify infra-all@ before "go to build" and before we ship. The more advance notice the better.
  • We have to notify metrics once we release

Bugs hit

To be hit on next release unless fixed

You won't hit these bugs if they are fixed first:

  • bug 628795 - en-US.xpi uploaded to the wrong place during 1.9.1/1.9.2 releases
  • bug 616512 - Linux repacks will always fail to upload if run before win32 build finishes
  • bug 633961 - pre push check fails
  • [got backed out] bug 632399 ctors steps
  • bug 632412 - signing regression for win32-EUBallot
  • bug 632449 - generate-candidate-build-updates.py fails when locales are removed between buildX and buildY of the same release
    • this is only hit for a build2
  • bug 633695 - EUBallot should be renamed to EUballot (lower case)

Sporadic/Not easily hitable

  • bug 628390 - macosx_repack_4/6 failed because post_upload failed for 'ko'
  • bug 628386 - win32_repack_5/6 failed to timeout on w32-ix-slave23 being slow
  • bug 557268 - "update_verify" dependent scheduler did not fire
  • bug 633336 - 2nd run of mac partner repacks fails because previous run did not remove mount point
  • bug 629256 - Fix snippets for some locales
  • macosx_repack_2/6 died with SIGUSR1

Build data

Build # Type Build ID Build machine Time to build
1 Linux 20110121145245 mv-moz2-linux-ix-slave11 38 mins, 28 secs
Mac 20110121145415 bm-xserve16 1 hrs, 39 mins, 47 secs
Windows 20110121145327 w32-ix-slave23 4 hrs, 3 mins, 53 secs
2 Linux 20110207130507 mv-moz2-linux-ix-slave11 39 mins, 27 secs
Mac 20110207130523 bm-xserve16 1 hrs, 39 mins, 5 secs
Windows 20110207134212 w32-ix-slave23 (slow IX) 5 hrs, 11 mins, 35 secs
3 Linux 20110218124314 moz2-linux-slave23
Mac 20110218124600 bm-xserve16
Windows 20110218125750 w32-ix-slave23 (slow IX)

Build 1

  • Clobbered with "Any master/any release/any builder"
  • tagged buildbotcustom, compare-locales and tools with FIREFOX_3_6_14_BUILD1 and FIREFOX_3_6_14_RELEASE
  • landed config updates, including l10n-changesets from here
  • tagged buildbot-configs on "production" branch with FIREFOX_3_6_14_BUILD1 and FIREFOX_3_6_14_RELEASE
  • merge to production
  • updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
  • reconfigured master (Not needed - release_sanity.py takes care of it)
  • Left tree open, 0.8.0 automation doesn't require that it closes
  • kicked off automation with: 
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python /builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py -u armenzg -V 3.6.14 --branch mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010

Tag

No problems.

Version bump

Please verify on your next release that bug 607372 is fixed.

Once you do VERIFY the bug and remove this section.

Update Bouncer

Used "Force Build" to start the "release-mozilla-1.9.2-bouncer_submitter" builder.

All good.

Stage EU Ballot Builds

# ffxbld@stage
cd /home/ftp/pub/firefox/nightly/3.6.14-candidates/build1
mv -v partner-repacks/euballot/win32 win32-EUballot

Please remove this section once bug 614227 is FIXED.

Source / XULRunner Source

No problems.

Build

  • No problems

Remove en-US.xpi

You want to remove the en-US.xpi before autosigning creates the MD5SUMS and the SHA1SUMS.

Please remove this section once bug 628795 is fixed.

Check here if you don't do remove en-US.xpi on time and want to know how to recover.

Start autosigning

If you are expecting to sign during your workday you can start auto signing to take care of the deliverables as they are uploaded to FTP.

L10n Repack

  • macosx_repack_2/6 died with SIGUSR1 before attempting any repacks
    • Manually rebooted bm-xserve16
    • Retriggered the builder with "rebuild"
    • check bugs hit section for any filed bug
  • macosx_repack_4/6 failed because post_upload failed for 'ko'
    • used "macosx_standalone_repack" to re-trigger 'ko' as documented in here
    • check bugs hit section for any filed bug
  • win32_repack_5/6 failed to timeout on w32-ix-slave23 being slow
    • gracefully shutdown the slave and re-triggered the job
    • check bugs hit section for any filed bug

Partner Repack

  • Linux & Mac partner repack failed to upload builds bug 616512
    • the way to recover until the bug is fixed is to wait for win32 build to finish and then just re-trigger; no clobber is needed as it is not used by the builder
    • instead I failed to read note from bhearsum and I did the following
      • tried to debug locally on "bm-xserve22" why it did not work but when run it worked
cd /builds/slave/rel-192-lnx-partner-rpk/./partner-repacks/scripts/repacked_builds/3.6.14/build1
rsync -av -e ssh -oIdentityFile=~/.ssh/ffxbld_dsa . \
ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build1/unsigned/partner-repacks
bash -c 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa ffxbld@stage-old.mozilla.org \
touch /pub/mozilla.org/firefox/nightly/3.6.14-candidates/build1//unsigned/partner-repacks/partner_build_linux'
# after touching the autosign system can proceeed
  • 2nd resubmitted Mac repack failed
    • the reason is that the first job that failed did not run the cleanup job
    • check bugs hit section for any filed bug
  • It is all fine now

XULRunner Build

All good.

Unittests

Linux:

  • hit bug 565437 timeout which aborted the run; I re-triggered it

Darwin:

Windows:

  • green

Signing

Followed CombinedSigning doc.

I did remove the en-US.xpi file too late.

To recover I did the following: 

#ffxbld@stage-old.mozilla.org
rm /home/ftp/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build1/en-US.xpi

On the keymaster: 

cd ~/signing-work/firefox-3.6.4/signed_build1
rm en-US.xpi MD5SUMS* SHA1SUMS*
# I run the following command but by running each individually
# because I didn't know I could run it in one shot
# NOTE that "stage", "verify" and "fake-upload" are targets 
# that happen after "sign2" is run
# http://hg.mozilla.org/build/tools/file/tip/release/signing/Makefile#l73
make checksum-files create-sigs verify-sigs quick-verify stage verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1

Then continued by uploading as it would normally be done.

Notify AV vendors

L10n Verify

Many string changes across many platforms/locales. All seemed legitimate, didn't dig further.

Updates

  • completion on Monday Jan. 24th @ 15:30pm PST.
  • QA signed off @ 4:22pm PST

3.6* -> 3.6.13

Generation

Got triggered after signing bits appeared on ftp.

No problems.

Update Verify

I had to "force build" the "update_verify" since reconfigures happened since the release was triggered. (added to bugs hit section)

All went green.

Socorro product data updates

Added '3.6.14' & '3.6.15pre' to crash-stats with this admin page.

Going to beta day!!

Do this in the morning before the rel-driver gives you the GO. 

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
~/bin/backupsnip 20110124-Firefox-3.6.14-beta #takes 20-50mins - depends if other releases are backing up

Push to Beta Channel

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of PHX datacenter
~/bin/pushsnip 20110124-Firefox-3.6.14-beta

Fix snippets for some locales

See bug 629256 

cd /opt/aus2/snippets/staging

rsync -a 20110124-Firefox-3.6.14/ 20110126-Firefox-3.6.14-bug629256fix/
touch 20110126-Firefox-3.6.14-bug629256fix/
rm -rf 20110126-Firefox-3.6.14-bug629256fix/Firefox/3.6*/Darwin*/*/{ml,te}

rsync -a 20110124-Firefox-3.5.17-3.6.14-MU/
20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix/
touch 20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix/
rm -rf
20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix/Firefox/3.5*/Darwin*/*/{kn,ml,te}

Email mirrors about release day date

They need to know 24-48 hours ahead of time. 

On $DATE1 or $DATE2, we plan to "push to
mirrors" for the Firefox '''$3_6_VERSION''' and Firefox '''$3_5_VERSION''' security releases.

If you believe this will cause any problems, or there are other releases
scheduled for those dates on mirrors, please let us know at
'''$RELEASE_MAILING_LIST'''.

Thanks
'''$NAME'''

Going live day!

Final checks before push & virus scan

Go to the "pre push checks" builder and add these two values:

  • Revision to build FIREFOX_3_6_14_RELEASE
  • Property 1 name: release_config: mozilla/release-firefox-mozilla-1.9.2.py

Push to Mirrors

Skipping; doing a build 2.

Final Verification

Skipping; doing a build 2.

Push release channel snippets

Skipping; doing a build 2.

Please use the 20110126-Firefox-3.6.14-bug629256fix directory when pushing to the release channel for 3.6.x -> 3.6.14 updates, for bug 629256. We drop updates for a few locales which have strings that make the updater crash. NThomas 23:52, 26 January 2011 (PST)

Redirect EU Ballot build and add to bouncer

Skipping; doing a build 2.

Release

Skipping; doing a build 2.

XULRunner Signing

Skipping; doing a build 2.

Push XULRunner to mirrors

Skipping; doing a build 2.

Update XULRunner entries on wiki

Skipping; doing a build 2.

Build 2

  • Clobbered with "Any master/any release/any builder"
  • tagged buildbot-configs on production, buildbotcustom on production-0.8, compare-locales and tools with FIREFOX_3_6_14_BUILD2 and FIREFOX_3_6_14_RELEASE
  • [ landed build2 changes]
  • updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
  • Left tree open, 0.8.0 automation doesn't require that it closes
  • kicked off automation with: 
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python /builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py -u armenzg -V 3.6.14 --branch mozilla-1.9.2 --build-number 2 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010

Tag

No problems. I triggered automation twice by mistake so I had to cancel one of the tag jobs.

Version bump

Not needed as it is build2.

Update Bouncer

Not needed as it is build2.

Source / XULRunner Source

No problems.

Build

No problems besides:

  • win32 was run on an slow IX slave and QA did not get the builds before EOD

Start autosigning

If you are expecting to sign during your workday you can start auto signing to take care of the deliverables as they are uploaded to FTP.

L10n Repack

No problems.

Partner Repack

No problems.

XULRunner Build

  • Linux xulrunner build has a couple of properties going red
    • This is not important. The status is red but nothing bad happeneded
    • This is a fallout from ctors bug
    • Filed as bug 632399

Unittests

  • Found and notified release-drivers of orange found in all 3 platforms
39527 ERROR TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_bug481335.xhtml | Should be visited now - "rgb(0, 0, 238)" should equal "rgb(85, 26, 139)"
39528 ERROR TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_bug481335.xhtml | Should still be visited after setting pathname to its existing value - "rgb(0, 0, 238)" should equal "rgb(85, 26, 139)"
39531 ERROR TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_bug481335.xhtml | Should be visited now after third load - "rgb(0, 0, 238)" should equal "rgb(85, 26, 139)"

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6-Release/1297114970.1297116997.23122.gz http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6-Release/1297118544.1297120352.5609.gz http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6-Release/1297131303.1297132922.29438.gz

Signing

Followed CombinedSigning doc.

The en-US.xpi change landed on 1.9.1/1.9.2 branches but I forgot that it did not land on the RELBRANCH. Therefore, I forgot again to remove it and I have to recover again.

To recover I did the following: 

#ffxbld@stage-old.mozilla.org
rm /home/ftp/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build2/en-US.xpi

On the keymaster: 

cd ~/signing-work/firefox-3.6.14/signed-build2
rm en-US.xpi MD5SUMS* SHA1SUMS*
cd ..
rm unsigned-build2/en-US.xpi
# http://hg.mozilla.org/build/tools/file/tip/release/signing/Makefile#l73
# NOTE: From the sign2 target we only need to run "checksum-files",
# "create-sigs", "verify-sigs" and "quick-verify"
# http://hg.mozilla.org/build/tools/file/tip/release/signing/Makefile#l218
make checksum-files create-sigs verify-sigs quick-verify stage verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1
  • We also hit a signing regression from BUILD1 to BUILD2 - see bug 632412.

To recover I did this: 

cd ~/signing-work/firefox-3.6.14
# The next 4 steps are jsut in case we need them for any reason
mkdir -p euballot/signed-build2
mkdir -p euballot/unsigned-build2/unsigned
mv unsigned-build2/unsigned/win32-EUBallot euballot/unsigned-build2/unsigned
mv signed-build2/win32-EUBallot euballot/signed-build2
# Resuming where it bailed out
make verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1
# Uploading
make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

So we postponed the signing of the EUBallot until bhearsum run a fix on staging. These are the steps followed: 

cd ~/hg-tools
hg pull
hg up -r FIREFOX_3_6_14_RELEASE
cd ~/signing-work/firefox-3.6.14
cp ~/hg-tools/release/signing/* .
rsync -av -e 'ssh -i /home/cltsign/.ssh/ffxbld_dsa' ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build2/unsigned/win32-EUBallot/ unsigned-build2/unsigned/win32-EUBallot/
mkdir -p unsigned-build2-euballot-fix/unsigned
rsync -av unsigned-build2/unsigned/win32-EUBallot/ unsigned-build2-euballot-fix/unsigned/win32-EUBallot/
rsync -av --include=unsigned --include=win32 --include=update --include=en-US --include=*.exe --include=*.mar --exclude=* unsigned-build2/unsigned/ unsigned-build2-euballot-fix/unsigned/
./sign-release.py --keydir=${KEYDIR} -j4 --product Firefox -o signed-build2 -p unsigned-build2-euballot-fix
make create-sigs verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1
# Upon inspection
make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Even after the above, EUBallot builds were not properly signed. This time, because of bug 632793. To fix, did the following: 

cd ~/signing-work/firefox-3.6.14
rm -rf cache
# Create a new directory to plop signed ballot builds in, just in case we screw something up
rsync -av --exclude=win32-EUBallot signed-build2/ signed-build2-euballot-attempt2/
# Populate the cache by running sign-release.py w/ --keep-cache and --first-locale against the original directory. This causes it to unpack the first locale's build, cache it, and then exit because there's nothing to sign
for l in `ls unsigned-build2/unsigned/win32-EUBallot`; do ./sign-release.py --keydir=${KEYDIR} -j4 --product Firefox -o signed-build2-euballot-attempt2 -p --first-locale ${l%\/} --keep-cache unsigned-build2/; done
# Now, run sign-release.py on the directory with only the EUBallot builds in it. This will have a 100% hit rate, because all of the files from these locales are in the cache.
./sign-release.py --keydir=${KEYDIR} -j4 --product Firefox -o signed-build2-euballot-attempt2 -p --keep-cache unsigned-build2-euballot-fix/
# Generate detached sigs for them
cd signed-build2-euballot-attempt2
../signing/sign-files .
# Run verify-signatures to ensure everything is OK
./verify-signature.py --product firefox unsigned-build2 signed-build2-euballot-attempt2 > verify-signatures-build2-euballot.fix.log
# Upload!
rsync -e "ssh -i /home/cltsign/.ssh/ffxbld_dsa" -av signed-build2-euballot-attempt2/ ffxbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.6.14-candidates/build2/

Rename win32-EUBallot for win32-EUballot

This is bug 633695 - EUBallot should be renamed to EUballot.

There are two patches to fix it but the fix on stage was: 

# ffxbld@stage
cd /home/ftp/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build2/
mv win32-EUBallot win32-EUballot

Notify AV vendors

L10n Verify

I don't even see the point of this builder. It's always red (expected) and the output does not tell you if there is anything wrong with it.

Updates

3.6* -> 3.6.13

Generation

Failed.

Something had landed to fix crashing locales in bug 629256. generate-candidate-build-updates.py fails when locales are removed between buildX and buildY of the same release - see bug 632449

We won't be generating buildX to buildY updates and this is how we recovered (from looking at build1 log): 

cd /builds/slave/rel-192-updates/
find build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build2 -type f -exec chmod 644 {} \;
find build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build2 -type d -exec chmod 755 {} \;
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build2
rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' --exclude=*complete.mar update ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build2/
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20110208-Firefox-3.6.14-test
cd /builds/slave/rel-192-updates/build
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/backupsnip 20110208-Firefox-3.6.14-test'
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip 20110208-Firefox-3.6.14-test'
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20110208-Firefox-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20110208-Firefox-3.6.14-beta
bash /builds/slave/rel-192-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest
Update Verify

"Force build" was used since the updates builder failed.

  • linux and windows update verify succeded
  • mac update verify failed for "ml" and "te"

I grep the log with "grep -E (FAIL|Using)" and extracted the blocks that had failures: 

Using  https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/ml/betatest/update.xml?force=1
FAIL: binary files found in diff 
FAIL: check_updates returned failure for Darwin_Universal-gcc3 downloads/Firefox 3.6.13.dmg vs. downloads/Firefox 3.6.14.dmg: 1
Using  https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/ml/betatest/update.xml?force=1
FAIL: binary files found in diff 
FAIL: check_updates returned failure for Darwin_Universal-gcc3 downloads/Firefox 3.6.13.dmg vs. downloads/Firefox 3.6.14.dmg: 1

Using  https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/te/betatest/update.xml?force=1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Darwin_Universal-gcc3 downloads/Firefox 3.6.13.dmg vs. downloads/Firefox 3.6.14.dmg: 1
Using  https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/te/betatest/update.xml?force=1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Darwin_Universal-gcc3 downloads/Firefox 3.6.13.dmg vs. downloads/Firefox 3.6.14.dmg: 1

Socorro product data updates

Already done for build 1

Going to beta day!!

Do this in the morning before the rel-driver gives you the GO. 

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20110208-Firefox-3.6.14-beta # takes more than 45 minutes

Push to Beta Channel

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of PHX datacenter
cd /opt/aus2/snippets/staging/
~/bin/pushsnip 20110208-Firefox-3.6.14-beta

Fix snippets for some locales

Nothing to be done since nthomas landed some fixes in bug 629256

Email mirrors about release day date

They need to know 24-48 hours ahead of time. 

On $DATE1 or $DATE2, we plan to "push to
mirrors" for the Firefox '''$3_6_VERSION''' and Firefox '''$3_5_VERSION''' security releases.

If you believe this will cause any problems, or there are other releases
scheduled for those dates on mirrors, please let us know at
'''$RELEASE_MAILING_LIST'''.

Thanks
'''$NAME'''

Going live day!

Final checks before push & virus scan

Go to the "pre push checks" builder and add these two values:

  • Revision to build FIREFOX_3_6_14_RELEASE
  • Property 1 name: release_config: mozilla/release-firefox-mozilla-1.9.2.py
  • It failed the first time because of permission issues. bug 633961
  • I triggered it a second time

Skipping; doing a build 3.

Push to Mirrors

Skipping; doing a build 3.

Final Verification

Skipping; doing a build 3.

Push release channel snippets

Skipping; doing a build 3.

Push EU Ballot build

Skipping; doing a build 3.

Release

Skipping; doing a build 3.

XULRunner Signing

Skipping; doing a build 3.

Push XULRunner to mirrors

Skipping; doing a build 3.

Update XULRunner entries on wiki

Skipping; doing a build 3.

Build 3

NOTE: We might not send this build to the users but build#2. This will be decided on Tuesday

  • Clobbered with "Any master/any release/any builder"
  • tagged with FIREFOX_3_6_14_BUILD3 and FIREFOX_3_6_14_RELEASE
    • buildbot-configs on production
    • buildbotcustom on production-0.8
    • compare-locales & tools
  • landed build2 changes
  • updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
  • Left tree open, 0.8.0 automation doesn't require that it closes
  • kicked off automation with: 
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python /builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py -u armenzg -V 3.6.14 --branch mozilla-1.9.2 --build-number 3 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010

Tag

No problems.

Version bump

Not needed as it is build3.

Update Bouncer

Not needed as it is build3.

Source / XULRunner Source

No problems.

XULRunner Build

No problems.

Build

No problems.

Start autosigning

If you are expecting to sign during your workday you can start auto signing to take care of the deliverables as they are uploaded to FTP.

L10n Repack

No problems.

  • repack 5/6 failed due to checkout timeout on slow slave
    • re-triggered

Partner Repack

No problems.

Unittests

All three platforms fails with these: 

39509 ERROR TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_bug481335.xhtml | Should be visited now - "rgb(0, 0, 238)" should equal "rgb(85, 26, 139)"
39510 ERROR TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_bug481335.xhtml | Should still be visited after setting pathname to its existing value - "rgb(0, 0, 238)" should equal "rgb(85, 26, 139)"

Notified LegNeato.

Signing

Followed CombinedSigning doc.

Notify AV vendors

L10n Verify

I don't even see the point of this builder. It's always red (expected) and the output does not tell you if there is anything wrong with it.

Updates

3.6* -> 3.6.13

Generation

We landed a fix to avoid this from happening but I forgot to reconfigure. 

cd /builds/slave/rel-192-updates/
find build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build3 -type f -exec chmod 644 {} \;
find build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build3 -type d -exec chmod 755 {} \;
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build3
rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' --exclude=*complete.mar update ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build3/
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20110219-Firefox-3.6.14-test
cd /builds/slave/rel-192-updates/build
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/backupsnip 20110219-Firefox-3.6.14-test'
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip 20110219-Firefox-3.6.14-test'
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20110219-Firefox-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/2011019-Firefox-3.6.14-beta
bash /builds/slave/rel-192-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest
Update Verify

Same as with update verify for build2. 'mk' and 'te' failed for mac.

See bug 629256 for more details.

Socorro product data updates

Already done for build 1

Going to beta day!!

Do this in the morning before the rel-driver gives you the GO. 

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20110219-Firefox-3.6.14-beta # takes more than 45mins

To make backupsnip to work we had to re-enable updates (bug 636218): 

#on aus2-staging and phx
cd /opt/aus2/incoming/3/Firefox/
chmod 700 -v 3.6*

Push to Beta Channel

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of PHX datacenter
cd /opt/aus2/snippets/staging/
~/bin/pushsnip 20110208-Firefox-3.6.14-beta

Email mirrors about release day date

They need to know 24-48 hours ahead of time. 

On $DATE1 or $DATE2, we plan to "push to
mirrors" for the Firefox '''$3_6_VERSION''' and Firefox '''$3_5_VERSION''' security releases.

If you believe this will cause any problems, or there are other releases
scheduled for those dates on mirrors, please let us know at
'''$RELEASE_MAILING_LIST'''.

Thanks
'''$NAME'''

Going live day!

Final checks before push & virus scan

I trigger the builders last week and it all went well.

I trigger it once more manually to re-verify it. 

clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.14-candidates/build3/
/mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.14-candidates/build3: OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 5466.948 sec (91 m 6 s)

Backup release snippets

Do this in the morning ahead of time before the "push snippets to release". 

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20110219-Firefox-3.6.14

Push to Mirrors and modify exclude file

After some re-tagging and reconfigures for getting some changes that had landed on default we manage to trigger this builder and succeed to completion. http://hg.mozilla.org/build/buildbotcustom/rev/03084b14ff4b

Push to community mirrors: 

# cltbld@stage
vim -n /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.6.13 with 3.6.14

Final Verification

Triggered automatically.

Push release channel snippets

The backupsnip was done earlier in the Releases/Firefox_3.6.14/BuildNotes#Backup_release_snippets backup release snippets section. 

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
~/bin/pushsnip 20110219-Firefox-3.6.14

Redirect EU Ballot build and add bouncer entry

Reopened bug 632052 for IT to fix it when the snippets are live

NOTE: I think there is a bug that got fixed recently that adds the bouncer entry for the EUballot builds automatically. I can't find it right now.

  • Filed bug 632052 to change download redirect
  • On bouncer we add "Firefox-3.6.14-EUballot" product under "Mirrors->Products->Add new product"
  • Once added, we add a new mirror location:
/firefox/releases/3.6.14/win32-EUballot/:lang/Firefox%20Setup%203.6.14.exe

Release

Update symlinks 

#ffxbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.6 && ln -s 3.6.14 latest-3.6

Sent the following mail to metrics@mozilla.com: 

Firefox,firefox,firefox,3.6.13,3.6
Firefox,firefox,firefox,3.6.14pre,3.6

XULRunner Signing

Create ~/signXR1.9.2.14.bash file with chmod +x and this content: 

#!/bin/bash
set -x
set -e

PRODUCT=xulrunner
VERSION=1.9.2.14
BUILD=3
TAG=FIREFOX_3_6_14_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=armenzg@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Push XULRunner to mirrors

# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip \
  /pub/mozilla.org/xulrunner/nightly/1.9.2.14-candidates/build3/ \
  /pub/mozilla.org/xulrunner/releases/1.9.2.14/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added 

- xulrunner/releases/1.9.2.13

Update XULRunner entries on wiki

Updated the links in these two wiki pages for the version change:

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_3.6.14/BuildNotes&oldid=288022"