~~Ship the l10n milestone ^DESKTOP ^MOBILE ^RELEASE ^BETA~~
~~Set reserved_slaves ^DESKTOP ^MOBILE ^RELEASE ^BETA ^ESR~~
~~Check to make sure you have all of the different types of slaves that you need ^DESKTOP ^MOBILE ^RELEASE ^BETA ^ESR~~
~~Start the automation ^DESKTOP ^MOBILE ^RELEASE ^BETA ^ESR~~
~~Sign Android builds ^MOBILE ^RELEASE ^BETA~~
~~Reset reserved_slaves and unlock slaves ^DESKTOP ^MOBILE ^RELEASE ^BETA ^ESR~~
~~Push to mirrors ^DESKTOP ^RELEASE~~
~~Publish Fennec ^MOBILE ^RELEASE ^BETA~~
~~Run pushsnip ^DESKTOP ^RELEASE ^BETA ^ESR~~
~~Update bouncer links ^DESKTOP ^RELEASE ^BETA~~

Bugs hit

bug 799599 - don't use mock slaves for releases until bug 798361 is done
bug 799806 - compile failures in libmar for repacks
bug 799347 - Investigate packet loss within SCL3
bug 799954 - linux repacks failed in firefox 17.0b1
bug 799988 - Install android-sdk-r16 RPM to IX machines
[no bug] - Add --enable-stdcxx-compat to l10n release mozconfigs for Linux mozilla-beta, to fix compiler errors.
bug 800329 - multiple locales donn't build properly on win32, empty values break
bug 800457 - 401 error hitting bouncer
bug 800925 - update verify config bumper doesn't handle dropped locales properly
bug 800924 - PatcherConfig class returns wrong update paths when a locale is dropped

Notes

Tracking bug: bug 796970
Release engineer: jhopkins
Buildbot master: bm13

Build 1

Starting the automation

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u jhopkins \
    -V 17.0b1 --branch mozilla-beta --build-number 1 \
    --release-config release-firefox-mozilla-beta.py \
    --release-config release-fennec-mozilla-beta.py --products firefox,fennec  \
    --dryrun localhost:9001

The initial tagging build and Linux builds failed because mozilla-beta was configured to use mock (release automation is not using mock yet). Rail wrote a patch to allow release configs to override "use_mock" setting of the platform config (eg. pf['linux']['use_mock']). We also needed to restore the "disable mock" block for mozilla-beta in config.py.

Fix repacks

First we hit bug 799806. nthomas wrote a patch to resolve the issue.

After that, Windows and Mac repacks worked fine but Linux hit:

===
=== If you get failures below, please file a bug describing the error
=== and your environment (compiler and linker versions), and use
=== --disable-elf-hack until this is fixed.
===
# Fail if the library doesn't have INIT_ARRAY .dynamic info
readelf -d test-array.so | grep '(INIT_ARRAY)'
 0x00000019 (INIT_ARRAY)                 0x54a8
/builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack -b -f test-array.so
/builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack: /tools/gcc-4.3.3/installed/lib/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /builds/slave/rel-m-beta-lnx-rpk-2/mozilla-beta/obj-l10n/build/unix/elfhack/elfhack)
make[4]: *** [test-array.so] Error 1

Catlee suggested that adding --enable-stdcxx-compat to the mozconfig would fix it. I (bhearsum) tested this by hand to verify, and then landed a fix in buildbot-configs: https://hg.mozilla.org/build/buildbot-configs/rev/e0994e97bf3a. After retagging and testing one repack chunk, I triggered the rest.

Respin declared for the same issue as the 16.0 chemspill

Build 2

Starting the automation

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u catlee \
    -V 17.0b1 --branch mozilla-beta --build-number 2 \
    --release-config release-firefox-mozilla-beta.py \
    --products firefox \
    --dryrun localhost:9001

Fennec beta tagging delayed due to SCL3 network issues. See bug 799347. Once the workaround was done:

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas \
    -V 17.0b1 --branch mozilla-beta --build-number 2 \
    --release-config release-fennec-mozilla-beta.py \
    --products fennec \
    --dryrun localhost:9001

which was clean, ran without the --dryrun.

Sign Android builds

Ran the following as cltsign@signing1:

cd ~/signing-work/mozharness
hg pull -u && hg up -C
cd ..
python2.6 mozharness/scripts/sign_android.py --config-file signing/android_mozilla-beta.py

Update generation failed because of missing 'km' build

Update generation failed with:

2012-10-11 08:22:03,160 - DEBUG - requests.packages.urllib3.connectionpool._make_request#267: "GET /pub/mozilla.org/firefox/nightly/17.0b1-candidates/build2/win32/km/firefox-17.0b1.checksums HTTP/1.1" 404 288
Traceback (most recent call last):
  File "tools/scripts/updates/create-snippets.py", line 90, in <module>
    info = getChecksum(platform, locale, checksumsFile)
  File "tools/scripts/updates/create-snippets.py", line 72, in getChecksum
    contents = requests.get(checksumsUrl, config={'danger_mode': True}).content
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/api.py", line 51, in get
    return request('get', url, **kwargs)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/api.py", line 39, in request
    return s.request(method=method, url=url, **kwargs)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request
    r.send(prefetch=prefetch)
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send
    self.response.raise_for_status()
  File "/builds/slave/rel-m-beta-updates/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status
    raise http_error
requests.exceptions.HTTPError: 404 Client Error

To fix, had to manually change the patcher config to exclude 'km' from win32 for 17.0b1: https://hg.mozilla.org/build/tools/rev/f85e15d49942. Retagged tools with FIREFOX_17_0b1_{RELEASE,BUILD2} and used "force build" to start updates again.

Failed again because "updates" regenerated the patcher config from shipped locales. Manually changed shipped-locales this time, on the relbranch: https://hg.mozilla.org/releases/mozilla-beta/rev/a3cbcfe120bd. Retagged tools with FIREFOX_17_0b1_{RELEASE,BUILD2} and used "force build" to start updates again.

Failed again because it still tried to download checksums file. FIXME there's a bug in the new patcher config code that means we don't support removed locales. Next around attempt: faking out the checksums and removing the snippets later:

# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates/17.0b1-candidates/build2/win32
mkdir km
cd km
cp ../ach/*.checksums .

Then forced the builder again.

After it completed, had to do some cleanup on stage:

# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates/17.0b1-candidates/build2/win32
rm -rf km

And in the snippets:

# ffxbld@aus3-staging
cd /opt/aus2/snippets/staging
find Firefox-17.0b1-build2 -maxdepth 5 -type d -wholename '*WINNT*km*' -exec rm -rf {} \;

Didn't bother deleting km test snippets from the test snippets. They'll get overwritten in 17.0b2.

Push to Mirrors

[2:37pm] bhearsum: jhopkins: because of the bug noted above bug 800457, you'll need
 to manually monitor 17.0b1's mirror uptake 
 (https://bounceradmin.mozilla.com/stats/uptake/?p=2038&p=2039&p=2041&p=2040), 
 and force build the final verify builders + "ready for releasetest" + "ready for release"
 when there's uptake
[2:41pm] jhopkins: bhearsum: i thought CDN uptake was instant?
[2:41pm] Callek: catlee-buildduty: bhearsum: just to say it -- bouncer in prod is working now -- at least for my seamonkey bouncer script
[2:42pm] bhearsum: jhopkins: near-instant
[2:42pm] bhearsum: sentry takes a few minutes to notice that it has uptake
[2:42pm] bhearsum: bounceradmin seems to be showing 50000002 availability now though, which means you have uptake
[2:43pm] jhopkins: so it's sentry we're waiting for, really
[2:43pm] bhearsum: kind of
[2:43pm] bhearsum: we're waiting for the bouncer system (which is bounceradmin + download.m.o + sentry) to be fully aware of the release
[2:43pm] bhearsum: if we don't, we'll serve 404s
[2:44pm] rail: plus sync from ftp.m.o to cdn?
[2:44pm] bhearsum: rail: that's near-instant, aiui
[2:46pm] jhopkins: i think how it works is the CDN only caches requested files.
 so it'll be cached on the CDN when the first user requests a file.  it will probably
 pull from us multiple times due to more than one distribution node plus when TTL expires
 for a file
[2:47pm] jhopkins: the other way to do it is to upload to the CDN manually and they store
 all the files but we're not operating that way

Bouncer indicates that uptake is over 50000000 for all platforms.

Kicked off final_verification builders. Some time later, IT fixed the bouncer issue and buildbot started a duplicate set of final_verification builders.

Investigated win32_update_verify failures. All '404' errors were due to the missing 'km' locale.

win32_final_verification '404' errors also failed predictably on the 'km' locale.

[5:12pm EST] QA has given approval for 17.0b1 _desktop_. lsblakk talking to PR and getting ready to give a final release 'go' - we have not received one yet. No word on Fennec QA progress yet.

Publish Fennec

(Done by Hwine due to jhopkins wanting to have a life.)

Grabbed:

   $ curl -O 'https://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/17.0b1-candidates/build2/android-armv6/mult
   i/fennec-17.0b1.multi.android-arm-armv6.apk'
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   100 20.8M  100 20.8M    0     0   140k      0  0:02:31  0:02:31 --:--:--  142k
   $ curl -O 'https://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/17.0b1-candidates/build2/android/multi/fennec-17.0b1.multi.android-arm.apk'
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   100 19.5M  100 19.5M    0     0   206k      0  0:01:36  0:01:36 --:--:--  568k
   0 [Hal@Hals-MacBook-Air Desktop]
   $ ll *apk
   -rw-r--r--  1 Hal  staff  21840767 Oct 11 14:59 fennec-17.0b1.multi.android-arm-armv6.apk
   -rw-r--r--  1 Hal  staff  20449419 Oct 11 15:25 fennec-17.0b1.multi.android-arm.apk
   0 [Hal@Hals-MacBook-Air Desktop]

Posted per instructions - neither the "recent changes" nor "description" content needed version specific updates. (Tried to clarify in notes.)

Pushed snippets w/o incident.

Publish Desktop

Pushsnips

   [ffxbld@dp-ausstage01 ~]$ cd /opt/aus2/snippets/staging
   [ffxbld@dp-ausstage01 staging]$ ~/bin/pushsnip Firefox-17.0b1-build2/

Ran w/o incident

Bouncer Admin update worked fine:

   $ curl -I 'http://download.mozilla.org/?product=firefox-beta-latest&os=win&lang=en-US'
   HTTP/1.1 302 Found
   Server: Apache
   X-Backend-Server: bouncer2.webapp.phx1.mozilla.com
   Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private
   Content-Type: text/html; charset=UTF-8
   Date: Thu, 11 Oct 2012 23:46:21 GMT
   Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/17.0b1/win32/en-US/Firefox%20Setup%2017.0b1.exe
   Pragma: no-cache
   Transfer-Encoding: chunked
   Connection: Keep-Alive

Releases/Firefox 17.0b1/BuildNotes

Contents

Bugs hit

Notes

Build 1

Starting the automation

Fix repacks

Build 2

Starting the automation

Sign Android builds

Update generation failed because of missing 'km' build

Push to Mirrors

Publish Fennec

Publish Desktop

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

How to Contribute

MozillaWiki

Around Mozilla

Tools