Firefox

Build 1

~~(RelMan) Ship the l10n milestone (build1 only)~~

~~(RelMan) Submit to Ship It~~

ABANDONED.

Issues

Intermittent/silly things

linux64 repack 2/10 hang downloading mar file for the first locale

Downloading http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0-candidates/build1/linux-x86_64/en-US/firefox-40.0.tar.bz2 to firefox.tar.bz2
Downloading http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/38.0.5-candidates/build4/update/linux-x86_64/bn-BD/firefox-38.0.5.complete.mar to firefox-38.0.5.bn-BD.complete.mar
Downloading http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/35.0.1-candidates/build1/update/linux-x86_64/bn-BD/firefox-35.0.1.complete.mar to firefox-35.0.1.bn-BD.complete.mar
Downloading http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/37.0.2-candidates/build1/update/linux-x86_64/bn-BD/firefox-37.0.2.complete.mar to firefox-37.0.2.bn-BD.complete.mar

command timed out: 2400 seconds without output ... program finished with exit code 247

win32 repack 10/10 ran out of space with two locales to do. bug 1190644 to increase the requirement. It auto-retried but that was missed, and the standalone_repack job was used
win32 repack 6/10 also failed, got 9.0G after asking for 7G. Auto-retried.

Release channel updates builder failure

firefox_release_updates failed with duplicate past-update lines, fixed with a config tweak

This probably also caused the bustage in beta_final_verification and beta_update_verify (some but not all, strangely). Should we block on both updates builders completing ?

beta update verify failures in chunks 3 and 4 (all platforms

Both of these are showing the "complete" block missing from update response for 39.0 release (they failures look slightly different, because chunk 3 is doing a full check, and chunk 4 is doing a quick check).

When Ben investigated this, Balrog was returning a response that included a complete and a partial, but the partial was pointing at Bouncer instead of directly at FTP. This is because the 39.0 -> 40.0 partial is shared with the release channel, so they Balrog submitter didn't submit it for Beta as well. We should fix that, and also bug 1170919.

To fix:

Adjusted the blob as follows:

--- Firefox-40.0-build1.json.orig	2015-08-04 08:57:32.402237257 -0400
+++ Firefox-40.0-build1.json	2015-08-04 08:59:24.654899234 -0400
@@ -15,7 +15,8 @@
         },
         "beta": {
             "partials": {
-                "Firefox-40.0b9-build1": "http://download.mozilla.org/?product=firefox-40.0build1-partial-40.0b9build1&os=%OS_BOUNCER%&lang=%LOCALE%"
+                "Firefox-40.0b9-build1": "http://download.mozilla.org/?product=firefox-40.0build1-partial-40.0b9build1&os=%OS_BOUNCER%&lang=%LOCALE%",
+                "Firefox-39.0-build6": "http://download.mozilla.org/?product=firefox-40.0build1-partial-39.0build6&os=%OS_BOUNCER%&lang=%LOCALE%"
             },
             "completes": {
                 "*": "http://download.mozilla.org/?product=firefox-40.0build1-complete&os=%OS_BOUNCER%&lang=%LOCALE%"
@@ -35,7 +36,8 @@
         },
         "beta-cdntest": {
             "partials": {
-                "Firefox-40.0b9-build1": "http://download.mozilla.org/?product=firefox-40.0build1-partial-40.0b9build1&os=%OS_BOUNCER%&lang=%LOCALE%"
+                "Firefox-40.0b9-build1": "http://download.mozilla.org/?product=firefox-40.0build1-partial-40.0b9build1&os=%OS_BOUNCER%&lang=%LOCALE%",
+                "Firefox-39.0-build6": "http://download.mozilla.org/?product=firefox-40.0build1-partial-39.0build6&os=%OS_BOUNCER%&lang=%LOCALE%"
             },
             "completes": {
                 "*": "http://download.mozilla.org/?product=firefox-40.0build1-complete&os=%OS_BOUNCER%&lang=%LOCALE%"
@@ -43,7 +45,8 @@
         },
         "beta-localtest": {
             "partials": {
-                "Firefox-40.0b9-build1": "http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0-candidates/build1/update/%OS_FTP%/%LOCALE%/firefox-40.0b9-40.0.partial.mar"
+                "Firefox-40.0b9-build1": "http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0-candidates/build1/update/%OS_FTP%/%LOCALE%/firefox-40.0b9-40.0.partial.mar",
+                "Firefox-39.0-build6": "http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0-candidates/build1/update/%OS_FTP%/%LOCALE%/firefox-39.0-40.0.partial.mar"
             },
             "completes": {
                 "*": "http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0-candidates/build1/update/%OS_FTP%/%LOCALE%/firefox-40.0.complete.mar"

Retriggered the failed update verify chunks

Mac release update verify failures

5 of the mac release update_verify failed with either an empty response, or completes that specified download.m.o instead of stage.m.o (despite being on the release-localtest channel). Not reproducible now, so must have been caching issues. We may need a delay at the end of the updates builder again. Also should look to see if caching is doing what we expect.

Build 2

~~(RelMan) Ship the l10n milestone (build1 only)~~

~~(RelMan) Submit to Ship It~~

~~Publish RC to Beta channel~~

ABANDONED

Issues

Updates builder losing race condition

firefox_beta_updates lost the push race into the tools repo, reran it.

Manually updated balrog blob for 39.0 on beta channels

Updated the Firefox-40.0-build2 with partial for 39.0 on beta channels, same as build1 only s/build1/build2/.

Beta final verification failures

Beta final verification failed because it attempted to download a bunch of MARs from release-style download.m.o links, eg: http://download.mozilla.org/?product=firefox-40.0-complete&os=linux64&lang=ja&force=1. It should be using versions like http://download.mozilla.org/?product=firefox-40.0build2-complete&os=linux64&lang=ja&force=1 instead. Couldn't repro by hand, so reran it. There's definitely some sort of issue around sometimes not serving the right thing for RCs, possibly related to bug 1190779.

INVESTIGATE ME FURTHER!

Windows update verify failure

One chunk failed when purging space. Reran it.

Build 3

~~(RelMan) Ship the l10n milestone (build1 only)~~

~~(RelMan) Submit to Ship It~~
Abandoned for build with correct m-r revision

Update patcher config before the release started

Due to the chemspill that came up, the patcher config needed a tweak before the release started. bug 1191832 has more information and the patch.

Build 4

~~(RelMan) Ship the l10n milestone (build1 only)~~

~~(RelMan) Submit to Ship It~~

~~Publish RC to Beta channel~~

ABANDONED

Issues

firefox_release_updates lost the push race into tools with firefox_beta_updates, reran it
beta_final_verification failed on 39.0 - Patch type 'complete' not present in the downloaded update.xml file
- added partials for Firefox-39.0build6 to the three beta channels, reran final verify

Build 5

~~(RelMan) Ship the l10n milestone (build1 only)~~

~~(RelMan) Submit to Ship It~~
Publish RC to Beta channel - not planning to do this, too close to shipping day to get any feedback
Set up whatsnew page - no what's new page for this release
~~Push to mirrors~~

Apply Throttling

Verify throttling

Publish in Balrog

~~Post-release tasks~~

Issues

Hit bug 1191353 in one 32-bit Linux repack chunked. Using standalone repack to do the rest of the locales.
four win32 repacks were interrupted by a network problem, auto-retried successfully
beta_final_verification and update verify 5/12 & 8/12 - as in earlier builds, added the Firefox-39.0build6 lines to the fileUrls block in the blob. bug 1192674 to fix this for good

XULRunner repack

bug 1193177 requests adding another file to the win32 sdk. The alternatives are not great

land fix and redo all firefox & xulrunner builds (unjustifiable so close to release, but consistent)
land fix and move the FIREFOX_40_0_RELEASE tag, redo win32 xulrunner build & checksums (no tag on rev for firefox builds)
update the win32 xulrunner sdk manually, leave source & tag alone (inconsistent to 40.0.1 or 41)

The last one is least evil.

Working on b-2008-ix-0115, which did the original build

# update the sdk zip, and upload it
$ cd /c/builds/moz2_slave/rel-m-rel-xr_w32_bld-000000000/build/obj-firefox/dist
$ mkdir sdk.bak; cp -rp sdk/* sdk.bak/
$ cp -p lib/mozcrt.lib xulrunner-sdk/sdk/lib/
$ zip -u sdk/xulrunner-40.0.en-US.win32.sdk.zip xulrunner-sdk/sdk/lib/mozcrt.lib
  adding: xulrunner-sdk/sdk/lib/mozcrt.lib (140 bytes security) (deflated 78%)
$ scp -i ~/.ssh/xrbld_dsa sdk/xulrunner-40.0.en-US.win32.sdk.zip \
    xrbld@stage.mozilla.org:/pub/mozilla.org/xulrunner/candidates/40.0-candidates/build5/sdk/
$ openssl dgst -sha1 sdk/xulrunner-40.0.en-US.win32.sdk.zip
SHA1(sdk/xulrunner-40.0.en-US.win32.sdk.zip)= 1f9c111562fcf8419e3f4d5b48f534e97f2d47fc

Then on partner-repack1 (as a convenient place to do signing)

# getting signing token
$ mkdir ~/bug1193177-deleteme201509 && cd $_
$ wget https://hg.mozilla.org/build/partner-repacks/raw-file/FIREFOX_40_0_RELEASE/scripts/signing-server-setup.sh
$ ./signing-server-setup.sh
$ export MOZ_SIGN_CMD="python /Users/cltbld/bug1193177-deleteme201509/tools/release/signing/signtool.py -t /Users/cltbld/bug1193177-deleteme201509/token -n /Users/cltbld/bug1193177-deleteme201509/nonce -c /Users/cltbld/bug1193177-deleteme201509/tools/release/signing/host.cert -H gpg:osslsigncode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120"

# generate detached sig for sdk zip
$ mkdir -p build5/sdk
$ wget -O build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip "https://stage.mozilla.org/pub/mozilla.org/xulrunner/candidates/40.0-candidates/build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip?123"
$ ${MOZ_SIGN_CMD} -f gpg -o build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip.asc build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip
2015-08-11 03:38:03,109 - 1f9c111562fcf8419e3f4d5b48f534e97f2d47fc: processing build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip on https://signing4.srv.releng.scl3.mozilla.com:9120
2015-08-11 03:38:03,125 - 1f9c111562fcf8419e3f4d5b48f534e97f2d47fc: uploading for signing
2015-08-11 03:38:08,572 - 1f9c111562fcf8419e3f4d5b48f534e97f2d47fc: processing build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip on https://signing4.srv.releng.scl3.mozilla.com:9120
2015-08-11 03:38:08,588 - 1f9c111562fcf8419e3f4d5b48f534e97f2d47fc: OK

# new data for *SUMS files
## zip
$ for hash in md5 sha1 sha512; do openssl dgst -${hash} build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip; done
MD5(build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip)= 28b5c50ed0242a5eeb1e4a7fffee83a3
SHA1(build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip)= 1f9c111562fcf8419e3f4d5b48f534e97f2d47fc
SHA512(build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip)= a840252739bc5ad31e50c8c553d68608cbcec8e63e40820e4a31c22cc3cc6196ece998cd312d97b97dee8af187530ca0c0ab1024d768173b6816e00c46d42188

## zip.asc
$ for hash in md5 sha1 sha512; do openssl dgst -${hash} build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip.asc; done
MD5(build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip.asc)= 19534bcbbbaa1e489a2784be1020bb9c
SHA1(build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip.asc)= 815a71851fa6607bd694a3f318835b1e8f63c03d
SHA512(build5/sdk/xulrunner-40.0.en-US.win32.sdk.zip.asc)= f3085bd8217525b3991e6d3fe4661d8b4c41e0ff1935fd9e3e6ddb63ded38701d50be6f8772842d17113fb40e3558c39467f7aac3c076d241619d9b7ebe826a3

# grab old checksums files
wget -O build5/MD5SUMS \
  https://ftp.mozilla.org/pub/mozilla.org/xulrunner/candidates/40.0-candidates/build5/MD5SUMS
wget -O build5/SHA1SUMS \
  https://ftp.mozilla.org/pub/mozilla.org/xulrunner/candidates/40.0-candidates/build5/SHA1SUMS
wget -O build5/SHA512SUMS \
  https://ftp.mozilla.org/pub/mozilla.org/xulrunner/candidates/40.0-candidates/build5/SHA512SUMS
# update sums files by hand

# sign checksums files
$ for f in build5/*SUMS; do ${MOZ_SIGN_CMD} -f gpg -o ${f}.asc ${f}; done

# upload, tested with -n first
$ rsync -av -e "ssh -i $HOME/.ssh/xrbld_dsa" build5/ \
  xrbld@stage.mozilla.org:/pub/mozilla.org/xulrunner/candidates/40.0-candidates/build5/
building file list ... done
./
MD5SUMS
MD5SUMS.asc
SHA1SUMS
SHA1SUMS.asc
SHA512SUMS
SHA512SUMS.asc
sdk/
sdk/xulrunner-40.0.en-US.win32.sdk.zip.asc

sent 5707 bytes  received 336 bytes  12086.00 bytes/sec
total size is 55180845  speedup is 9131.37

# ssh into xrbld@stage to push into release, also checked with rsync -n first
$ rsync -av --link-dest=/pub/mozilla.org/xulrunner/candidates/40.0-candidates/build5/ --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=*.checksums.asc --exclude=logs --exclude=jsshell* --exclude=host --exclude=*.json --exclude=*mar-tools* --exclude=gecko-unsigned-unaligned.apk --exclude=robocop.apk /pub/mozilla.org/xulrunner/candidates/40.0-candidates/build5/ /pub/mozilla.org/xulrunner/releases/40.0/
./
MD5SUMS
MD5SUMS.asc
SHA1SUMS
SHA1SUMS.asc
SHA512SUMS
SHA512SUMS.asc
sdk/
sdk/xulrunner-40.0.en-US.win32.sdk.zip
sdk/xulrunner-40.0.en-US.win32.sdk.zip.asc

sent 55188552 bytes  received 173 bytes  36792483.33 bytes/sec
total size is 1694285152  speedup is 30.70

Fennec

Build 1

~~(RelMan) Ship the l10n milestone (build1 only)~~

(RelMan) Submit to Ship It

Push to mirrors

~~Post-release tasks~~

Releases/Firefox 40.0/BuildNotes

Contents

Firefox

Build 1

Issues

Intermittent/silly things

Release channel updates builder failure

beta update verify failures in chunks 3 and 4 (all platforms

Mac release update verify failures

Build 2

Issues

Updates builder losing race condition

Manually updated balrog blob for 39.0 on beta channels

Beta final verification failures

Windows update verify failure

Build 3

Update patcher config before the release started

Build 4

Issues

Build 5

Issues

XULRunner repack

Fennec

Build 1

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

How to Contribute

MozillaWiki

Around Mozilla

Tools