Releases/Firefox 18.0/BuildNotes
Contents
Build 1
Checklist
-
Ship the l10n milestone DESKTOP MOBILE RELEASE BETA - Land patches and reconfigDESKTOP MOBILE RELEASE BETA ESR
- Start the automation DESKTOP MOBILE RELEASE BETA ESR
- Sign Android builds MOBILE RELEASE BETA
- Push to mirrors DESKTOP RELEASE
- Publish Fennec MOBILE RELEASE BETA NB: Push armv6 apk as well, like we do for betas
- Run pushsnip DESKTOP RELEASE BETA ESR
- Update bouncer links DESKTOP BETA RELEASE
- Update XULRunner wiki pages DESKTOP RELEASE
- Post-release tasks DESKTOP RELEASE BETA ESR
- Get the Browser Choice website updated DESKTOP RELEASE
bugs hit
- bug 826825 Remove mn, sw during beta-release migration for Firefox 18
- bug 827224 No multi locale builds for Firefox 18.0
- bug 827226 Android signing should exit if downloads fail
- bug 830722 xulrunner src and md5checksums are missing in FF 18.0
Notes
Tracking bug 810325 release engineer kmoir
Release was submitted using release-runner
bug 826825 Remove mn, sw during beta-release migration for Firefox 18
A bug wasn't opened to remove the mn and sw locales for the release, so I removed them manually in the changesets submitted to the release runner. However, they still existed in https://hg.mozilla.org/releases/mozilla-release/file/3a7b66445659/browser/locales/shipped-locales so relman had to fix this and issue a new changeset to us. Once this was releasesd, bhearsum retagged our repos and I reran the tagging builds that failed.
kmoir, jhopkins: so, we all need to "force build" our "source", "build", "bouncer submitter", and "xulrunner build" builders (if you have them) because buildbot schedulers are dumb. Firefox (source, build:win32,macosx64,linux,linux64, bouncer submitter, xulrunner: win32,macosx64,linux,linux64) Fennec(source, build, android, android-armv6)
xulrunner_macosx64 had a compile error, bhearsum identfied it as a known intermittent issue and reran the build
Because of rebuilding tag instead of re-doing sendchanges, all repacks are failing because "script_repo_revision" isn't set. These can be fixed by forcing the failed builders and setting a "script_repo_revision" property to the release tag (eg, FIREFOX_18_0_RELEASE). bhearsum did this for all failed repacks, except for Linux and Android.
Linux repacks failed with "cannot create executable" during configure. This is because the gcc path was incorrect. The mozconfigs were updated for official betas recently for mock slaves in bug 798361, but we missed the esr's -release dir's:
Catlee fixed this http://hg.mozilla.org/build/buildbot-configs/rev/e8b7870b1d96
The android repacks were failing because bug 814191 wasn't landed. I didn't realize that this had to be landed before the beta. I landed this patch, catlee landed the buildbot configs and retagged them and restarted the failed repacks.
catlee re-tagged buildbot-configs with:
FENNEC_18_0_RELEASE FENNEC_18_0_BUILD1 FIREFOX_18_0_RELEASE FIREFOX_18_0_BUILD1
catlee re-triggered:
Firefox 18 32/64-bit linux repacks Fennec 18 repacks
When signing the Android builds, this message appeared
5:18:53 ERROR - Unable to download android:multi unsigned apk! 15:18:53 ERROR - Unable to download android-armv6:multi unsigned apk! 15:18:53 ERROR - Downloaded 29 of 31 unsigned apks successfully. 15:18:53 INFO - Signed 29 of 29 apks successfully. 15:18:53 INFO - Copying logs to upload dir... 15:18:53 INFO - mkdir: /home/cltsign/signing-work/release/upload/logs
Don't recall that from before, not sure if this is normal.
No, it's not, nthomas opened bug 827224
Fennec multilocale
This was caused by a missing --package-multi option in the worst possible place.
Happily, mozharness allows you to run actions separately, and we don't clobber release directories by default.
- Looked at the logs; we used bld-centos6-hp-007 and 016 for android-armv6 and android, respectively.
- Disabled in slavealloc temporarily
- Backed up objdir
# armv6 cltbld@bld-centos6-hp-007 cd /builds/slave/rel-m-rel-andrd-armv6-bld/build tar czvf obj-firefox.tar.gz obj-firefox
# armv7 cltbld@bld-centos6-hp-016 cd /builds/slave/rel-m-rel-andrd-bld/build tar czvf obj-firefox.tar.gz obj-firefox
- Tried to package-multi (from log, but with actions changed to --package-multi --summary)
mock_mozilla -v -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-rel-andrd-armv6-bld --unpriv --shell '/usr/bin/env MOZ_OBJDIR="obj-firefox" PATH="/tools/buildbot/bin:/usr/local/bin:/bin:/usr/bin" SYMBOL_SERVER_USER="ffxbld" DISPLAY=":2" CCACHE_UMASK="002" SHIP_LICENSED_FONTS="1" MOZ_SYMBOLS_EXTRA_BUILDID="android-armv6" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_mob/" MOZILLA_REV="FENNEC_18_0_RELEASE" JAVA_HOME="/tools/jdk6" HG_SHARE_BASE_DIR="/builds/hg-shared" LC_ALL="C" SYMBOL_SERVER_HOST="symbols1.dmz.phx1.mozilla.com" TINDERBOX_OUTPUT="1" COMM_REV="FENNEC_18_0_RELEASE" CCACHE_DIR="/builds/ccache" CCACHE_COMPRESS="1" MOZ_UPDATE_CHANNEL="release" SYMBOL_SERVER_SSH_KEY="/home/mock_mozilla/.ssh/ffxbld_dsa" JARSIGNER="/builds/slave/rel-m-rel-andrd-armv6-bld/tools/release/signing/mozpass.py" MOZ_CRASHREPORTER_NO_REPORT="1" python mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-release_android-armv6.json --merge-locales --tag-override=FENNEC_18_0_RELEASE --package-multi --summary'
This failed out because of a missing 'aapt' in the mock env. Poked around a bit, figured I needed to recreate the mock environment for Android.
- Recreated mock env (copy-pasted from previous log)
# cltbld@bld-centos6-hp-007 and 016 sh -c 'rm -f /builds/mock_mozilla/mozilla-centos6-i386/buildroot.lock; mock_mozilla -v -r mozilla-centos6-i386 --orphanskill' mock_mozilla -v -r mozilla-centos6-i386 --init mock_mozilla -r mozilla-centos6-i386 --copyin /home/cltbld/.ssh /home/mock_mozilla/.ssh mock_mozilla -v -r mozilla-centos6-i386 --cwd / --shell '/usr/bin/env chown -R mock_mozilla /home/mock_mozilla/.ssh' mock_mozilla -r mozilla-centos6-i386 --copyin /home/cltbld/.android /builds/.android mock_mozilla -v -r mozilla-centos6-i386 --cwd / --shell '/usr/bin/env chown -R mock_mozilla /builds/.android' mock_mozilla -r mozilla-centos6-i386 --copyin /home/cltbld/.mozpass.cfg /builds/.mozpass.cfg mock_mozilla -v -r mozilla-centos6-i386 --cwd / --shell '/usr/bin/env chown -R mock_mozilla /builds/.mozpass.cfg' mock_mozilla -r mozilla-centos6-i386 --copyin /home/cltbld/.hgrc /builds/.hgrc mock_mozilla -v -r mozilla-centos6-i386 --cwd / --shell '/usr/bin/env chown -R mock_mozilla /builds/.hgrc' mock_mozilla -v -r mozilla-centos6-i386 --cwd / --unpriv --shell '/usr/bin/env mkdir -p /builds/slave/rel-m-rel-andrd-bld/build' mock_mozilla -v -r mozilla-centos6-i386 --install autoconf213 mozilla-python27-mercurial ccache android-sdk15 android-sdk16 android-ndk5 android-ndk8 zip java-1.6.0-openjdk-devel zlib-devel glibc-static openssh-clients mpfr wget
- Packaged multi
# android-armv6 cltbld@bld-centos6-hp-007 mock_mozilla -v -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-rel-andrd-armv6-bld --unpriv --shell '/usr/bin/env MOZ_OBJDIR="obj-firefox" PATH="/tools/buildbot/bin:/usr/local/bin:/bin:/usr/bin" SYMBOL_SERVER_USER="ffxbld" DISPLAY=":2" CCACHE_UMASK="002" SHIP_LICENSED_FONTS="1" MOZ_SYMBOLS_EXTRA_BUILDID="android-armv6" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_mob/" MOZILLA_REV="FENNEC_18_0_RELEASE" JAVA_HOME="/tools/jdk6" HG_SHARE_BASE_DIR="/builds/hg-shared" LC_ALL="C" SYMBOL_SERVER_HOST="symbols1.dmz.phx1.mozilla.com" TINDERBOX_OUTPUT="1" COMM_REV="FENNEC_18_0_RELEASE" CCACHE_DIR="/builds/ccache" CCACHE_COMPRESS="1" MOZ_UPDATE_CHANNEL="release" SYMBOL_SERVER_SSH_KEY="/home/mock_mozilla/.ssh/ffxbld_dsa" JARSIGNER="/builds/slave/rel-m-rel-andrd-armv6-bld/tools/release/signing/mozpass.py" MOZ_CRASHREPORTER_NO_REPORT="1" python mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-release_android-armv6.json --merge-locales --tag-override=FENNEC_18_0_RELEASE --package-multi --summary'
# android cltbld@bld-centos6-hp-016 mock_mozilla -v -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-rel-andrd-bld --unpriv --shell '/usr/bin/env HG_SHARE_BASE_DIR="/builds/hg-shared" JARSIGNER="/builds/slave/rel-m-rel-andrd-bld/tools/release/signing/mozpass.py" SHIP_LICENSED_FONTS="1" MOZ_CRASHREPORTER_NO_REPORT="1" SYMBOL_SERVER_HOST="symbols1.dmz.phx1.mozilla.com" CCACHE_DIR="/builds/ccache" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" MOZILLA_REV="FENNEC_18_0_RELEASE" SYMBOL_SERVER_SSH_KEY="/home/mock_mozilla/.ssh/ffxbld_dsa" PATH="/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/" MOZ_UPDATE_CHANNEL="release" TINDERBOX_OUTPUT="1" CCACHE_COMPRESS="1" COMM_REV="FENNEC_18_0_RELEASE" MOZ_OBJDIR="obj-firefox" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_mob/" LC_ALL="C" SYMBOL_SERVER_USER="ffxbld" JAVA_HOME="/tools/jdk6" DISPLAY=":2" CCACHE_UMASK="002" python mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-release_android.json --merge-locales --tag-override=FENNEC_18_0_RELEASE --package-multi --summary'
- Uploaded multi
# android-armv6 cltbld@bld-centos6-hp-007 mock_mozilla -v -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-rel-andrd-armv6-bld/build/obj-firefox --unpriv --shell '/usr/bin/env MOZ_OBJDIR="obj-firefox" PATH="/tools/buildbot/bin:/usr/local/bin:/bin:/usr/bin" SYMBOL_SERVER_USER="ffxbld" DISPLAY=":2" CCACHE_UMASK="002" SHIP_LICENSED_FONTS="1" MOZ_SYMBOLS_EXTRA_BUILDID="android-armv6" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_mob/" UPLOAD_USER="ffxbld" MOZILLA_REV="FENNEC_18_0_RELEASE" JAVA_HOME="/tools/jdk6" HG_SHARE_BASE_DIR="/builds/hg-shared" SYMBOL_SERVER_HOST="symbols1.dmz.phx1.mozilla.com" UPLOAD_SSH_KEY="~/.ssh/ffxbld_dsa" TINDERBOX_OUTPUT="1" COMM_REV="FENNEC_18_0_RELEASE" MOZ_CRASHREPORTER_NO_REPORT="1" LC_ALL="C" UPLOAD_TO_TEMP="1" MOZ_UPDATE_CHANNEL="release" SYMBOL_SERVER_SSH_KEY="/home/mock_mozilla/.ssh/ffxbld_dsa" UPLOAD_EXTRA_FILES="android-armv6_info.txt android-armv6_info.txt" UPLOAD_HOST="stage.mozilla.org" POST_UPLOAD_CMD="post_upload.py -p mobile -n 1 -v 18.0 --builddir android-armv6/multi --release-to-mobile-candidates-dir --nightly-dir=candidates" CCACHE_DIR="/builds/ccache" CCACHE_COMPRESS="1" python '"'"'/builds/slave/rel-m-rel-andrd-armv6-bld/tools/buildfarm/utils/retry.py'"'"' -s 1 -r 5 -t 3660 make upload AB_CD=multi'
# android cltbld@bld-centos6-hp-016 mock_mozilla -v -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-rel-andrd-bld/build/obj-firefox --unpriv --shell '/usr/bin/env MOZ_OBJDIR="obj-firefox" PATH="/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/" SYMBOL_SERVER_USER="ffxbld" DISPLAY=":2" CCACHE_UMASK="002" SHIP_LICENSED_FONTS="1" UPLOAD_USER="ffxbld" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_mob/" MOZILLA_REV="FENNEC_18_0_RELEASE" JAVA_HOME="/tools/jdk6" HG_SHARE_BASE_DIR="/builds/hg-shared" SYMBOL_SERVER_HOST="symbols1.dmz.phx1.mozilla.com" UPLOAD_SSH_KEY="~/.ssh/ffxbld_dsa" TINDERBOX_OUTPUT="1" COMM_REV="FENNEC_18_0_RELEASE" MOZ_CRASHREPORTER_NO_REPORT="1" LC_ALL="C" UPLOAD_TO_TEMP="1" MOZ_UPDATE_CHANNEL="release" SYMBOL_SERVER_SSH_KEY="/home/mock_mozilla/.ssh/ffxbld_dsa" UPLOAD_EXTRA_FILES="android_info.txt android_info.txt" UPLOAD_HOST="stage.mozilla.org" POST_UPLOAD_CMD="post_upload.py -p mobile -n 1 -v 18.0 --builddir android/multi --release-to-mobile-candidates-dir --nightly-dir=candidates" CCACHE_DIR="/builds/ccache" CCACHE_COMPRESS="1" python '"'"'/builds/slave/rel-m-rel-andrd-bld/tools/buildfarm/utils/retry.py'"'"' -s 1 -r 5 -t 3660 make upload AB_CD=multi'
- Re-signed multi. This skipped all single locales, but re-signed en-US as well as multi.
# cltsign@signing1 cd signing-work python2.6 mozharness/scripts/sign_android.py --config-file signing/android_mozilla-release.py --locale multi
- Reenabled slaves in slavealloc
- Rebooted slaves
- Pinged ioanachiorean in #build and sent r-d email.
Run euballot bouncer submitter
QA found that the euballot bouncer links weren't working. After some investigation we discovered that this was because the euballot bouncer entries were never added. That builder never ran because of the problems with tagging, and we forgot to trigger it. Used "force build" to start the builder and fix the problem.
had to run release-mozilla-release-xulrunner_push_to_mirrors
manually as it wasn't invoked during release automation
- bug 830722 xulrunner src and md5checksums are missing in FF 18.0
ran xulrunner source build, revealed that the xulrunner mozconfigs pointed to the old compiler path. Fixed this and reran release-mozilla-release-xulrunner_source build and md5 builds, then push to mirrors.