Releases/Fennec 4.0b2/BuildNotes
Contents
- 1 Build Engineers
- 2 Bugs encountered
- 3 Signed-off Revision(s)
- 4 Tags
- 5 Build data
- 6 Notes
- 6.1 Build 1
- 6.2 Build 2
- 6.3 Build 3
- 6.4 Build 4
Build Engineers
lsblakk, aki - bug 607238
Bugs encountered
Signed-off Revision(s)
Build # | mozilla-central | mobile-browser |
build 1 | b0132c16f574 | 3f6795993406 |
build 2 | 57bc64a3949f | 9218ed68347b |
build 3 | ec7521865660 | a73da861034a |
Tags
On mozilla-central:
Build # | Tag | Moz Changeset | Mobile Changeset |
build 1 | GECKO20b7pre_20101029_RELBRANCH | 35b148bd6ba0 | ae7a1d4dab7d |
FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD1 | def2833dd2ac | 6b7cbad98c1b | |
build 2 | FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD2 | 57bc64a3949f | 9218ed68347b |
build 3 | FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD3 | ec7521865660 | a73da861034a |
build 4 | FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD4 | ec7521865660 | a73da861034a |
Build data
Build # | Type | Build ID | Build machine |
build 1 | Android | 20101029154619 | mv-moz2-linux-ix-slave15 |
Maemo5-GTK | 20101029170031 | moz2-linux-slave42 | |
Linux | 20101029161724 | moz2-linux-slave43 | |
Mac | 20101029165620 | moz2-darwin9-slave35 | |
Windows | failed compile (3x) | mw32-ix-slave07, mw32-ix-slave20, mw32-ix-slave07 | |
build 2 | Android | 20101101113708 | moz2-linux-slave36 |
Maemo5-GTK | 20101101122918 | mv-moz2-linux-ix-slave18 | |
Linux | 20101101104511 | mv-moz2-linux-ix-slave13 | |
Mac | 20101101122921 | moz2-darwin9-slave35 | |
Windows | 20101101162647 | mw32-ix-slave07 | |
build 3 | Android | 20101102172540 | mv-moz2-linux-ix-slave06 |
Maemo5-GTK | 20101102163940 | mv-moz2-linux-ix-slave18 | |
Linux | 20101102172628 | moz2-linux-slave40 | |
Mac | 20101102183947 | moz2-darwin9-slave35 | |
Windows | 20101102184204 | mw32-ix-slave07 | |
build 4 | Android | 20101103123628 | mv-moz2-linux-ix-slave16 |
Maemo5-GTK | 20101103114710 | mv-moz2-linux-ix-slave18 | |
Linux | 20101103132008 | moz2-linux-slave42 | |
Mac | 20101103134910 | moz2-darwin9-slave35 | |
Windows | 20101103122720 | mw32-ix-slave07 |
Notes
Build 1
- Landed debsign bumps - this has to happen before tagging starts
- Landed configs
- Reconfig master
- Set pm02:8010 to clobber
- removed mobile_source from slaves that had done this step in October
- Notify #developers and #mobile that tree is closing
- Sent release-drivers email that tree is closing for tagging
- Closed tree
- Sendchange:
buildbot sendchange --username lsblakk --master=localhost:9010 --branch=mobile-browser "Fennec 4.0b2 build1" go
Then I tagged buildbot-configs, buildbotcustom, tools, and mozharness with FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD1
Tag
Completed without problem.
Bumped the version in mobile-browser to 4.0b3 on pm02 as ffxbld:
hg push -e 'ssh -l ffxbld -i ~/.ssh/ffxbld_dsa' -f ssh://hg.mozilla.org/mobile-browser
Source
Source failed in create_bundle (timeout)
- cleaned out mobile_source from moz2-linux-slave45
- kicked off a rebuild from builder page
Build/Repack
mobile_win32-i686_desktop_build
- failed hg pull due to checkout fail from hg.mozilla.org on mw32-ix-slave07
- hit rebuild on builder page (don't need to clobber on slave because that is part of this builder's steps), got mw32-ix-slave20
- second build failed compile
- no obvious reasons why this failed (there is space on slave, it had recently done a nightly successfully)
- force rebuild a third time, got mw32-ix-slave07, still failed compile so filed bug bug 608456
mobile_linux-i686_desktop_build
- finished no problems
mobile_macosx-i686_desktop_build
- no problems
android-r7
- Android-r7 finished with warnings because upload complete snippet failed - this is part of bug bug 608430 along with the moving of unsigned builds (see Signing step)
maemo5-gtk
- build complete no problems (just slow on a vm)
Sign Android
21G available on keymaster d:
Moved the android builds to unsigned (this will be addressed in bug 608430)
# ffxbld@staging cd /pub/mozilla.org/mobile/candidates/4.0b2-candidates/build1 mkdir unsigned mv android-r7 unsigned/ mkdir android-r7
Signing improvement bug filed bug 608432
# on keymaster in ~/signing-work mkdir fennec-4.0b2 cd fennec-4.0b2/ # we signed both en-US and multi since no one was sure in #mobile which to do # need sep dirs since they are both named gecko-unsigned-unaligned.apk mkdir -p build1/en-US mkdir build1/multi # get en-US apk cd build1/en-US/ wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build1/unsigned/android-r7/en-US/gecko-unsigned-unaligned.apk # get multi apk cd ../multi/ wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build1/unsigned/android-r7/gecko-unsigned-unaligned.apk # put the signing script in the ~/signing-work/fennec-4.0b2/ dir cp ~/hg-tools/release/signing/* . # copy the en-US apk up to pwd for signing -- there is a bug on this (608432 ), it's because sign_android.sh relies on mozpass.py and that has to be in the same dir as .apk to work cp build1/en-US/gecko-unsigned-unaligned.apk . ./sign_android.sh * if any problems with signing, remove gecko-unaligned.apk and start again # signing successful? copy back to the en-US mv gecko-unaligned.apk build1/en-US/ # rename the fennec.apk, then move mv fennec.apk fennec-4.0b2.en-US.eabi-arm.apk mv fennec-4.0b2.en-US.eabi-arm.apk build1/en-US/ # clean up any left overs rm *.apk # repeat for multi cp build1/multi/gecko-unsigned-unaligned.apk . ./sign_android.sh mv gecko-unaligned.apk build1/multi/ mv fennec.apk fennec-4.0b2.multi.eabi-arm.apk mv fennec-4.0b2.multi.eabi-arm.apk build1/multi/ rm *.apk # upload to stage cd build1/ scp -i ~/.ssh/ffxbld_dsa -r * ffxbld@stage.mozilla.org:/home/ftp/pub/mobile/candidates/4.0b2-candidates/build1/android-r7
Update Android Snippets
update 4.0b1 snippets, look for the newest files
# as cltbld@aus2-staging cd /opt/aus2/incoming/3/Fennec find . -mtime -1 . ./4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US ./4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test ./4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test/complete.txt ./4.0b2 ./4.0b2/Android_arm-eabi-gcc3 ./4.0b2/Android_arm-eabi-gcc3/20101029154619 ./4.0b2/Android_arm-eabi-gcc3/20101029154619/en-US ./4.0b2/Android_arm-eabi-gcc3/20101029154619/en-US/beta-cck-test ./4.0b2/Android_arm-eabi-gcc3/20101029154619/en-US/beta-cck-test/complete.txt ./4.0b2/Android_arm-eabi-gcc3/20101029154619/en-US/beta-cck-test/partial.txt # edited 4.0b1 complete.txt to point at multi by changing vim 4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test/complete.txt url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build1/android-r7/multi/fennec-4.0b2.multi.eabi-arm.apk hashValue=fbdbd0668c36dd202b7ae2187acd67e6177831f1e25e64066e85d191ee9fbc138d68185066fbfb39e93c602483111024ba80d6e6b312bff6f1bfda97b3414b36 size=14488447 # if the en-US is preferred, the details are: url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build1/android-r7/en-US/fennec-4.0b2.en-US.eabi-arm.apk hashValue=0a32c7e7faf709480b9b278954df2731d268dbae19ca5f148323dc64df77b788aed5b52ca344f478ddf8967c72e4f831365554aa2a79214af4facb557fea7ae4 size=12239733
Deb Repos
On production-mobile-master force build on deb sign release builder
First run of deb sign release on production-mobile-master failed because of config dir name conflict between mozharness/configs and buildbot-configs being cloned as configs. Aki put up a patch and bumped up the mozharness PRODUCTION tag, and then another force build could be triggered.
Second run, all passed.
Send email to r-d about signed builds being available, hand off to QA. (except win32 build because of bug 608456
At this point a respin was called via Release-Drivers email
Build 2
- Set pm02:8010 clobber
- rm -rf mobile_source from mv-moz2-linux-ix-slave12, moz2-linux-slave45
- rm -rf *release from /scratchbox/users/cltbld/home/cltbld/build on slaves that did l10n repacks
- landed config bumps
- landed debsign bumps and moved PRODUCTION tag
- notify #developers and #mobile that tree is closing for tagging
- email to r-d that tree is closing for tagging
- update pm02:8010 configs
- reconfig master
- Start automation with:
buildbot sendchange --username lsblakk --master=localhost:9010 --branch=mobile-browser "Fennec 4.0b2 build2" doit
Tag
mv-moz2-linux-ix-slave18
No problems, re-opened tree.
Tagged buildbotcustom, buildbot-configs, tools, mozharness with FENNEC_4_0b2_BUILD2, and bumped up FENNEC_4_0b2_RELEASE
Source
moz2-linux-slave45
no problems.
Build/Repack
mobile_win32-i686_desktop_build
mw32-ix-slave07
- First run through failed with same tests linkage error in compile as build1 so landed a change to mozconfig to explicitly disable-tests and re-build from buildbot page.
mw32-ix-slave07
- Failed package-tests because tests were disabled!
- landed a dir name shortening and re-enabled test patch, reconfig'd pm02:8010, and kicked it off again
mw32-ix-slave07
- third time's a charm. successful build and upload - closed bug 608456
mobile_linux-i686_desktop_build
mv-moz2-linux-ix-slave13
no problems
mobile_macosx-i686_desktop_build
moz2-darwin9-slave35
no problems.
android-r7
moz2-linux-slave36
- Android-r7 finished with warnings because upload complete snippet failed - this is part of bug bug 608430 along with the moving of unsigned builds (see Signing step)
maemo5-gtk
moz2-linux-slave28
- Failed to upload en-US to stage because of DNS bustage, forced rebuild in the hopes that it could make it through without multi-locale issues.
mv-moz2-linux-ix-slave18
- successful
repack
had to manually trigger all the locales with en_revision, l10n_revision, and locale property settings which were all successful as well. checked on stage to be sure that the count of locale dirs matched build1 (both 41)
Sign Android
20G available on keymaster d:
Moved the android builds to unsigned (this will be addressed in bug 608430)
# ffxbld@staging cd /pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2 mkdir unsigned mv android-r7 unsigned/ mkdir android-r7
# on keymaster in ~/signing-work cd fennec-4.0b2/ # make dirs for both en-US and multi mkdir -p build2/en-US mkdir build2/multi # get en-US apk cd build1/en-US/ wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2/unsigned/android-r7/en-US/gecko-unsigned-unaligned.apk # get multi apk cd ../multi/ wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2/unsigned/android-r7/gecko-unsigned-unaligned.apk # copy the en-US apk up to pwd for signing cp build1/en-US/gecko-unsigned-unaligned.apk . ./sign_android.sh # copy back to the en-US mv *.apk build2/en-US/ # repeat for multi cp build1/multi/gecko-unsigned-unaligned.apk . ./sign_android.sh mv *.apk build2/multi/ # upload to stage cd build1/ scp -i ~/.ssh/ffxbld_dsa -r * ffxbld@stage.mozilla.org:/home/ftp/pub/mobile/candidates/4.0b2-candidates/build2/android-r7
# ffxbld@stage # rename en-US and multi fennec apk: mv fennec.apk fennec-4.0b2.en-US.eabi-arm.apk mv fennec.apk fennec-4.0b2.multi.eabi-arm.apk
Update Android Snippets
Get the size and hash for the apk
# ffxbld@stage cd /pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2/android-r7 ls -la ./* # gets file size # sha512 hash openssl dgst -sha512 {path to each fennec apk} en-US: url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2/android-r7/en-US/fennec-4.0b2.en-US.eabi-arm.apk size=12239829 hashValue=b51c3828d840518832d86bf74551007a55a72b52f8a52a2a890683c13b49d5b52b91900c3fa39e8b553b6d15425cf75442fa14db28bab1406f0809fbaf23008f multi: url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build2/android-r7/multi/fennec-4.0b2.multi.eabi-arm.apk size=14488519 hashValue=aea2895f48c3c44ac119a07b9e82f46d34163a140c28364b156be7a4daaa86ded55ada5452138f2bc2c56e5928953f440c3b47c97a31748bf3950ef716d36b49 # edited 4.0b1 snippet to add /multi to stage path, correct size and hashValue (using multi because that's what we are shipping for 4.0b2) vim 4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test/complete.txt
Deb Repos
Forced build on pmm, didn't reconfig since only changes were config bumps in mozharness.
No problems.
Update testing
- did a symlink for 4.0b2 build1 -> 4.0b2 build2 on aus2-staging:
# in /opt/aus2/incoming/3/Fennec/4.0b2/Android_arm-eabi-gcc3/20101029154619/en-US/beta-cck-test ln -s ../../../../../4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test/complete.txt complete.txt
- assigned bug 608837 to me, we'll need a better solution going forward like a latest-beta-cck-test or something
At this point I handed off to QA in an email to release-drivers.
Then we got the email that "We unfortunately found a crasher on a couple major sites in build 2, which has been fixed on trunk." and a respin was called.
Build 3
- Set pm02:8010 to clobber
- clobbered repacks from build2 slaves (all linux ix)
- removed mobile_source from build2 slaves (moz2-linux-slave45)
- Landed debsign bumps - this has to happen before tagging starts
- Landed configs
- Reconfig master
- Notify #developers and #mobile that tree is closing
- Sent release-drivers email that tree is closing for tagging
- Closed tree
- Sendchange:
buildbot sendchange --username lsblakk --master=localhost:9010 --branch=mobile-browser "Fennec 4.0b2 build3" go
Then I tagged buildbot-configs, buildbotcustom, tools, and mozharness with FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD3
Tag
No problem, re-opened tree.
Source
No Problem
Build/Repack
mobile_win32-i686_desktop_build
No Problem
mobile_linux-i686_desktop_build
No Problem
mobile_macosx-i686_desktop_build
No Problem
android-r7
build completed with the expected snippet warning, same as previous builds.
- aki moved build3/android-r7 to build3/unsigned/android-r7 and created an empty build3/android-r7 directory
maemo5-gtk
No Problem
repack
No Problem
At this point build4 respin was called because of l10n bustage in German locale, Axel provided an updated list of l10-changesets for build4.
Build 4
- Set pm02:8010 to clobber
- clobbered repacks from build2 slaves (all linux ix)
- removed mobile_source from build2 slaves (mv-moz2-linux-ix-slave21)
- Landed debsign bumps - this has to happen before tagging starts
- Landed configs
- Reconfig master
- Notify #developers and #mobile that tree is closing
- Sent release-drivers email that tree is closing for tagging
- Closed tree
- Sendchange:
buildbot sendchange --username lsblakk --master=localhost:9010 --branch=mobile-browser "Fennec 4.0b2 build4" go
Note, did this after the fact but for future builds - use https://l10n-stage-sj.mozilla.org/shipping/confirm-ship?ms=fennec4.0b2 to generate the l10n json and then be sure to 'ship it' (you must be logged in) so that Axel's records of what was actually used is up to date.
Tag
No problems.
I tagged buildbot-configs, buildbotcustom, tools, and mozharness with FENNEC_4_0b2_RELEASE, FENNEC_4_0b2_BUILD4
Source
First run on mv-mox2-ix-linux-slave21 failed because of a failed clobber (which is my fault from a bad autocomplete in csshx), I rm -rf /builds/slave/mobile_source and did a rebuild.
Second run, no problems.
Build/Repack
mobile_win32-i686_desktop_build
No problems
mobile_linux-i686_desktop_build
No problem
mobile_macosx-i686_desktop_build
No problems
android-r7
build completed with the expected snippet warning, same as previous builds.
- moved build3/android-r7 to build4/unsigned/android-r7 and created an empty build4/android-r7 directory
maemo5-gtk
No problems
repack
No problems
Sign Android
20G available on keymaster d:
# on keymaster in ~/signing-work cd fennec-4.0b2/ # we signed both en-US and multi since no one was sure in #mobile which to do # need sep dirs since they are both named gecko-unsigned-unaligned.apk mkdir -p build4/en-US mkdir build4/multi # get en-US apk cd build4/en-US/ wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build4/unsigned/android-r7/en-US/gecko-unsigned-unaligned.apk # get multi apk cd ../multi/ wget http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build4/unsigned/android-r7/gecko-unsigned-unaligned.apk # copy the en-US apk up to pwd for signing -- there is a bug on this (608432 ), it's because sign_android.sh relies on mozpass.py and that has to be in the same dir as .apk to work cp build4/en-US/gecko-unsigned-unaligned.apk . ./sign_android.sh mv fennec.apk fennec-4.0b2.en-US.eabi-arm.apk mv *apk build4/en-US/ # repeat for multi cp build1/multi/gecko-unsigned-unaligned.apk . ./sign_android.sh mv fennec.apk fennec-4.0b2.multi.eabi-arm.apk mv *.apk build4/multi # upload to stage cd build4/ scp -i ~/.ssh/ffxbld_dsa -r * ffxbld@stage.mozilla.org:/home/ftp/pub/mobile/candidates/4.0b2-candidates/build4/android-r7
Update Android Snippets
Get the size and hash for the apk
# ffxbld@stage cd /pub/mozilla.org/mobile/candidates/4.0b2-candidates/build4/android-r7 ls -la ./* # gets file size # sha512 hash openssl dgst -sha512 {path to each fennec apk} en-US: url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build4/android-r7/en-US/fennec-4.0b2.en-US.eabi-arm.apk size=12245366 hashValue=cef4095145cfc478e13ef7e2b1c330343d2900116a2eb456bd4a990851ba4c444b07359d265dd2801378cc88f9dee4d469d08a7261ef4b08d39737f210df7298 multi: url=http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/4.0b2-candidates/build4/android-r7/multi/fennec-4.0b2.multi.eabi-arm.apk size=13922106 hashValue=9f2380c46512d5fbde349e02e0b15c7bb1ac3bad9c0fc47988b0520106a77af769a5cb3c603ca6e50175a62e67d4c0c9d3ea40d42c1b998227f5364f8f0eae04 # edited 4.0b1 snippet to add correct size and hashValue for multi (what we are shipping for 4.0b2) vim 4.0b1/Android_arm-eabi-gcc3/20101006203925/en-US/beta-cck-test/complete.txt # symlinks for 4.0b2 build2,build3 -> 4.0b2 build 4 cd /opt/aus2/incoming/3/Fennec/4.0b2/Android_arm-eabi-gcc3/20101101113708/en-US/beta-cck-test rm complete.txt && ln -s /opt/aus2/incoming/3/Fennec/latest-beta-cck-test complete.txt cd /opt/aus2/incoming/3/Fennec/4.0b2/Android_arm-eabi-gcc3/20101102172540/en-US/beta-cck-test rm complete.txt && ln -s /opt/aus2/incoming/3/Fennec/latest-beta-cck-test complete.txt
Deb Repos
Force build on deb signing builder on pmm, no reconfig since nothing but bumps on debsign dirs, completed without problems.
-- QA handoff
Update testing
Push Live
Copy bits to releases/
We didn't release single locales, so this is different than the ReleaseDocs
cd /home/ftp/pub/mobile/releases mkdir 4.0b2 cd 4.0b2 cp -r ../../candidates/4.0b2-candidates/build4/*-i686 . cp -r ../../candidates/4.0b2-candidates/build4/source . cp -r ../../candidates/4.0b2-candidates/build4/android-r7/multi android-r7/multi cp -r ../../candidates/4.0b2-candidates/build4/maemo5-gtk/multi maemo5-gtk/multi (all the cp instead of rsync brought over test, crashreporter, txt, unsigned, unaligned files that were not needed and I removed them from the various dirs as needed) # symlinks for en-US because we are shipping multi and want en-US users to update to it ln -s android-r7/multi android-r7/en-US ln -s maemo5-gtk/multi maemo5-gtk/en-US # and repos mkdir -p repos/multi cp -r ../../candidates/4.0b2-candidates/repos/multi/* repos/multi ln -s repos/multi repos/en-US # softlinks for maemo ln -s repos maemo cd .. # latest- softlinks: # latest-alpha is the alpha "channel", latest-beta is the beta "channel" # latest is the release "channel" # rm latest-alpha latest-beta ln -s 4.0b2 latest-alpha ln -s 4.0b2 latest-beta # hack chinook for now... long term fix later? cd 4.0b2/repos/en-US/dists ln -s fremantle chinook
Create new .install files
nothing to be done here for this release.
Create bouncer entries
# added product Firefox-Mobile-4.0b2 android /mobile/releases/4.0b2/android-r7/multi/fennec-4.0b2.multi.eabi-arm.apk maemo /mobile/releases/maemo/multi/firefox-beta.install
Updates to beta channel
# cltbld on aus2-staging cd /opt/aus2/incoming/3/Fennec cp latest-beta-cck-test beta.tmp # edit beta.tmp to point to url=http://download.mozilla.org/?product=Firefox-Mobile-4.0b2&os=android&lang=multi # Put snippet in 4.0b1/Android_arm-eabi-gcc3/en-US/20101006203925/beta as complete.txt ln -s /opt/aus2/incoming/3/Fennec/latest-beta complete.txt
We're live, sent email to release-drivers.