Be sure to take notes on manual steps and any problems you encounter.

Checklist

Notes

tracking bug: bug 772810
buildbot master: 30

Build 1

• l10n changesets
• reserved slaves:

echo 8 > /builds/buildbot/build1/master/reserved_slaves'"

• clobbered release-comm-beta
• created candidates dir:

ssh -l tbirdbld -i ~/.ssh/tbirdbld_dsa stage.mozilla.org
version=15.0b1
product=thunderbird
cd /pub/mozilla.org/$product/nightly/
mkdir ../candidates/$version-candidates
ln -s ../candidates/$version-candidates $version-candidates
[tbirdbld@upload1.dmz.scl3 nightly]$ ls -l $version-candidates
lrwxrwxrwx 1 tbirdbld thunderbird 31 Jul 17 10:17 15.0b1-candidates -> ../candidates/15.0b1-candidates

• updated & tagged
• reconfig

[cltbld@buildbot-master30 build1]$ source bin/activate
(build1)[cltbld@buildbot-master30 build1]$ make update && make reconfig
(cd buildbotcustom && /usr/bin/hg pull && /usr/bin/hg update -C)
pulling from http://hg.mozilla.org/build/buildbotcustom
searching for changes
adding changesets
adding manifests
adding file changes
added 4 changesets with 4 changes to 2 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
(cd buildbot-configs && /usr/bin/hg pull && /usr/bin/hg update -C)
pulling from http://hg.mozilla.org/build/buildbot-configs
searching for changes
adding changesets
adding manifests
adding file changes
added 9 changesets with 9 changes to 8 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
7 files updated, 0 files merged, 0 files removed, 0 files unresolved
(cd tools && /usr/bin/hg pull && /usr/bin/hg update -C)
pulling from http://hg.mozilla.org/build/tools
searching for changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 2 files
(run 'hg update' to get a working copy)
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
(python buildbot-configs/update-master-json.py tools/buildfarm/maintenance/production-masters.json master/master_config.json)
writing new master_config.json file to master/master_config.json
cd master && /builds/buildbot/build1/bin/buildbot reconfig $PWD
sending SIGHUP to process 20790
2012-07-17 10:49:19-0700 [-] loading configuration from /builds/buildbot/build1/master/master.cfg
Never saw reconfiguration finish.
(build1)[cltbld@buildbot-master30 build1]$ cd master && /builds/buildbot/build1/bin/buildbot reconfig $PWD
sending SIGHUP to process 20790
Never saw reconfiguration finish.

• release sanity:

   PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u armenzg \
       -V 15.0b1 --build-number 1 --branch comm-beta \
       --release-config release-thunderbird-comm-beta.py --product thunderbird \
       --dryrun localhost:9001

• reset reserved:

   [cltbld@buildbot-master30 master]$ pwd -P
   /builds/buildbot/build1/master
   [cltbld@buildbot-master30 master]$ echo 0 >reserved_slaves 
   [cltbld@buildbot-master30 master]$ cat reserved_slaves
   0

• push snippets:

   $ ssh aus3-staging.mozilla.org
   Last login: Wed Jul 11 13:45:06 2012 from vpn1.dmz.releng.scl3.mozilla.com
   [hwine@dp-ausstage01 ~]$ sudo su - tbirdbld
   [tbirdbld@dp-ausstage01 ~]$ cd /opt/aus2/snippets/staging
   [tbirdbld@dp-ausstage01 staging]$ ~/bin/pushsnip Thunderbird-15.0b1-build1

• we will have to push again since we hit the trigger too early - WIP
• pushed again

   $ ssh aus3-staging.mozilla.org
   Last login: Wed Jul 11 13:45:06 2012 from vpn1.dmz.releng.scl3.mozilla.com
   [hwine@dp-ausstage01 ~]$ sudo su - tbirdbld
   [tbirdbld@dp-ausstage01 ~]$ cd /opt/aus2/snippets/staging
   [tbirdbld@dp-ausstage01 staging]$ ~/bin/pushsnip Thunderbird-15.0b1-build1

• re-triggered update verification failed jobs

XXX: check those jobs in the morning

Problems Encountered

• release sanity errors: mozconfig and L10n

2012-07-17 10:50:35,651 : WARNING : if test "$IS_NIGHTLY" != ""; then not in win32 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', '. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags', 'mk_add_options MOZ_MAKE_FLAGS=-j1']!
2012-07-17 10:50:35,651 : ERROR : found in mail/config/mozconfigs/win32/nightly but not in mail/config/mozconfigs/win32/release: if test "$IS_NIGHTLY" != ""; then
2012-07-17 10:50:35,651 : WARNING : ac_add_options --disable-auto-deps not in win32 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', '. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags', 'mk_add_options MOZ_MAKE_FLAGS=-j1']!
2012-07-17 10:50:35,652 : ERROR : found in mail/config/mozconfigs/win32/nightly but not in mail/config/mozconfigs/win32/release: ac_add_options --disable-auto-deps
2012-07-17 10:50:35,652 : WARNING : fi not in win32 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', '. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags', 'mk_add_options MOZ_MAKE_FLAGS=-j1']!
2012-07-17 10:50:35,652 : ERROR : found in mail/config/mozconfigs/win32/nightly but not in mail/config/mozconfigs/win32/release: fi
2012-07-17 10:50:37,865 : ERROR : Error verifying mozconfigs

2012-07-17 10:51:04,515 : ERROR : l10n_changesets and shipped_locales differ on locales: ['el', 'zh-TW', 'ja-JP-mac', 'ta-LK', 'vi', 'ca', 'it', 'cs', 'pa-IN', 'ar', 'bn-BD', 'nb-NO', 'eu', 'et', 'gl', 'es-ES', 'id', 'en-GB', 'ru', 'sr', 'nl', 'sl', 'tr', 'lt', 'rm', 'ro', 'is', 'pl', 'be', 'fr', 'bg', 'sv-SE', 'ast', 'hr', 'de', 'da', 'pt-BR', 'br', 'fi', 'hu', 'ja', 'he', 'hy-AM', 'pt-PT', 'es-AR', 'sq', 'ko', 'nn-NO', 'ga-IE', 'sk', 'zh-CN', 'si', 'gd', 'uk', 'fy-NL']

• • bhearsum told me I can ignore the mozconfig errors and file a follow up bug (used "-m" to skip it)
  • shipped-locales had been migrated only with "en-US" in it
  • I bumped the comm-beta revision and re-tagged
• One L10n repack failed

command: START
command: hg update -C -r THUNDERBIRD_15_0b1_RELEASE
command: cwd: mozilla-beta/ru
command: output:
abort: data/.hgtags.i@b44e93cb42b2: no match found!
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-lnx-rpk-5/scripts/scripts/l10n/../../lib/python/util/commands.py", line 42, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python-2.5.1/lib/python2.5/subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['hg', 'update', '-C', '-r', 'THUNDERBIRD_15_0b1_RELEASE']' returned non-zero exit status 255
command: END (0.37s elapsed)

• • re-trigger "ru" with standalone repack (script_repo_revision: THUNDERBIRD_15_0b1_RELEASE, release_config: mozilla/release-thunderbird-comm-beta.py and locale: ru)
  • trigger "repack complete"
• I pushed the snippets by mistake early, to shut off updates did:

chmod 700 /opt/aus2/incoming/3/Thunderbird

• we had to use the full back up nightly of the snippets since the incremental misses old releases, this is the short version of how we fixed them up

cd /opt/aus2/snippets/backup
mkdir 15.0b1-restore2 && cd mkdir 15.0b1-restore2
tar --exclude=Firefox -jvxf ../20120718-nightly-1.tar.bz2
rsync -av --delete --exclude=1\.5* --exclude=2.0* --exclude=3.0* Thunderbird/ /opt/aus2/incoming/3/Thunderbird/ > /home/tbirdbld/changes.txt
cd /opt/aus2/snippets/staging/
~/bin/pushsnip Thunderbird-15.0b1-build1-test

We originally tried to restore from the backupsnip that was run prior to the 15.0b1 test snippets were pushed, but that ended up deleting a bunch of snippets that weren't in it due to a bad rsync. Due to this, we ended up using the full backup to restore as described above. There was also a lot of "rsync -n" commands run to verify things prior to doing them for real, not mentioned here for the sake of brevity.