Thunderbird 2.0.0.14:BuildNotes

Build Engineers

nthomas, rick

Version/config bump bug

bug 426259 Morphed from the cancelled Tb2.0.0.13 build

Bonsai queries

Checking for no changes on the rel branch since Fx20014

• cvsroot
• l10n

Tags

Using the GECKO181_20080311_RELBRANCH.

NB: Talkback not listed, we're using pre-built binaries (bug 403701). It was still tagged because this change is very new, and the automation needs remedial lessons.

Build data

See page history for BuildIDs of prior RCs

Automation

Bootstrap tag: RELEASE_AUTOMATION_M8_1

Setup before starting:

• used existing version bump bug to attached diff for tb-moz18-bootstrap.cfg
• revert mozilla/config/milestone.txt and mozilla/client.mk on the rel branch so that Tagging can substitute cleanly (FIXME: need bug)
• ensure that machines have enough resources (bug 393274)
  • On production-1.8-master, remove private and public areas:
    • rm -rf /data/cltbld/firefox-* (check with folks responsible for the releases being rm'd before removing ... in this case it was nthomas and bhearsum)
    • rm -rf /builds/tags/* - did not do this, plenty of space (and handy to have copies of what was tagged to work around CVS's lack of atomicity of tagging)
    • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
    • pull in 2.0.0.12 (previous release) files with rsync -av stage.mozilla.org:/home/ftp/pub/thunderbird/nightly/2.0.0.12-candidates /data/ftp/pub/thunderbird/nightly/
  • On the build machines (see table above), checked that there was sufficient working space (~ 1GB), and stopped tinderbox. Also verified there is space available on bm-xserve04 for l10n-verify.
  • On prometheus-vm, checked that there was space for another update run (4GB)

The following bits, to the end of this Automation section, still need to be done. Delayed due to ssh key issues on the various tinderbox clients. Will do the Tag and Source steps tho. 2008-04-17.

• on each machine, setup release automation using the tag above

mkdir -p /builds/logs
cd /builds/release   # or /cygdrive/X/builds/release
cvs -nq up -r RELEASE_AUTOMATION_M8_1
# check for merges (M) or conflicts (C); if any found, check for local changes
cvs -q up -r RELEASE_AUTOMATION_M8_1
cd configs
cvs up -A     # in case we need to make changes later
cd ..
rm bootstrap.cfg logs
ln -s configs/tb-moz18-bootstrap.cfg bootstrap.cfg
mkdir -p old-logs/logs.tb20014/rc1
ln -s old-logs/logs.tb20014/rc1 logs

make test

Some machines don't have the perl module Config::General so comment out

use Config::General;

at the top of Bootstrap/Step/PatcherConfig.pm. Only need this on prometheus-vm anyway, which has it.

Tag

On production-1.8-master in /builds/release:

./release -o Tag 2>&1 | tee logs/release-Tag-RC1.log

Source

On production-1.8-master in /builds/release:

./release -o Source 2>&1 | tee logs/release-Source-RC1.log

Failed with message

build-console.build.mozilla.org - Step Source died: No such config variable:
externalStagingUser at Bootstrap/Config.pm line 93.

Appears to be related to this configuration change which removed the externalStagingUser variable.

However, tools/release/Bootstrap/Util.pm on the RELEASE_AUTOMATION_M8_1 tag still references that variable. There were changes made on the trunk for Util.pm that removed those references.

Resolved with this commit which restored the external* variables.

This change will need to be undone once the thunderbird 2.0.0.14 release is complete.

The Source step was completed by doing the following on production-1.8-master

 cd /builds/release/configs
 cvs up
 cd ..
  ./release -o Source -p 2>&1 | tee logs/release-Source-RC1-Push.log

This completed successfully.

Build

There ssh key issues to workaround - the nightly builders were converted to tbirdbld but releases still use cltbld. On each box,

• shut down tinderbox builds and add a comment on the Mozilla1.8 tree
• in ~/.ssh

rm id_dsa; ln -s cltbld_dsa id_dsa
rm id_dsa.pub; ln -s cltbld_dsa.pub id_dsa.pub

• update /builds/tinderbox/Tb-Mozilla1.8-Release/tinderbox-configs/CVS/Root so that it uses the cltbld account (just for this release, we'll leave it on cltbld)
• update Bootstrap as in the Automation section

On Windows, check that lines 43-49 of Build.pm are commented out, since Cygwin complains "Permissied denied" trying to execute those lines. The disks are already mounted in binmode.

Then start the build, by changing directory to /builds/release (prefix with /cygdrive/e/ on Windows) and

./release -o TinderConfig 2>&1 | tee logs/release-TinderConfig.log
./release -o Build 2>&1 | tee logs/release-Build.log

After the build is done, and if the tinderbox is not needed for l10n Repacks (eg patrocles, crazyhorse),

• reverse the changes in ~/.ssh. NB: bm-xserve02 uses ffxbld, others use tbirdbld. Eg

rm id_dsa; ln -s tbirdbld_dsa id_dsa
rm id_dsa.pub; ln -s tbirdbld_dsa.pub id_dsa.pub

• restart tinderbox

cd /builds/tinderbox
./tinderbox multi start

NB: The Win32 build died at the very end of Build::Execute in PublishTalkbackSymbols() - Python is not installed on patrocles so the call to symbolstore.py fails. The end result is that we don't get T'bird 2.0.0.14 on the symbol server.

Repack

On the tinderboxes in the table above

• stop tinderbox
• do the ssh key changes as in the Build section
• update Bootstrap as in the Automation section

./release -o Repack -e 2>&1 | tee logs/release-Repack-Execute.log
./release -o Repack -p 2>&1 | tee logs/release-Repack-Push.log
./release -o Repack -a 2>&1 | tee logs/release-Repack-Announce.log

file has vanished: "/home/ftp/pub/thunderbird/nightly/2.0.0.14-candidates/rc1/.thunderbird-2.0.0.14.pa-IN.linux-i686.complete.mar.KwvZoP"

rsync: link_stat "/data/cltbld/thunderbird-2.0.0.12/." failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at main.c(702)

Analysis/Fixes (nthomas):

• Linux
  • Execute uploaded to cltbld@production-1.8-master.build.mozilla.org:/home/ftp/pub/thunderbird/nightly/2008-04-21-13-thunderbird2.0.0.14-l10n/. ie not .../2008/04/2008-04-21-13-thunderbird2.0.0.14-l10n/. This is because the tinderbox code was not symlinked properly in karma:/builds/tinderbox/Tb-Mozilla1.8-l10n-Release - effectively it was using older code. This would have been a long standing issue.
  • Fixed the builds on the master by moving the linux files into .../2008/04/2008-04-21-13-thunderbird2.0.0.14-l10n/ and replacing .../thunderbird/nightly/2008-04-21-13-thunderbird2.0.0.14-l10n/ with a symlink into 2008/04/.
  • Fixed tinderbox setup (backup in bak-20080422)
  • To verify all files are present, ran os=linux; ~/check_locales.sh /home/ftp/pub/thunderbird/nightly/2.0.0.14-candidates/rc1/ ${os} ~/thunderbird-2.0.0.14/batch-source/rc1/mozilla/mail/locales/shipped-locales on stage.m.o. Got no complaints about missing files.
  • Fixed up ~/.ssh symlinks (using ffxbld key), and restarted tinderbox
• Mac
  • Execute uploaded to the correct location (.../2008/04/2008-04-21-13-thunderbird2.0.0.14-l10n/)
  • Push failed to move any mac files into the candidates direcotry because it was using /home/ftp/pub/thunderbird/nightly/2008-04-21-13-thunderbird2.0.0.14-l10n/ as the source. So it got the Linux dir instead of a symlink to the mac dir.
  • The error file has vanished:"/home/ftp/pub/thunderbird/nightly/2.0.0.14-candidates/rc1/.thunderbird-2.0.0.14.pa-IN.linux-i686.complete.mar.KwvZoP" comes from the Linux Push running simultaneously with the Mac one (ie Linux was pushing into the candidates dir as Mac was syncing with stage, that's a rsync temp file). Ignoring this as harmless.
  • Ran Push again, which copied the Mac files properly, but got rsync error: some files could not be transferred (code 23) at main.c(702) when doing SyncToStage. Should have applied the .../thunderbird/nightly/2008-04-21-13-thunderbird2.0.0.14-l10n fix on stage too.
  • After fixing stage, ran rsync -av /home/ftp/pub/thunderbird/nightly/ cltbld@stage-old.mozilla.org:/home/ftp/pub/thunderbird/nightly/ on the master, and only the symlink was copied over.
  • the equivlanet check_locales.sh now says all mac files are present on stage
  • Fixed up ~/.ssh symlinks (using ffxbld key), and restarted tinderbox
• Windows
  • Execute uploaded to the correct location
  • Push copied the bits to /home/ftp/pub/thunderbird/nightly//2.0.0.12-candidates/rc1/unsigned/. Turns out tb-moz18-bootstrap.cfg was on the RELEASE_AUTOMATION_M8_1 tag rather than the tip.
  • On the master, cleaned out /home/ftp/pub/thunderbird/nightly/2.0.0.12-candidates/rc1/unsigned of 2.0.0.14 files, including all of windows-xpi/.
  • Did the same on stage, and restored the correct xpi's from 2008/02/2008-02-13-09-thunderbird2.0.0.12-l10n/windows-xpi
  • Updated the bootstrap configs/ dir to tip, moved logs/* to logs/attempt1/, and reran Repack Execute, Push, and Announce
  • Fixed up ~/.ssh symlinks (using ffxbld key), and restarted tinderbox

Sign

• Signing doc

l10n-Verify

Ran this on bm-xserve05 (via vnc)

./release -o Repack -v 2>&1 | tee logs/release-Repack-Verify.log

This was run a first time simply by ssh'ing into bm-xserve05. A screen session was fired up and the above Repack command was run. That failed with various hdiutil errors.

mounting thunderbird-2.0.0.14-rc1/thunderbird-2.0.0.14.en-US.mac.dmg
hdiutil: attach failed - Device not configured

It turns out that hdiutil will not work properly under a screen session.

Update Generation

Generated on production-prometheus-vm,

./release -o PatcherConfig 2>&1 | tee logs/release-PatcherConfig.log
./release -o Updates -e 2>&1 | tee logs/release-Updates-Execute.log
./release -o Updates -p 2>&1 | tee logs/release-Updates-Push.log

Push dirs were 20080423-Thunderbird-2.0.0.14 (& -test & -beta)

Update Verify

Run on production-pacifica-vm (windows), production-prometheus-vm (linux) and bm-xserve05 (mac)

./release -o Updates -v 2>&1 | tee logs/release-Updates-Verify.log

patrocles doesn't have wget, so used production-pacifica-vm instead - checked that the win32_logDir is /cygdrive/c/... rather than /cygdrive/e/... in the bootstrap config.

Push Updates to Beta Channel

While waiting for QA to give "ok", on aus2-staging do:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update . 
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
 $ time ~/bin/backupsnip 20080423-Thunderbird-2.0.0.14-beta
   real    44m17.226s
   user    0m25.183s
   sys     0m36.805s

After QA gives "ok" to push to beta channel, on aus2-staging do:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update mozilla/tools/release/bin/ 
 $ cd /opt/aus2/snippets/staging
 $ time ~/bin/pushsnip 20080423-Thunderbird-2.0.0.14-beta
   real    0m39.156s
   user    0m0.050s
   sys     0m3.772s

Stage

On production-1.8-master

./release -o Stage 2>&1 | tee logs/release-Stage.log

Sign Installers

• NOTE - production-1.8-master is not used from here on
• pull stage:/data/cltbld/thunderbird-2.0.0.14/batch1/stage-signed/ to signing machine
• follow installer-signing-instructions here
• push signed bits back to same location on stage
• complete stage-merged:

# on stage
cd /data/cltbld/thunderbird-2.0.0.14/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage as cltbld
cd /data/cltbld/thunderbird-2.0.0.12/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the detached sigs and the two SUM files respectively)

chown -R cltbld:thunderbird .
chmod 644 *SUMS

Release

• get the last formal "go" from QA/Dev/website/IT/release-drivers
• push the stage-merged directory to the releases area (had to create target dir as root first):

# on stage
rsync -av /data/cltbld/thunderbird-2.0.0.14/stage-merged/ /home/ftp/pub/thunderbird/releases/2.0.0.14/

• After pushing live, move forward the "latest" and "latest-2.0" symlinks.

 cd /home/ftp/pub/thunderbird/releases/
 rm latest
 ln -s 2.0.0.14 latest
 rm latest-2.0
 ln -s 2.0.0.14 latest-2.0

• Update /etc/rsyncd-mozilla-current.exclude on surf and dm-stage02 - not critical that this happens immediately.

Update Bouncer

• Add entries to download.mozilla.org

Final Verification

Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release/{common,updates}
cd mozilla/testing/release/updates
cat moz18-thunderbird-*.cfg | sed 's/betatest/releasetest/' > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

NB: Hit an issue with 2.0b1 and b2 locales having releasetest snippets pointing at download2.mozilla.org. That's a leftover from a bouncer migration that got reverted, and doesn't respond any more. Commented out the 6 lines for those versions since there shouldn't be anything on releasetest anyways.

Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20080423-Thunderbird-2.0.0.14
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080423-Thunderbird-2.0.0.14-test/$a");'

Push Updates to Release Channel

While waiting for QA to give "ok", on aus2-staging do:

 # put snippets on release
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update . 
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
 $ time ~/bin/backupsnip 20080423-Thunderbird-2.0.0.14
   real    20m9.742s
   user    0m24.242s
   sys     0m28.175s

After QA gives "ok" to push to release channel, on aus2-staging do:

 # put snippets on release
 $ sudo su - cltbld
 $ cd /opt/aus2/snippets/staging
 $ time ~/bin/pushsnip 20080423-Thunderbird-2.0.0.14