Firefox 3.0rc2:BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

nthomas

Bonsai queries

Last checkins:

Bugs

Tracking bug for Firefox 3.0rc2 - bug 434508

Tags

Module Tag Pull date
cvsroot/mozilla GECKO19_20080529_RELBRANCH HEAD @ 2008-05-28 17:00 PDT
FIREFOX_3_0rc2_BUILD1 GECKO19_20080529_RELBRANCH @ 2008-05-29 03:04 PDT
FIREFOX_3_0rc2_BUILD2 GECKO19_20080529_RELBRANCH @ 2008-05-30 08:23 PDT
FIREFOX_3_0rc2_RELEASE GECKO19b5_20080529_RELBRANCH @2008-05-30 08:23 PDT
l10n/l10n GECKO19_20080529_RELBRANCH HEAD @ 2008-05-28 13:37 PDT
FIREFOX_3_0rc2_BUILD1 GECKO19_20080529_RELBRANCH @ 2008-05-29 03:19 PDT
FIREFOX_3_0rc2_BUILD2 (unchanged) GECKO19_20080529_RELBRANCH @ 2008-05-30 08:35 PDT
FIREFOX_3_0rc2_RELEASE GECKO19_20080529_RELBRANCH @ 2008-05-29 03:19 PDT

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer] en-US & l10n:fx-win32-1.9-slave2
[Mac compressed] en-US & l10n:fx-mac-1.9-slave2
[Linux compressed] en-US & l10n:fx-linux-1.9-slave2

Notes

We've changed the naming scheme (bug 428063) so this sheet will track

  • 3.0 RC2 build 1
  • 3.0 RC2 build 2

etc until we publish 3.0 RC2 to the world (see the Tags section above for matching changes). We'll have a extra pages for RC3 and so on if they're required.

Build 1

Bootstrap tag: RELEASE_AUTOMATION_M9_1

NB: Italicized items were not required for this build (reason given for each).

Setup before starting:

  • compare nightly and release mozconfig/tinder-config.pl files, merge any desired changes - none this time
  • updated the bootstrap tag in master.cfg, and landed after review. Merged the change onto production-1.9-master, commented out the dep builders and reconfig'd the master - no changes
  • tagged mozilla/tools/tinderbox with RELEASE_AUTOMATION_M9_1 - we don't need the one change (bug 291167)
  • On fx-mac-1.9-slave2, fx-win32-1.9-slave2, fx-linux-1.9-slave2
    • restart linux and windows VMs to sort out any glitches from netapp problems
    • remember to use VNC if restarting mac slave
    • verify that linux has had "DISPLAY=:0 xhost +" since last reboot
    • remove the contents of /builds/verify/*
      • FIXME should be automatically backed up or removed - bug 413178
    • update tinderbox to RELEASE_AUTOMATION_M9_1
      • FIXME update should be automatic - bug 397554
      • on linux & mac
        • cd /builds/tinderbox/mozilla/tools/tinderbox
        • cvs up -r RELEASE_AUTOMATION_M9_1
      • on win32
cd /e/builds/tinderbox/mozilla/tools/tinderbox
cvs up -r RELEASE_AUTOMATION_M9_1
for dir in fx19rel fx19l10nrel xr19rel; do 
  cd /e/$dir; 
  for f in `ls *.pl *.html INSTALL README | grep -v tinder-config.pl`; do 
    cp -pv /e/builds/tinderbox/mozilla/tools/tinderbox/$f .;
  done
  cp -pv post-mozilla-rel.pl post-mozilla.pl
done
  • on fx-linux-1.9-slave2
    • cd /data/cltbld/bin && cvs up
    • rm -rf /builds/updates/*
    • rm -rf /builds/source/*
    • rm -rf /data/cltbld/firefox-3.0rc1
    • rm -rf /builds/tags/* (but only if you really need the space)
  • update fx-moz19-bootstrap.cfg, get review, land and move the bootstrap tag to your new revision (NB: this changed since bug 415970 - no need to cvs up the bootstrap.cfg on the master any longer but moving the tag is critical). This setup occurred relatively late because RC1 was pushed backed several times, all the other setup was done early.
  • kick off buildbot (run as cltbld on production-1.9-master):
buildbot sendchange --username=nthomas --master=localhost:9989 -m "Firefox 3.0 RC2 Build1" release

Tag

  • automated, no problems

Source

  • automated, no problems

Build

  • automated, but problems
  • mac builds were automated, no problems
  • Linux VM lost connection to the buildbot master when running ar cr libgklayout.a ..., which kills off the build. We tried re-running the build to rule out transient failure, but it was quite consistent. Resolved by IT moving fx-linux-1.9-slave2 to a VM host all of it's own. May be related to bug 435134, at build time netapp-d was handling traffic for netapp-c.
  • Windows symbol upload failed because the remote host was down, redid with (on the slave)
export SYMBOL_SERVER_HOST='dm-symbolpush01.mozilla.org'
export SYMBOL_SERVER_USER='ffxbld';
export SYMBOL_SERVER_PATH='/mnt/netapp/breakpad/symbols_ffx'
export SYMBOL_SERVER_SSH_KEY='/c/Documents and Settings/cltbld/.ssh/ffxbld_ds
make -C /e/fx19rel/WINNT_5.2_Depend/mozilla/obj-fx-trunk uploadsymbols

Sign

  • Signing doc
  • joduinn announced, and also copied over signing log, as per instructions. Automation wont proceed anyway, because of earlier linux VM problems, but doing for completeness.

l10n verify

  • automated, no problems
  • it reveals
    • changes to ja/ja-JP-mac strings in crashreporter (expected
    • added si and sl locales

Generate Updates

  • automated, no problems
  • We need to make a one-off change for the RC2 build1. In the <current-update> block of moz19-branch-patcher2.cfg, add "release" to the channel. This is to get updates from 3.0RC1 to 3.0RC2 on release. The patch is here, and was checked in after PatcherConfig ran and before patcher began to create the updates (ie while it was downloading the mar files).

Stopped - respinning.

Build 2

Respinning mac only for bug 434914, solved by backout of bug 357670. Linux and Windows builds and complete mars will be symlinked forward.

  • update fx-moz19-bootstrap.cfg, get review, land and move the bootstrap tag to your new revision (NB: this changed since bug 415970 - no need to cvs up the bootstrap.cfg on the master any longer but moving the tag is critical)
  • Modify master.cfg with
build_depscheduler = Dependent(
       name="build_dep",
       upstream=tag_scheduler,
-      builderNames=["source", "linux_build", "macosx_build", "win32_build"],
+      # builderNames=["source", "linux_build", "macosx_build", "win32_build"],
+      builderNames=["source", "macosx_build"],
       )

NB: this started builds for linux and windows off, need to figure out why. We stopped them

  • On fx-mac-1.9-slave2, fx-win32-1.9-slave2, fx-linux-1.9-slave2
    • remove the contents of /builds/verify/*
  • on fx-linux-1.9-slave2
    • cd /data/cltbld/bin && cvs up
    • rm -rf /builds/updates/*
    • rm -rf /builds/source/*
    • rm -rf /data/cltbld/firefox-3.0rc1
    • rm -rf /builds/tags/* (but only if you really need the space)
  • kick off buildbot (run as cltbld on production-1.9-master):
buildbot sendchange --username=nthomas --master=localhost:9989 -m "Firefox 3.0 RC2 Build2" release

Tag

  • automated, no problems
  • did a cvs rdiff to verify changes between FIREFOX_3_0rc2_BUILD1 FIREFOX_3_0rc2_BUILD2

Source

  • automated, no problems

Setup symlinks

On stage-old in /pub/mozilla.org/firefox/nightly/3.0rc2-candidates/build2 as cltbld: 

for i in `ls ../build1/*win32.installer.exe`; do ln -s $i; done
for i in `ls ../build1/*win32.complete.mar`; do ln -s $i; done
ln -s ../build1/windows-xpi
ln -s ../build1/win32_info.txt
ln -s ../build1/win32_signing_build1.log win32_signing_build2.log
ln -s ../build1/unsigned 
for i in `ls ../build1/*linux-i686.tar.bz2`; do ln -s $i; done
for i in `ls ../build1/*linux-i686.complete.mar`; do ln -s $i; done
ln -s ../build1/linux-xpi
ln -s ../build1/linux_info.txt

Build

  • linux and windows started off but were interrupted
  • mac finished OK

Sign

  • N/A

l10n Verify

  • Same as build1, no problems

Update Generation

  • automated, nothing to do

Update Verify

  • linux verified all green.
  • mac verify failed as expected for gu-IN b5 since it was not shipped from RC1 on
  • mac verify warned for partial updates and all locales (from bug 434696, we don't include removed-files in the partial mar): 
 ...
 diff -r source/Firefox.app/Contents/MacOS/removed-files target/Firefox.app/Contents/MacOS/removed-files
 569,570d568
 < dictionaries/PL.dic
 < dictionaries/PL.aff
  ...
  • win32 also has bug 404340, a long standing bug about how checksums show up different in updatedinstall-vs-cleaninstall. Not a blocker, we've shipped with this for a while now.
  • win32 also had an error at the start of each locale comparison, which seemed to be ignored. Is this expected? Filed and fixed bug 437039.
 ...
 cp: cannot stat `optional/*': No such file or directory
 cp: cannot stat `optional/*': No such file or directory
 ...

and some locales also had the following, which seemed to be ignored: 

 ...
 FINISH ADD .autoreg
 remove failed: -1,2 (.autoreg.moz-backup)
 FINISH ADD AccessibleMarshal.dll
 ...

This happens for complete updates with files which don't exist before the update - so there's no <file> to rename to <file>.moz-backup, and hence no <file>.moz-backup to remove once the new copy of <file> is successfully in place.

Stage

  • Failed out with: 
 ...
 log: Changed group of /data/cltbld/firefox-3.0rc2/batch1/stage-unsigned/firefox-3.0.nb-NO.mac.complete.mar to firefox
 Step Stage died: ASSERT: IsValidLocaleDeliverable(): Unknown file type in tree: /data/cltbld/firefox-3.0rc2/batch1/stage-unsigned/NOTE at Bootstrap/Step/Stage.pm line 660.
 program finished with exit code 1

Moved file up to firefox-3.0rc2 directory and restarted stage step. Retry worked fine.

Sign Installers

  • Done manually using these installer-signing-instructions here
  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-3.0rc2/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-3.0rc2/stage-merged/
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

  • done manually.

Push to mirrors

  • push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-3.0rc2/stage-merged/ /home/ftp/pub/firefox/releases/3.0rc2/

Publish Updates to Beta & Release Channel

If you need to cancel/abort updates, the quickest way to do this is here


While waiting for formal "go" and mirror sync, backup existing beta channel 

 # login to aus2-staging
 $ sudo su - cltbld
 # make sure using latest version of scripts
 $ cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot get -d bin mozilla/tools/release/bin/ 
   cvs checkout: Updating bin
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
 $ ~/bin/backupsnip 20080530-Firefox-3.0rc2

Once the mirror sync is good: on a linux/Mac build slave, run the quick updateverify test for the releastest channel 

cd /builds/verify/firefox-3.0rc2/updates
cat moz19-firefox-*.cfg > releasetest.cfg
sed -i.bak 's/betatest/releasetest/' releasetest.cfg
./verify.sh -t releasetest.cfg 2>&1 | tee releasetest.log

Should be all HTTP/200 OK results e.g.: 

grep 'HTTP' releasetest.log  | grep -v Found | grep -v 200

This test passed, but QA had issues, see bug 434065 for the details on the bouncer infrastructure problem.

On aus2-staging, ensure that releasetest channel contents match beta channel contents (we also checked this earlier) 

cd /opt/aus2/snippets/staging/20080530-Firefox-3.0rc2
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080530-Firefox-3.0rc2-test/$a");'

No diffs found - which is correct.

Once QA and overall driver gives formal "go" to put updates on beta channel 

 # login to aus2-staging
 $ sudo su - cltbld
 $ cd /opt/aus2/snippets/staging
 $ ~/bin/pushsnip 20080530-Firefox-3.0rc2
Retrieved from "https://wiki.mozilla.org/index.php?title=Firefox_3.0rc2:BuildNotes&oldid=96012"