ReleaseEngineering/Unified Release Process
Contents
Release prep
Create sign-off sheet
Seek l10n and code freeze signoff (including bonsai queries)
https://intranet.mozilla.org/Release:Template:Product
Copy to ${PRODUCT}-${VERSION} and link from https://intranet.mozilla.org/Release
Update config and copy into shell before running commands.
Configure each build
Edit the tree's tinder-config.pl settings. These are in CVS in /cvsroot/mozilla/tools/tinderbox-configs/ on the ${BRANCH_TAG}_release
$UseTimeStamp should be off Set BuildTag to ${PRODUCT_TAG}_RELEASE Set milestone to ${PRODUCT}${VERSION} $update_pushinfo should be 0 $update_package should be 1
File a bug with the changes and request review.
Tag with ${PRODUCT_TAG}_RELEASE and ${PRODUCT_TAG}_RC${RC} after checkin.
Bump version number
Three files need to be updated to the release version number (removing "pre") in cvsroot/mozilla/ :
- ${APP}/app/module.ver
- ${APP}/config/version.txt
- config/milestone.txt
These files should be changed on a minibranch. On the tagging machine (karma.build.mozilla.org):
cvs -d :ext:cltbld@cvs.mozilla.org:/cvsroot co -r ${BRANCH_TAG}\ mozilla/${APP}/app/module.ver mozilla/${APP}/config/version.txt \ mozilla/config/milestone.txt
cd mozilla/
# remove "pre" from all version numbers vi ${APP}/app/module.ver ${APP}/config/version.txt config/milestone.txt
# create minibranch cvs tag -b ${PRODUCT_TAG}_MINIBRANCH ${APP}/app/module.ver \ ${APP}/config/version.txt config/milestone.txt cvs up -r ${PRODUCT_TAG}_MINIBRANCH ${APP}/app/module.ver \ ${APP}/config/version.txt config/milestone.txt cvs ci -m"bumping version number on ${APP} on minibranch only"
Tag the release
Review the Tinderbox tree: http://tinderbox.mozilla.org/showbuilds.cgi?tree=$TBOX_TREE
We use karma (a build system) as a scratch area to tag repository files.
Set up the system to tag
# Log into karma as cltbld. # use "screen" to preserve session screen # Create the main tag directory. cd ~cltbld mkdir -p tags/${PRODUCT_TAG}_RC${RC} cd tags && ln -s ${PRODUCT_TAG}_RC${RC} ${PRODUCT_TAG}_RELEASE
Tagging cvsroot/mozilla
# Create the cvsroot tag directory. cd ~cltbld/tags/${PRODUCT_TAG}_RC${RC} mkdir cvsroot/ cd cvsroot/ # Check out Mozilla from the branch you want to tag. cvs co \ -r $BRANCH_TAG \ -D "$PULL_DATE" \ mozilla/client.mk cd mozilla gmake -f client.mk checkout \ MOZ_CO_PROJECT=all \ MOZ_CO_DATE="$PULL_DATE" \ 2>&1 | tee ../make_checkout.log
# create minibranch if its not already there cvs tag -b ${PRODUCT_TAG}_MINIBRANCH ${APP}/app/module.ver \ ${APP}/config/version.txt config/milestone.txt # pull version files from the version bump minibranch cvs up -r ${PRODUCT_TAG}_MINIBRANCH ${APP}/app/module.ver \ ${APP}/config/version.txt config/milestone.txt # Create the tag. cvs tag ${PRODUCT_TAG}_RELEASE 2>&1 | tee ../${PRODUCT_TAG}_RELEASE.log # Look for errors grep -v "^T" ../${PRODUCT_TAG}_RELEASE.log | grep -v '^cvs tag' # Create the RC tag. cvs tag ${PRODUCT_TAG}_RC${RC} 2>&1 | tee ../${PRODUCT_TAG}_RC${RC}.log # Look for errors grep -v "^T" ../${PRODUCT_TAG}_RC${RC}.log | grep -v '^cvs tag' # Create a minibranch for the pull scripts so we can change them without # changing anything on the original branch. cvs tag -b ${PRODUCT_TAG}_MINIBRANCH client.mk # Update client.mk to the minibranch you just created. cvs up -r ${PRODUCT_TAG}_MINIBRANCH client.mk # Add the new product tag to the client.mk sed "s/${BRANCH_TAG}/${PRODUCT_TAG}_RELEASE/g" client.mk > client.mk.tmp mv client.mk.tmp client.mk # Verify that client.mk only has the changes you expect cvs diff -u client.mk # Verify that client.mk has its sticky tag set properly. cvs status client.mk cvs commit -m "For $PRODUCT $VERSION, redirect client.mk onto the ${PRODUCT_TAG}_RELEASE tag." client.mk # Move the release tag onto the modified version of the pull scripts. cvs tag -F ${PRODUCT_TAG}_RELEASE client.mk cvs tag -F ${PRODUCT_TAG}_RC${RC} client.mk
Tagging mofo/talkback/fullsoft
# Create the mofo tag directory. cd ~cltbld/tags/${PRODUCT_TAG}_RC${RC} mkdir mofo/ cd mofo/ # Check out the talkback files from the branch you want to tag. cvs -d :ext:cltbld@cvs.mozilla.org:/mofo co \ -r $BRANCH_TAG \ -D "$PULL_DATE" \ talkback/fullsoft 2>&1 | tee checkout-output.log # Create the tag. cd talkback/fullsoft/ cvs tag ${PRODUCT_TAG}_RELEASE 2>&1 | tee ../${PRODUCT_TAG}_RELEASE.log # Look for errors grep -v "^T" ../${PRODUCT_TAG}_RELEASE.log | grep -v '^cvs tag'
Tagging l10n/l10n
# Create the l10n tag directory. cd ~cltbld/tags/${PRODUCT_TAG}_RELEASE mkdir l10n/ cd l10n/ # Grab list of shipped locales LOCALES=`awk '{print $1}' < ../cvsroot/mozilla/${APP}/locales/shipped-locales` # Check out the l10n files from the branch you want to tag. (for l in $LOCALES do cvs -d :ext:cltbld@cvs.mozilla.org:/l10n co \ -r $BRANCH_TAG \ -D "$l10n_PULL_DATE" \ l10n/${l} done) 2>&1 | tee checkout-output.log cd l10n/
# Create the release tag. cvs tag ${PRODUCT_TAG}_RELEASE 2>&1 | tee ../${PRODUCT_TAG}_RELEASE.log # Look for errors grep -v '^T' ../${PRODUCT_TAG}_RELEASE.log
# Create the RC tag. cvs tag ${PRODUCT_TAG}_RC${RC} 2>&1 | tee ../${PRODUCT_TAG}_RC${RC}.log # Look for errors grep -v '^T' ../${PRODUCT_TAG}_RC${RC}.log
Update repository information
Update http://developer.mozilla.org/en/docs/CVS_Tags with the time & date of the tag.
Build en-US
Identify the builds that will be used for the release.
Build
Remove the tree's last-built file.
For each platform, identify build machine.
On each build machine, log in as cltbld and run the following command in /builds/tinderbox/$BUILD_DIR :
./build-seamonkey.pl --once --mozconfig mozconfig --depend \ --config-cvsup-dir `pwd`/tinderbox-configs \ 2>&1 | tee ${PRODUCT_TAG}-RC${RC}.log
Annotate ("star") the build on the tinderbox page.
Verify build logs
As each build completes, the tinderbox cell representing that build should turn green. Click on the "L" link, then "View full log" and verify that there are no fatal errors, and that the build ends with builds being copied to the appropriate location on stage.
Copy en-US builds to candidates directory
As each build completes, copy to candidates directory.
- Review http://tinderbox.mozilla.org/showbuilds.cgi?tree=$TBOX_TREE
- Identify the directory the release build was pushed to (follow "D" link on desired build)
- Review logs ("L" link), there should be no errors.
- Set PUSH_DATE to YYYY-MM-DD-HH in sign-off sheet.
- Collect the builds together into a directory named "${VERSION}-candidates" under /home/ftp/pub/${PRODUCT}/nightly/.
#ssh to stage.mozilla.org as cltbld cd /home/ftp/pub/$PRODUCT/nightly/ mkdir -p $VERSION-candidates/rc${RC} rsync -av ${PUSH_DATE}-${PRODUCT}${VERSION}/ ${VERSION}-candidates/rc${RC}
Notify release team of availability.
Archive the source
Build the source tarballs. This should be performed on stage as the cltbld user.
NOTE - cltbld@stage:~/bin/ is a checkout of mofo/release/stage
cd ~/ mkdir -p ${PRODUCT}-${VERSION}/batch-source/rc${RC} cd ${PRODUCT}-${VERSION}/batch-source/rc${RC} ~/bin/${PRODUCT}-src-tarball-nobuild -r ${PRODUCT_TAG}_RELEASE -m $VERSION 2>&1 | tee ${PRODUCT_TAG}_RC${RC}.log chmod 644 *.bz2 rsync -n -av *.bz2 /home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC} # verify that it only copies the source tarball rsync -av *.bz2 /home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}
Build l10n
Set up the l10n build systems to repackage the en-US files.
Configure each build
Edit the tree's tinder-config.pl settings. These are in CVS in /cvsroot/mozilla/tools/tinderbox-configs/ on the ${BRANCH_TAG}_l10n_release branch.
mozconfig: Ensure that none of the mozconfigs define LOCALES_CO_DATE, only LOCALES_CO_TAG. mk_add_options LOCALES_CO_TAG="${PRODUCT_TAG}_RELEASE"
tinder-config.pl: Set %WGetFiles to the location of the en-US build. $UseTimeStamp should be off Set $BuildTag to ${PRODUCT_TAG}_RELEASE Set $milestone to ${PRODUCT}${VERSION}
File a bug with the changes and request review.
Tag with ${PRODUCT_TAG}_RELEASE_l10n and ${PRODUCT_TAG}_RC${RC}_l10n after checkin.
Build
NOTE - this can be done without stopping the multi-tinderbox builds.
Remove the tree's last-built file.
For each platform, identify build machine.
On each build machine, log in as cltbld and run the following command in /builds/tinderbox/$BUILD_DIR :
./build-seamonkey.pl --once --mozconfig mozconfig --depend \ --config-cvsup-dir `pwd`/tinderbox-configs \ 2>&1 | tee ${PRODUCT_TAG}-RC${RC}.log
Annotate ("star") the build on the tinderbox page.
Verify build logs
As each build completes, the tinderbox cell representing that build should turn green. Click on the "L" link, then "View full log" and verify that there are no fatal errors, and that the build ends with builds being copied to the appropriate location on stage.
Make a note of anywhere the word "busted" appears in the log next to a locale name; l10n builds attempt to repackage for all available locales, and will not turn red even if some or all of the locales fail to produce builds.
Minimum locales verification
- Note - shipped-locales should be from the $BRANCH_TAG
Verify that we have every locale listed in shipped_locales:
for os in win32 linux mac do ~/check_locales.sh /home/ftp/pub/${PRODUCT}/nightly/${l10n_PUSH_DATE}-${PRODUCT}${VERSION}-l10n/ ${os} \ ~/${PRODUCT}-${VERSION}/batch-source/rc${RC}/mozilla/${APP}/locales/shipped-locales done
Output of "en-US not present" is normal at this stage.
Gather l10n files
Collect the l10n builds together into a directory named "${VERSION}-candidates" under /home/ftp/pub/${PRODUCT}/nightly/.
# ssh to stage.mozilla.org as cltbld cd /home/ftp/pub/${PRODUCT}/nightly/ mkdir -p ${VERSION}-candidates/rc${RC} rsync -n -av ${l10n_PUSH_DATE}-${PRODUCT}${VERSION}-l10n/ ${VERSION}-candidates/rc${RC} # Verify that the output is what you expect!
rsync -av ${l10n_PUSH_DATE}-${PRODUCT}${VERSION}-l10n/ ${VERSION}-candidates/rc${RC}
l10n verification
Verify that l10n builds do not contain unexpected changes.
Run the l10n verification script on bm-xserve01 as cltbld (from VNC). The script is checked into /cvsroot/mozilla/testing/release/l10n/verify_l10n.sh
# convert version number SHORT_VERSION=`echo ${VERSION} | sed 's/\.//g'` SHORT_PREVIOUS_VERSION=`echo ${PREVIOUS_VERSION} | sed 's/\.//g'` mkdir -p /builds/${PRODUCT}-${VERSION}-verification/l10n cd /builds/${PRODUCT}-${VERSION}-verification/l10n cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d l10n mozilla/testing/release/l10n/ cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d common mozilla/testing/release/common/ cd l10n mkdir -p ${PRODUCT}-${SHORT_VERSION}-rc${RC} # download current release rsync --include="*.dmg" --include="*.exe" --include="*.tar.gz" --exclude='*' \ -e ssh -av stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}/ ${PRODUCT}-${SHORT_VERSION}-rc${RC} ./verify_l10n.sh ${PRODUCT}-${SHORT_VERSION}-rc${RC} 2>&1 | tee ${PRODUCT}-${SHORT_VERSION}-rc${RC}.log # download the previous release rsync --include="*.dmg" --include="*.exe" --include="*.tar.gz" --exclude='*' \ -e ssh -Lav stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${PREVIOUS_VERSION}-candidates/rc${PREVIOUS_RC}/ ${PRODUCT}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC} ./verify_l10n.sh ${PRODUCT}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC} 2>&1 | tee ${PRODUCT}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC}.log diff -r ${PRODUCT}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC}/diffs \ ${PRODUCT}-${SHORT_VERSION}-rc${RC}/diffs > l10n_metadiff_${SHORT_PREVIOUS_VERSION}_${SHORT_VERSION}-rc${RC}.txt
# look for missing files or binary diffs grep "^Only" l10n_metadiff_${SHORT_PREVIOUS_VERSION}_${SHORT_VERSION}-rc${RC}.txt grep "^Binary" l10n_metadiff_${SHORT_PREVIOUS_VERSION}_${SHORT_VERSION}-rc${RC}.txt
l10n QA tests
Send the metadiff and the location of the l10n builds to QA for testing.
Signing windows files
(installed bits, not the installer itself)
Repackaging
TODO - clean up and merge distro repack process here
Create Google and Yahoo repackaged builds as per https://intranet.mozilla.org/Build:Repackaging:CJK_Repackaging
ssh cltbld@
- argo-vm (linux)
- bm-xserve01 (mac)
- cerberus (win)
Download builds from stage
cd cjk-repackaging mkdir original_builds-${VERSION}-rc${RC} rsync -avn --include="*.ko*" --include="*.ja*" --include="*.zh*" --exclude="*" \ -e ssh stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}/* \ original_builds-${VERSION}-rc${RC}
Repack
# NOTE - on windows this /cygdrive/c/cygwin/ *MUST* be before all paths cd mozilla/extensions/brand-repackaging/ ./repack.pl --path_to_original_builds=/home/cltbld/cjk-repackaging/original_builds-${VERSION}-rc${RC} \ --output_path_for_repacks=/home/cltbld/cjk-repackaging/repackaged_builds-${VERSION}-rc${RC} \ --mar=/home/cltbld/cjk-repackaging/mar-tools/mozilla/dist/host/bin/mar 2>&1 | tee ${PRODUCT}-${VERSION}-RC${RC}.log
Copy to stage
ssh cltbld@stage.mozilla.org "mkdir -p ${PRODUCT}-${VERSION}/batch-cjk-repacks/original_builds" cd ~/cjk-repackaging/repackaged_builds-${VERSION}-rc${RC} rsync -av -e ssh * stage.mozilla.org:${PRODUCT}-${VERSION}/batch-cjk-repacks/original_builds/
Verify partner repacks
Verify that repacked builds do not contain unexpected changes.
Run the l10n verification script on bm-xserve01 as cltbld (from VNC) to produce a "metadiff". The script is checked into /cvsroot/mozilla/testing/release/l10n/verify_l10n.sh
# convert version number SHORT_VERSION=`echo ${VERSION} | sed 's/\.//g'` SHORT_PREVIOUS_VERSION=`echo ${PREVIOUS_VERSION} | sed 's/\.//g'` mkdir -p /builds/${PRODUCT}-${VERSION}-verification/repacks cd /builds/${PRODUCT}-${VERSION}-verification/repacks cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d l10n mozilla/testing/release/l10n/ cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d common mozilla/testing/release/common/ cd /builds/${PRODUCT}-${VERSION}-verification/repacks/l10n for partner in $PARTNERS do mkdir -p ${partner}-${SHORT_VERSION}-rc${RC} rsync -av -e ssh --include="*.en-US.mac.dmg" --include="*en-US.win32.installer.exe" --include="*.en-US.linux-i686.tar.gz" --exclude='*' \ "stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}/" \ ${partner}-${SHORT_VERSION}-rc${RC} rsync -av -e ssh --include="*.dmg" --include="*.exe" --include="*.tar.gz" --exclude='*' \ stage.mozilla.org:firefox-${VERSION}/batch-cjk-repacks/original_builds/${partner}/ \ ${partner}-${SHORT_VERSION}-rc${RC} ./verify_l10n.sh ${partner}-${SHORT_VERSION}-rc${RC} rsync -av -e ssh --include="*.en-US.mac.dmg" --include="*en-US.win32.installer.exe" --include="*.en-US.linux-i686.tar.gz" --exclude='*' \ stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${PREVIOUS_VERSION}-candidates/rc${PREVIOUS_RC}/ \ ${partner}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC} rsync -av -e ssh --include="*-${partner}.*.dmg" \ --include="*-${partner}.*.exe" --include="*-${partner}.*.tar.gz" --exclude='*' \ stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${PREVIOUS_VERSION}-candidates/rc${PREVIOUS_RC}/ \ ${partner}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC} ./verify_l10n.sh ${partner}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC} diff -r ${partner}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC} ${partner}-${SHORT_VERSION}-rc${RC} > l10n_metadiff_${partner}-${SHORT_VERSION}-rc${RC}.txt done
# look for missing files or binary diffs grep "^Only" l10n_metadiff_*-${SHORT_VERSION}-rc${RC}.txt grep "^Binary" l10n_metadiff_*-${SHORT_VERSION}-rc${RC}.txt
Partner repack QA tests
Send the metadiff and the location of the repacked builds to QA for testing.
Updates
Prepare the update from the previous releases to this release.
Review CVS:/mofo/release/patcher/moz*-branch-patcher2.cfg
- create new past-update for previous update path
- set up current-update to match new update path
- create a new <release> section
- get buildId from http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page
- verify that all locales in shipped-locales are present
Seek review for changes.
# ssh cltbld@prometheus-vm.mozilla.org mkdir /builds/${VERSION}-updates/ cd /builds/${VERSION}-updates/ cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d patcher mozilla/tools/patcher # config lives in private repo cvs -d cltbld@cvs.mozilla.org:/mofo co -d config release/patcher/moz180-branch-patcher2.cfg cd patcher export CVSROOT=cltbld@cvs.mozilla.org:/cvsroot # build tools ./patcher2.pl --build-tools --app=${PRODUCT} \ --config=../config/moz180-branch-patcher2.cfg 2>&1 | tee ${PRODUCT}_build-tools.log # download complete MARs ./patcher2.pl --download --app=${PRODUCT} \ --config=../config/moz180-branch-patcher2.cfg 2>&1 | tee ${PRODUCT}-download.log # Create partial patches and snippets ./patcher2.pl --create-patches --app=${PRODUCT} \ --config=../config/moz180-branch-patcher2.cfg 2>&1 | tee ${PRODUCT}-create-patches.log
# prepare aus2-staging # ssh aus2-staging.mozilla.org cd /opt/aus2/incoming/3-staging tmpdir="`date +%Y%m%d`-${SHORT_PRODUCT}-${VERSION}" sudo mkdir ${tmpdir}-test sudo chown cltbld ${tmpdir}-test sudo mkdir ${tmpdir} sudo chown cltbld ${tmpdir} # copy updates from prometheus-vm.mozilla.org # ssh prometheus-vm.mozilla.org cd /builds/${VERSION}-updates/patcher/temp/${PRODUCT}/${PREVIOUS_VERSION}-${VERSION}/ tmpdir="`date +%Y%m%d`-${SHORT_PRODUCT}-${VERSION}" rsync -nav -e "ssh -i $HOME/.ssh/aus" aus2.test/ \ aus2-staging.mozilla.org:/opt/aus2/incoming/3-staging/${tmpdir}-test/ rsync -nav -e "ssh -i $HOME/.ssh/aus" aus2/ \ aus2-staging.mozilla.org:/opt/aus2/incoming/3-staging/${tmpdir}/
# look at the files that would have been copied, it should only be # betatest and releasetest directories for the first rsync and # only beta and release directories for the second rsync rsync -av -e "ssh -i $HOME/.ssh/aus" aus2.test/ \ aus2-staging.mozilla.org:/opt/aus2/incoming/3-staging/${tmpdir}-test/ rsync -av -e "ssh -i $HOME/.ssh/aus" aus2/ \ aus2-staging.mozilla.org:/opt/aus2/incoming/3-staging/${tmpdir}/
### quick verification # ensure that there are only test channels # ssh aus2-staging.mozilla.org cd /opt/aus2/incoming/3-staging/${tmpdir}-test find . -type f | grep -v test # last command should return no output
# push partial MARs to stage # ssh prometheus-vm.mozilla.org # NOTE - this will also be pushed to the live server! # # Verify that this command only copied what you want, # and remove the "-n". rsync -n -av -e ssh ftp/ stage.mozilla.org:/home/ftp/pub/ # In particular, verify that no "complete" MARs are sent. rsync -av -e ssh ftp/ stage.mozilla.org:/home/ftp/pub/ # errors about "failed to set times ... Operation not permitted" are ok
# push snippets to aus2-staging # ssh to aus2-staging.mozilla.org # NOTE - this will also be pushed to the live server! # # backup current settings rsync -av ../../3/ ~/3-backup_${tmpdir}-test/
# Verify that this command only copied what you want, # and remove the "-n": rsync -n -av ${AUS_PRODUCT}/ ../../3/${AUS_PRODUCT}/ # In particular, verify that no non-test channels are present. rsync -n -av ${AUS_PRODUCT}/ ../../3/${AUS_PRODUCT}/ | grep -v test # Push to production sudo rsync -av ${AUS_PRODUCT}/ ../../3/${AUS_PRODUCT}/
Update verification
Verify that all updates are being served up on the expected channels.
Run the following on bm-xserve01:
# convert version number SHORT_VERSION=`echo ${VERSION} | sed 's/\.//g'` SHORT_PREVIOUS_VERSION=`echo ${PREVIOUS_VERSION} | sed 's/\.//g'` mkdir -p /builds/${PRODUCT}-${VERSION}-verification/updates cd /builds/${PRODUCT}-${VERSION}-verification/updates cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d updates mozilla/testing/release/updates/ cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d common mozilla/testing/release/common/
Customize updates/updates.cfg to contain the channels you are interested in testing. It should only have lines that are "channel=releasetest". It should have a line for each platform/version combination, but no lines for the current release. The lines for the previous release should have "from=" and "to=" parameters. Old versions should not. Don't forget to update the build_id parameter.
cd /builds/${PRODUCT}-${VERSION}-verification/updates/updates ./verify.sh -t > ${PRODUCT}-${VERSION}-rc${RC}.log grep ^HTTP/1.1 ${PRODUCT}-${VERSION}-rc${RC}.log | grep -v -E "200 OK|304 Not Modified|302 Found" # The last command should have no output unless there were errors
Send email to QA letting them know that the updates are ready to test on the betatest and releasetest channels.
Stage release
Prepare staging area
Prepare the staging directory for the release.
NOTE: This has changed- you need to use the new GPG key attached to the bug. joduinn 29aug2007
# Create the staging directory. cd ~/ mkdir -p ${PRODUCT}-${VERSION}/; cd ${PRODUCT}-${VERSION}/ mkdir -p batch-skel/stage/ # Create the contrib and contrib-localized directories with expected access # rights. cd ~/${PRODUCT}-${VERSION}/batch-skel/stage/ mkdir -p contrib contrib-localized chown cltbld:${PRODUCT} contrib* chmod 2775 contrib* # NOTE - should have a standard "master" copy somewhere else # Copy the KEY file from the previous release directory. # rsync -a /home/ftp/pub/${PRODUCT}/releases/1.5/KEY . # # The new KEY is attached to bug 377781. # or can be copied from rsync -av /home/ftp/pub/firefox/releases/2.0.0.9/KEY .
# Prepare the merging directory. cd ~/${PRODUCT}-${VERSION}/ mkdir -p stage-merged/ rsync -av batch-skel/stage/ stage-merged/
Collect builds
Collect the release files onto stage.mozilla.org.
mkdir -p ~/${PRODUCT}-${VERSION}/batch1/prestage cd ~/${PRODUCT}-${VERSION}/batch1/prestage rsync -Lav /home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}/ . find . -type f -exec chmod -v 644 {} \; cd ..
Remove unreleased builds
Remove the unneeded files.
cd ~/${PRODUCT}-${VERSION}/batch1 rsync -av prestage/ prestage-trimmed/ cd prestage-trimmed/ rm -v *.zip *.log *.txt rm -rv unsigned/ # is this still needed? find . -name xforms.xpi -exec rm -fv {} \; find . -type d -name "*-xpi" -empty -exec rmdir {} \; # ja-JP-mac is the JA locale for mac, do not ship ja rm -v *.ja.mac* rm -v mac-xpi/ja.xpi # ja is the JA locale for win32/linux, do not ship ja-JP-mac find . -name "*ja-JP-mac.win32*" -exec rm -fv {} \; find . -name "*ja-JP-mac.linux*" -exec rm -fv {} \; find . -name "en-US.xpi" -exec rm -fv {} \; rm -v linux-xpi/ja-JP-mac.xpi rm -v windows-xpi/ja-JP-mac.xpi cd ../
Rename builds
Rename the files. If its an Alpha/Beta, do this manually, otherwise use groom-files below.
rsync -Lav prestage-trimmed/ stage/ cd stage/ rm *.mar ~/bin/groom-files --short=${VERSION} . mv -v linux-xpi linux-i686/xpi mv -v windows-xpi win32/xpi mv -v mac-xpi mac/xpi cd ../
Verify locales
cd stage ~/bin/verify-locales.pl -m ../../batch-source/rc${RC}/mozilla/${APP}/locales/shipped-locales cd ../
It's normal to get messages like this
Missing XPI locale for platform 'linux': en-US Missing non-locale XPI: linux-i686/xpi/browser Missing non-locale XPI: linux-i686/xpi/talkback Missing non-locale XPI: linux-i686/xpi/xpcom Missing non-locale XPI: linux-i686/xpi/adt Missing XPI locale for platform 'win32': en-US Missing non-locale XPI: win32/xpi/browser Missing non-locale XPI: win32/xpi/talkback Missing non-locale XPI: win32/xpi/xpcom Missing non-locale XPI: win32/xpi/adt Missing XPI locale for platform 'osx': en-US Missing non-locale XPI: mac/xpi/browser Missing non-locale XPI: mac/xpi/talkback Missing non-locale XPI: mac/xpi/xpcom Missing non-locale XPI: mac/xpi/adt
because we no longer ship an en-US.xpi or the app components as xpi files. Any other messages (eg missing or extra locales) should be fixed before continuing.
Sign builds
See here for current Fx2.0, Tb2.0, Fx3.0 signing instructions.
Define the variables
In order to generalise this document, we'll use environment vars for substitutions. For example:
PRODUCT=firefox VERSION=2.0.0.15 BUILD=1 TAG=FIREFOX_2_0_0_15_RELEASE
You should adjust these for the release you are working on, and enter them at the Cygwin prompt.
Download files to keymaster
After logging into keymaster:
- check there is enough free disk space
- then run the following commands *on* keymaster to download the files to keymaster.
mkdir -p ~/signing-work/${PRODUCT}-${VERSION} cd ~/signing-work/${PRODUCT}-${VERSION} rsync -e ssh -av cltbld@stage.mozilla.org:/data/cltbld/${PRODUCT}-${VERSION}/batch1/stage-signed/ ./stage-unsigned/ rsync -av stage-unsigned/ stage-signed/
Sign win32 EXEs
first, start signcodepwd.exe (in seperate window)
cd ~/signing-work/${PRODUCT}-${VERSION}/stage-signed/win32 sign-release . verify-signatures .
cd ../
Sign all files with PGP key
Sign all of the release files with the PGP key into detached signatures.
cd ~/signing-work/${PRODUCT}-${VERSION}/stage-signed sign-files .
Verification
Verify that there is a detached signature for each file.
find -not -name "*.xpi" -not -name "*.asc" -type f | wc -l find -name "*.asc" -type f | wc -l
Upload the signatures back to stage-signed/
Verify that only .exe and .asc files are modified.
cd ~/signing-work/${PRODUCT}-${VERSION} rsync -n -e ssh -av stage-signed/ cltbld@stage.mozilla.org:${PRODUCT}-${VERSION}/batch1/stage-signed/ | grep -v asc | grep -v exe rsync -e ssh -av stage-signed/ cltbld@stage.mozilla.org:${PRODUCT}-${VERSION}/batch1/stage-signed/
Merge files
Merge update (MAR) files
The following steps are to publish the update files for the release.
cd ~/${PRODUCT}-${VERSION}/ mkdir -p batch1/mar/stage cd batch1/mar/stage rsync -av ../../prestage-trimmed/ ./ find -type f -not -regex '.*.mar$' -exec rm -fv {} \; find -type d -empty -exec rmdir -v {} \; ~/bin/groom-files --short=${VERSION} . rsync -n -av update /data/cltbld/${PRODUCT}-${VERSION}/stage-merged/
Verify that only the files you expect are copied.
rsync -av update /data/cltbld/${PRODUCT}-${VERSION}/stage-merged/
Merge signed files
Merge the signed file set into stage-merged.
cd ~/${PRODUCT}-${VERSION}/ rsync -n -av batch1/stage-signed/ stage-merged/
Verify that only the files you expect are copied.
rsync -av batch1/stage-signed/ stage-merged/ cd stage-merged/ ~/bin/checksum-files .
Update bouncer links
Create bouncer (http://download.mozilla.org/admin) links for all products (MARs and builds).
Push the button
NOTE - copying to FTP must be done 12 hours in advance of the release announcement to ensure that mirrors are synchronized.
Make the release live
Make the release live.
Create the FTP directory.
mkdir -p /home/ftp/pub/${PRODUCT}/releases/${VERSION}
Copy the staging directory's files into the FTP directory.
cd ${PRODUCT}-${VERSION}/ rsync -av -n stage-merged/ /home/ftp/pub/${PRODUCT}/releases/${VERSION}/ # Verify that the output is what you expect, e.g. the expected directories and files are copied to the expected location.
rsync -av stage-merged/ /home/ftp/pub/${PRODUCT}/releases/${VERSION}/
Move the "latest" symlink
cd /home/ftp/pub/${PRODUCT}/releases rm latest && ln -s ${VERSION} latest
Update the mozilla-current rsync module, which only has the current releases. On stage.m.o, modify /etc/rsyncd-mozilla-current.exclude to add the new release, replacing the previous one. IT & Nick can do this.
Request QA sign-off that build IDs in the release files match expectations.
Make the update live
Make the update live.
https://intranet.mozilla.org/Build:Automatic_Updates_for_a_Release
ssh aus2-staging.mozilla.org cd /builds/${VERSION}-updates/3-staging/
### quick verification # ensure that test channels have the same content as release channels # TODO # compare production differences diff -r ${AUS_PRODUCT}-${tmpdir}/${AUS_PRODUCT}/ ../3/incoming/${AUS_PRODUCT}/
# ensure that only the files you expect are copied! rsync -n -av ${AUS_PRODUCT}-${tmpdir}/${PRODUCT}/ ../3/incoming/${AUS_PRODUCT}/
# push to production rsync -av ${AUS_PRODUCT}-${tmpdir}/${AUS_PRODUCT}/ ../3/incoming/${AUS_PRODUCT}/
Reset for next release
Bump version number
Bump mozilla version number to ${NEXT_VERSION}pre/${PLATFORM_NEXT_VERSION}pre.
Three files need to be updated to the right version number in cvsroot/mozilla/ :
- ${APP}/app/module.ver
- ${APP}/config/version.txt
- config/milestone.txt
File a "version bump" bug and attach a patch before committing.