ReleaseEngineering/How To/Update Mercurial
Mercurial is one of our core binaries that we need to install everywhere.
Contents
Philosophy
We install Mercurial into /tools/python27-mercurial/ everywhere* (* - On windows we override the mozilla-build version). This is due to wanting a single, common, place for mercurial, and to avoid needing to update system packages that may depend on mercurial.
This document is designed so it can/should be followed top-down, but with self-contained sections to allow easy skimming/read-through.
Future Issues
There are some forseen reasons this doc could be incorrect for your needs. Below are those reasons, and suggested solutions.
- We change the way PuppetAgain goes about doing packaging
- Follow the new documentation to achieve a similar result, this page is informative not normative for how we do things.
- We deployed PuppetAgain on Windows (adding a need for Windows to the puppet code)
- We'll need to do this from scratch once, and update these docs.
- OSX 10.6 will soon no longer be used in our infra
- Delete the 10.6 instructions from this wiki page when that happens
Prepare for the Upgrade
- File a bug for the upgrade, (if not already filed)
- Send an e-mail to
Dev Services <developer-services@mozilla.org>
asking if there are any concerns (set a deadline, so you are not indefinitely blocked on replies)
Create the Packages
CentOS
6.x
(This is an informative description of The PuppetAgain How to Build RPMs)
- In
us-east
AWS console find and startrpmpackager1.srv.releng.use1.mozilla.com
- Connect
USER=`whomai` ssh $USER@rpmpackager1.srv.releng.use1.mozilla.com
- Get Old RPM, Spec from puppet, and extract spec from rpm, Also get new sources
NEWHGVER=4.4.1 # Set this appropriately NEWHGREL=1 #Set this too OLDHGVER=3.9.1 OLDHGREL=1 OLDHGFULL=$OLDHGVER-$OLDHGREL NEWHGFULL=$NEWHGVER-$NEWHGREL PUPPETAGAIN=http://puppetagain.pub.build.mozilla.org HGREPO=data/repos/yum/custom/mozilla-mercurial/ mkdir mercurial-$NEWHGVER; cd mercurial-$NEWHGVER wget $PUPPETAGAIN/$HGREPO/x86_64/mozilla-python27-mercurial-$OLDHGFULL.el6.x86_64.rpm wget $PUPPETAGAIN/$HGREPO/i386/mozilla-python27-mercurial-$OLDHGFULL.el6.i686.rpm wget $PUPPETAGAIN/$HGREPO/x86_64/mozilla-python27-mercurial-$OLDHGFULL.el6.src.rpm mkdir old-extracted; cd old-extracted rpm2cpio ../mozilla-python27-mercurial-$OLDHGFULL.el6.src.rpm | cpio -ivd cd .. wget https://www.mercurial-scm.org/release/mercurial-$NEWHGVER.tar.gz
- Check for any unexpected differences between puppet and old version
diff py27_mercurial.spec old-extracted/py27_mercurial.spec
- Update spec for versions
sed -i "s/\(Version:.*\)$OLDHGVER/\1$NEWHGVER/" py27_mercurial.spec sed "s/\(Release:.*\)$OLDHGREL/\1$NEWHGREL/" py27_mercurial.spec
- Build the src rpm
mock -r puppetagain-centos6.5-64 --buildsrpm --sources $PWD --spec py27_mercurial.spec
- Copy output to a safe place
mkdir output cp /var/lib/mock/epel-6-x86_64/result/mozilla-python27-mercurial-$NEWHGFULL.el6.src.rpm output/
- Now build the actual rpms (and clean caches first)
sudo rm -rf /var/cache/mock/epel-6-x86_64/ccache sudo rm -rf /var/cache/mock/epel-6-i386/ccache/ mock --rebuild -r puppetagain-centos6.5-64 output/mozilla-python27-mercurial-$NEWHGFULL.el6.src.rpm mock --rebuild -r puppetagain-centos6.5-32 output/mozilla-python27-mercurial-$NEWHGFULL.el6.src.rpm
- Copy those out (if nothing failed):
cp /var/lib/mock/epel-6-x86_64/result/*.x86_64.rpm output/ cp /var/lib/mock/epel-6-x86_64/result/*.i?86.rpm output/
- IF there are any ERRORS
- Edit the spec file to account for them and retry subsequent steps, you'll need to carry these spec file changes over into the Puppet Section of the work.
- scp the files in output to somewhere local, in prep for the #Deploy to PuppetAgain Section
- Halt the system (avoids need for aws console)
sudo su - users # Check for other logged in users first, be nice! halt
Ubuntu
Precise (12.04)
(This is an informative description of The PuppetAgain How to Build DEBs)
- In
us-east
AWS console find and startubuntu64packager1.srv.releng.use1.mozilla.com
- Connect
USER=`whomai` ssh $USER@ubuntu64packager1.srv.releng.use1.mozilla.com
- Get old from repo
NEWHGVER=4.4.1 # Set this appropriately OLDHGVER=3.9.1 PUPPETAGAIN=http://puppetagain.pub.build.mozilla.org HGREPO=data/repos/apt/custom/mozilla-mercurial/pool/precise mkdir old wget $PUPPETAGAIN/$HGREPO/mozilla-python27-mercurial_$OLDHGVER.debian.tar.gz cd old tar -xzf ../mozilla-python27-mercurial_$OLDHGVER.debian.tar.gz cd ..
- Get old deb folder from puppet
hg clone https://hg.mozilla.org/build/puppet repo-puppet cp -r puppet/modules/packages/manifests/mozilla/mozilla-python27-mercurial-debian ./
- Check for surprising differences
diff -r ./mozilla-python27-mercurial-debian/ old/debian/
- Edit changelog (adding new entry)
- Note DEB Changelogs follow a specific format, add to top, match the one below yours.
EDITOR=vim $EDITOR ./mozilla-python27-mercurial-debian/changelog
- Get the new .tar.gz from selenic (and make sure its named correctly post-download)
wget https://www.mercurial-scm.org/release/mercurial-$NEWHGVER.tar.gz \ -O mozilla-python27-mercurial_$NEWHGVER.orig.tar.gz
- Build it
puppetagain-build-deb precise amd64 mozilla-python27-mercurial-debian puppetagain-build-deb precise i386 mozilla-python27-mercurial-debian
- Copy the artifacts out
cp /tmp/mozilla-python27-mercurial-precise-amd64/* output/ cp /tmp/mozilla-python27-mercurial-precise-i386/* output/
- scp the files in output to somewhere local, in prep for the #Deploy to PuppetAgain Section
- copy out any/all changes you have made to the puppet deb directory (the changelog edit above) - You'll include that with your puppet patch.
- Halt the system (avoids need for aws console)
sudo su - users # Check for other logged in users first, be nice! halt
OSX
(This is an informative description based on The PuppetAgain How to Build DMGs)
OSX is funky and needs a package built separately for each version, on the OS version its targeted for. This means we need to pull active slaves from rotation and install XCode on them via a puppet user environment.
I'm not repeating the puppet user environment instructions here, but I will detail what you need to change for these nodes.
10.7 (Lion)
- Disable a random production slave, commenting with whatever bug you need it for, (you're essentially doing a loan - but need puppet running)
- There is no need for puppet environment for this, its a builder and already has xcode
- login to your host as cltbld
- Get the necessary file:
mkdir mercurial cd mercurial wget -O py27_mercurial-dmg.sh \ http://hg.mozilla.org/build/puppet/raw-file/default/modules/packages/manifests/mozilla/py27_mercurial-dmg.sh chmod 755 ./py27_mercurial-dmg.sh
- Update the script to be the correct version
OLDHGVER=3.9.1 NEWHGVER=4.4.1 sed -i "s/version=$OLDHGVER/version=$NEWHGVER/" ./py27_mercurial-dmg.sh
- Get the new tar.gz
wget https://www.mercurial-scm.org/release/mercurial-$NEWHGVER.tar.gz
- Run the script
./py27_mercurial-dmg.sh
- The script will output where the dmg is, copy it outside the host (storing it in a way that you know it was for 10.7)
- You now want to reimage the slave following this mana doc, and put it back in production in slavealloc
10.10 (Yosemite)
- Disable a random production slave, commenting with whatever bug you need it for, (you're essentially doing a loan - but need puppet running)
- Make sure you have a (clean/updated) puppetagain environment setup you can use (See The Docs)
- Add packagemaker and xcode to the node def for testers (applies to all OSX jobs)
- Note This will break any tester puppet runs for non OSX (but is fixable by reverting this)
# linux64 and OS X node /t.*-\d+\.test\.releng\.scl3\.mozilla\.com/ { # hosts starting with t and ending in -digit.test.releng.scl3.mozilla.com $slave_trustlevel = 'try' include toplevel::slave::releng::test::gpu # New Lines Here include packages::packagemaker include packages::xcode }
- login to your host and run puppet
- CHANGE THE PUPPET ARGS APPROPRIATELY'
SLAVE=talos-t-yosemite-r7-0089 ssh root@$SLAVE.test.releng.scl3.mozilla.com USER=yourusername puppet agent --server foo --test --environment $USER
- change to cltbld
su - cltbld
- Get the necessary file:
mkdir mercurial cd mercurial wget -O py27_mercurial-dmg.sh \ http://hg.mozilla.org/build/puppet/raw-file/default/modules/packages/manifests/mozilla/py27_mercurial-dmg.sh chmod 755 ./py27_mercurial-dmg.sh
- Update the script to be the correct version
OLDHGVER=3.9.1 NEWHGVER=4.4.1 sed -i "s/version=$OLDHGVER/version=$NEWHGVER/" ./py27_mercurial-dmg.sh
- Get the new tar.gz
wget http://mercurial.selenic.com/release/mercurial-$NEWHGVER.tar.gz
- Run the script
./py27_mercurial-dmg.sh
- The script will output where the dmg is, copy it outside the host (storing it in a way that you know it was for 10.10)
- You now want to reimage the slave following this mana doc, and put it back in production in slavealloc
Windows
As of this doc's writing; this is done via GPO installing to <mozilla-build>/hg
as a blow-away-what-was-there install (on hardware machines) and via updating AMIs with the new hg version (on AWS machines). This needs to be done for both builders (c.f. bug 1302375) and testers (c.f. bug 1302376)
Package Deployment
CentOS
Note: This is an informative depiction of https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#CentOS:_Adding_New_Packages
CentOS needs the files on releng-puppet2 (the distinguished puppetmaster).
- Transfer the files from local to releng-puppet2 (your user dir)
USER=yourusername scp ./*.rpm $USER@releng-puppet2.srv.releng.scl3.mozilla.com:mercurial/
- As root, copy the files over and fixup
**CHANGE THE USERNAME** sudo su - USER=yourusername for i in $(ls /home/$USER/mercurial/mozilla-python27-mercurial-*.el6.{src,x86_64}.rpm); do cp $i /data/repos/yum/custom/mozilla-mercurial/x86_64/; done for i in $(ls /home/$USER/mercurial/mozilla-python27-mercurial-*.el6.{src,i686}.rpm); do cp $i /data/repos/yum/custom/mozilla-mercurial/i386/; done pushd /data/repos/yum/releng/public/CentOS/6/i386 puppetmaster-fixperms popd pushd /data/repos/yum/releng/public/CentOS/6/x86_64 puppetmaster-fixperms popd
- Update repo metadata
cd /data/repos/yum/custom/mozilla-mercurial/x86_64/; createrepo --update . cd /data/repos/yum/custom/mozilla-mercurial/i386/; createrepo --update . puppetmaster-fixperms
Ubuntu
Note: This is an informative depiction of https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Ubuntu:_Adding_New_Packages
- Transfer the files from local to releng-puppet2 (your user dir)
USER=yourusername scp ./* $USER@releng-puppet2.srv.releng.scl3.mozilla.com:mercurial-apt/
- As root, copy the files over and fixup
**CHANGE THE USERNAME** sudo su - USER=yourusername cd /data/repos/apt/custom/mozilla-mercurial cp -v /home/$USER/mercurial-apt/mozilla-python27-mercurial_*{_amd64.changes,_amd64.deb,_i386.changes,_i386.deb,.debian.tar.gz,.dsc,.orig.tar.gz} ./pool/precise/ ./update.sh puppetmaster-fixperms
OSX
Note: This is an informative depiction of https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Darwin
- Transfer the files from local to releng-puppet2 (your user dir)
USER=yourusername scp ./*.dmg $USER@releng-puppet2.srv.releng.scl3.mozilla.com:mercurial-DMGs/ ** This instruction assumes your DMGs are located at 10.X/ under that directory
- As root, copy the files over and fixup
**CHANGE THE USERNAME** sudo su - USER=yourusername cd /data/repos/DMGs cp -v ~$USER/mercurial-DMGs/10.6/* 10.6/ cp -v ~$USER/mercurial-DMGs/10.7/* 10.7/ cp -v ~$USER/mercurial-DMGs/10.10/* 10.10/ puppetmaster-fixperms
Create puppet patch
- replace the spec file for CentOS
- The final spec file you used above for the CentOS build you should place at
<path-to-local-puppet-repo>/modules/packages/manifests/mozilla/py27_mercurial.spec
- The final spec file you used above for the CentOS build you should place at
- replace all files for the Debian packaging
- The final files you use above for Ubuntu should be replace the entire contents of the directory at
<path-to-local-puppet-repo>modules/packages/manifests/mozilla/mozilla-python27-mercurial-debian
- The final files you use above for Ubuntu should be replace the entire contents of the directory at
- replace the dmg creation script
- If you changed the package building script above for OSX, replace the one at
<path-to-local-puppet-repo>modules/packages/manifests/mozilla/py27_mercurial-dmg.sh
- If you changed the package building script above for OSX, replace the one at
- Update the mercurial versions in
<path-to-local-puppet-repo>modules/packages/manifests/mozilla/py27_mercurial.pp
- You can do this by hand too, be sure to change the versions specified
sed -i "s/3.7.3/3.9.1/" modules/packages/manifests/mozilla/py27_mercurial.pp
- Update the repoflag to cause the system package repository cache to refresh
- Inside modules/packages/manifests/setup.pp you want to increase the repoflag var everwhere
- Post the patch to the upgrade bug, and request review. (testing first if desired, see next section)
Testing the patch
- On a puppetmaster you should apply the above created patch and run against one of each of the host types (centos builder, ubuntu testers both arch's, all vers of OSX)
Deploy the change
- Commit the reviewed patch to default branch of puppet
- Merge patch to production branch of puppet
- push patch
- Inform sheriffs of the deploy
- Allow 24 hours for AWS slaves to regen their "golden image" to ensure no breakage there.
- Patch should begin deploying to all server and "atboot" class machines within about 30 minutes of push.