ReleaseEngineering/How To/Cleanup Old Candidates Directories
We accumulate release candidates as time goes by. Since we have a finite amount of space in that location (500G at the time of writing) we must tidy these up as we periodically. It can be useful to have old candidates around for later debugging so we move them to other storage first, and later remove all the binaries while retaining logs. The sequence of locations is
/pub/mozilla.org/firefox/candidates /data/cltbld/old-candidates-dirs/firefox /data/cltbld/old-candidates-dirs/firefox/stripped
We also have old xulrunner and mobile candidates in /data/cltbld/old-candidates-dirs/. We'll talk about these in reverse order, since that's the way you need to maintain space.
This is all subject to change with the move to SCL3 and a new staging server.
Cleaning up /data/cltbld/old-candidates-dirs/firefox/
Releases end up in /data/cltbld/old-candidates-dirs/firefox/stripped/ as a final resting place, with only the buildbot and signing logs, SUMS files + sigs, plus the <platform>_info.txt files kept for buildIDs. This is a few 10s of MB instead of 10+ GB.
We move files from /data/cltbld/old-candidates-dirs/firefox/ into stripped/ like this
# ffxbld@stage cd /data/cltbld/old-candidates-dirs/firefox/ ls -ltr | head
to get a list of the oldest candidates. You can run du on those directories if you want to know how much you're going to free. Then start moving them, eg
./move-stripped 6.0
returning
Saving important objects from 6.0-candidates Deleting 6.0-candidates Free space is now: Filesystem Size Used Avail Use% Mounted on /dev/cciss/c0d0p1 1.9T 1.6T 289G 85% /data
Cleaning up /pub/mozilla.org/firefox/candidates/
NB: Leave the last build from each branch present for update generation. This includes old branches (eg 3.0.19, 3.5.19, 3.6.28). Otherwise move the oldest first
Assuming you have space in /data/cltbld/old-candidates-dirs/firefox/ it's as simple as
# ffxbld@stage cd /data/cltbld/old-candidates-dirs/firefox/ ls -1 /pub/mozilla.org/firefox/candidates/
You can run du on those directories to see how much space you need. Then start moving them,
./move-from-candidates 10.0esr 11.0b2 10.0.1
This will take take quite a while, as each is multiple GB and ionice is used to run at low-priority. It'll show you progress along the way though.
TODO - figure out a work around for contrib/ files that aren't owned by ffxbld. In the meantime this will stop deleting after it fails on the first version. Run as root to avoid this.