Build:StageMigrationNotes

Current Configuration

Currently, contributors and Tinderbox machines upload directly to surf (CNAMEed to stage.m.o). manna (CNAMEd to ftp.m.o) pulls these builds from surf via rsync at set intervals.

surf is responsible for virus scanning and verifying other build-related issues.

Both surf and manna are constantly dangerously low on diskspace; the goals of this migration are to:

• Solve this continuing disk space problem
• Ensure that bits are virus scanned and otherwise verified before reaching the mirrors, which is currently not the case. Infected bits will eventually get removed from ftp.m.o/mirrors, but it's not instantaneous.

Migration Plan

On the evening of Thursday, 14 September, from 6 pm to midnight PDT:

1. surf/stage will be moved inside the firewall and become responsible for virus scanning and verifying all bits that go to the mirror farm/ftp.m.o. Because of this, access will be restricted to this host.
2. surf/stage will have a read/write mount to shared, central storage.
3. manna will be renamed stage (as a CNAME), and will mount this shared, central storage read-only. This shared, central storage will become the new definitive copy of ftp.m.o.
4. Some tinderboxen will change to publish their builds to surf
5. Bits in the archive.m.o rsync module will move entirely to the shared, central storage.
6. Bits in the release.m.o rsync module will continue to exist on surf and the shared central storage
7. surf will pull bits onto its local disk via a scheduled process and perform various verification steps on them (virus scanning, permissions checks, etc.); when these checks pass, they will be uploaded to the shared, central storage, and immediately appear on the ftp mirrors/ftp.m.o.

How These Changes Affect External Contributors

• External contributors may have to accept a new SSH host key; this key will be published after the switchover is complete, so contributors can verify it before accepting it
• Access to surf will be limited
• As part of this plan, we will be auditing access to manna; if access is removed, but it shouldn't have been, please email build@mozilla.org.
• Builds may be spotty on Friday; we'll do our best to ensure the builds get uploaded as expected.

Process

This is a rough, ordered timeline of the steps necessary to fulfill the requirements.

Step	Build	IT
Prerequ.	Initial list of rsync directories from surf -> netapp	Netapp mounts available to surf (r/w) Netapp mounts available to manna (r/o) LDAP up and running on both for authentication? Reduce TTL on stage.m.o in anticipation of switchover
1		Move surf inside firewall Connect Netapp mount to surf
2	rsync the contents of surf -> the netapp	Setup/verify as appropriate the http/rsync/ftp configs on manna; test with builds that begin to be available (it may make sense to make stage.m.o/ftp.m.o completely unavailable to mirrors/outsiders at this point, since we don't want the mirrors picking up incomplete directory tree)
3	Delete the contents of the archive rsync module on surf; delete ftp logs on surf	Verify that ftp.m.o/stage.m.o appear as before, but accessed via manna (can diff manna's local disk vs. the Netapp to verify)
4	Point Tinderboxen at surf	Start manna-upload-area -> surf rsync scripts on surf
5	Verify builds in the build farm get to surf Verify builds outside the build farm can get to manna's upload area	Setup virus scanning script Start surf -> Netapp rsync scripts
6	Verify end-to-end tinderbox -> ftp.m.o status	Verify end-to-end tinderbox -> ftp.m.o status

To be integrated into this process

• Transfer home directories for contributors from surf to manna