ReleaseEngineering/How To/VCSSync

From MozillaWiki
Jump to: navigation, search

This is documentation on the mozharness-based vcs sync processes.

See legacy docs for the legacy (aka "old") process.

Quick notes

"VCS-sync" is a system which has evolved from it's original purpose to being the tool used whenever "this repository needs to be publicly visible 'over here'", perhaps with modifications, and a few exceptions on "publicly visible". The current (2016 Q1) business needs it addresses are:

  1. Support for gecko developers who prefer to work with git:
    • All gecko primary development and release branches are merged into a single git repository (with CVS history) in "gecko-dev.git".
    • All gecko twig and misc branches are merged into a single git repository (with CVS history) in "gecko-projects.git".
  2. Support for Mozilla development tools and processes which (for operational needs (e.g. travis-ci), or developer preference, should appear in both Mercurial and Git repositories.
    • These are "mirrors" from one system to the other.
    • One repo is always designated the RoR (Repository of Record), and is the only one where commits will "stick".
    • Refer to the system configurations for each mirror to determine the RoR. (It can, and has, changed over time for some repositories.)
  3. Support for Partner facing FxOS development processes.
    • The URL's for these repositories can not change, as they are referenced in manifest files which comprise the delivery to Partners.
    • These repositories on git.mozilla.org satisfy various contractual needs with FxOS partners.
    • These repositories must not contain either object deletions, or non-fast-forward commits.
    • These repositories only contain content that was "delivered" to Partners, with branches named according to FxOS conventions.
    • These repositories include any "Mozilla Contributions" to FxOS, which includes code beyond gecko.git and gaia.git.
  4. Support for internal FxOS development processes.
    • Many Android repositories have been mirrored to git.mozilla.org for the purpose of having a known-good-repository for the build systems to clone from.
    • Some tests still require translation from Git → Mercurial format. Each such gaia.git branch is mapped to a distinct Mercurial repository.


What's running where

Anything not listed below is probably running in the new system. The system pushing the production content is listed first. If the other system is listed, we're in parallel operations mode and changes should be made to both systems.

Repositories needed to Build Mozilla Products

Repository Old/New Web views at Notes
gecko-dev.git new github, Moz
gecko-projects.git new github, Moz
releases/gecko.git old, new Moz Partner facing FxOS [1]
releases/gaia.git old, new Moz Partner facing FxOS [1]
releases/l10n/<locale>/{gecko,gaia}.git old, new Moz (browse from there) Partner facing FxOS [1]
integration/gaia* old Moz (browse from there) Hg versions of gaia branches to support older tooling
b2g mirrors old Moz browse external and b2g from there (r/o) Local copies to ensure availability during builds.

Notes:

  1. Partner facing repositories are configured to:
    • not accept non fast forward commits
    • not accept deletes
    • only contain branches related to shipping versions of FxOS


Repositories Supporting Mozilla Web Products

Repository Old/New Web views at Notes
bugzilla-{bugzilla,qa}.git old Moz (r/w), github (r/o) Only the bugzilla.git & qa.git repos are mirrored to github
webtools-bmo-{bugzilla,qa}.git old Moz (r/w), webtools-bmo-* on github (r/o)

Repositories Supporting Infrastructure, Tooling, and Miscellaneous

See also full list of RelEng repositories.

Repository Old/New Web views at Notes
build-releng-api.git old github (r/o), Moz (r/o)
automation/fuzz-tools.git old Moz (r/w), github (r/o)
qa/mozmill-tests old Moz (r/w), github (r/o)

Machines

2013.10.14: We're currently using the following machines:

Machine Process Location User
vcssync1.srv.releng.usw2.mozilla.com beagle (gecko-dev) /opt/vcs2vcs vcs2vcs
vcssync2.srv.releng.usw2.mozilla.com project-branches (gecko-projects) /opt/vcs2vcs vcs2vcs
github-sync2.dmz.scl3.mozilla.com build repos (autoland, buildapi, buildbot-configs, buildbotcustom, cloud-tools, mozharness, opsi-package-sources, partner-repacks, preproduction, puppet, puppet-manifests, rpm-sources, talos, tools) /opt/vcs2vcs vcs2vcs
github-sync4.dmz.scl3.mozilla.com gecko l10n and gaia l10n /opt/vcs2vcs vcs2vcs

Plus a few other github-sync machines for legacy processes.

Branch

2013.10.14: We're based off the mozharness production branch.

Specific instructions

Emails

Successful noop emails

From: vcs2vcs@vcssync2.srv.releng.usw2.mozilla.com
Subject: [vcs2vcs] Successful conversion for project-branches (12s) <EOM>

These are pretty much there to tell you that the job's running, everything's fine, and the noop time was (in this case) 12 seconds. The <EOM> tells you that the body of the email is empty (end of message).

The noop time varies depending on hg.m.o load, as well as log upload times.

See here for enabling/disabling these... they can be pretty spammy.

Successful emails with messages

From: vcs2vcs@vcssync1.srv.releng.usw2.mozilla.com
Subject: [vcs2vcs] Successful conversion for beagle (72s)


Summary is non-zero:

info - Successfully pushed mozilla-aurora.
error - Error getting changes for mozilla-inbound; skipping!


02:16:08    ERROR -  abort: HTTP Error 500: Internal Server Error
02:16:08    ERROR -  Automation Error: hg not responding
02:16:08    ERROR - Return code: 65280
02:16:08    ERROR - Error getting changes for mozilla-inbound; skipping!

Whenever the summary of a job is non-zero, or the error log is non-zero, we'll get a message body in the success email. This is usually a successful push notification, but can include the contents of the error log or any other information we put in the summary. (At some future point we may add l10n repo creation on git.m.o in here).

In this case, the ISE from hg.m.o is an intermittent issue. In general, the script should auto-fix by clobbering+recloning, and the conversion will happen later, though it'll be delayed by this action.

Failure emails

From: vcs2vcs@vcssync-dev.build.mozilla.com
Subject: [vcs2vcs] Failed conversion for beagle

Unable to push these repos:
mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
This was a test push that failed; not proceeding any further with mozilla-beta!

Summary is non-zero:

error - Unable to push mozilla-beta. failed.


14:32:30    ERROR -  remote: error: denying non-fast-forward refs/heads/GECKO90_2011121217_RELBRANCH (you should pull first)
14:32:30    ERROR -   ! [remote rejected] GECKO90_2011121217_RELBRANCH -> GECKO90_2011121217_RELBRANCH (non-fast-forward)
14:32:30    ERROR -  error: failed to push some refs to '/opt/vcs2vcs/build/target/beagle/.git'
14:32:30    ERROR - Return code: 256
14:32:30    ERROR - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
14:32:30    ERROR - This was a test push that failed; not proceeding any further with mozilla-beta!
14:32:30    ERROR - Unable to push mozilla-beta. failed.
14:32:30    FATAL - Unable to push these repos:
14:32:30    FATAL - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
14:32:30    FATAL - This was a test push that failed; not proceeding any further with mozilla-beta!
14:32:30    FATAL - 
14:32:30    FATAL - Running post_fatal callback...

This is a failure email involving this issue. Luckily, we have a fix for that one already.

Failure emails will be sent for every failure currently, so will be very spammy until they're fixed.

Maintenance

How to add a repo to gecko-git

CAUTION: production gecko.git is still handled by legacy vcs-sync - see legacy docs.
This shouldn't be done lightly. Gecko.git is our partner-oriented repo and should be treated with kid gloves.

How to add another repo to mirror for b2g OS

Mirroring repositories for purpose of b2g os builds is handled by the legacy docs. See here.

How to add a repo to gecko-dev

gecko-dev is our developer focused repository with all gecko branches related to release-trains and inbound-branches only.

You should follow the example here. Annotated: 

"repo": "https://hg.mozilla.org/releases/mozilla-beta",  # where we should pull the repo from.
"revision": "default",                                   # what revision to pull
"repo_name": "mozilla-beta",                             # the name we'll know this repo by.  This should be unique within this job.

"targets": [{                                            
    "target_dest": "beagle/.git",                        # This is a location on disk (test_push is True)
    "vcs": "git",                                        # This was used for non-fastforward detection; might be overkill at this point.
    "test_push": True,
}, {

    "target_dest": "gitmo-beagle",                       # This is a location specified in the remote targets
                                                         # http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l332
}, {

    "target_dest": "github-beagle",                      # Also in remote_targets
}],

"vcs": "hg",                                             # The "vcs" settings are kind of glossed over at the moment,
                                                         # but when we get git<->git and git->hg going it'll matter.

"branch_config": {
    "branches": {
        "default": "beta",                               # the 'default' hg branch should become a 'beta' git branch.
                                                         # This is really needed by the all-in-one repos where there are multiple
                                                         # 'default' branches from hg that need to each be their own unique branch.
    },
    "branch_regexes": [
        "^GECKO[0-9_]*RELBRANCH$",                       # Whitelist the gecko and mobile release branches without having to
        "^MOBILE[0-9_]*RELBRANCH$",                      # list them all by name.
    ],
},
"tag_config": {
    "tag_regexes": [
        "^(B2G|RELEASE_BASE)_",                          # Whitelist the B2G and RELEASE_BASE tags.
                                                         # Explicitly avoid the release tags since they currently move in hg-land.
    ],
},

How to add a project to project-branch conversion

If the project is under hg.m.o/projects, add it here.

If the project isn't under hg.m.o/projects, add it here.

How to add a project cloning to a custom github account/project

  • Key setup
    • Generate an ssh key for pushing to the github project (no passphrase)
    • Add the key to github/org-name/project-name as a deploy key
    • Add the key to the vcs2vcs account on the vcssync server
  • Add the project to mapper for hg to git hash mapping
  • Filesystem setup on the vcssync server
    • Clone the source hg repo to /opt/vcs2vcs/vcs_sync_build/build/stage_source/
    • Clone the hg repo from stage_source to conversion 
      hg clone /opt/vcs2vcs/vcs_sync_build/build/stage_source/$PROJECT /opt/vcs2vcs/vcs_sync_build/build/conversion/$PROJECT

How to add a locale to l10n conversion

L10n conversions are based on this config file.

These point at gecko all-locales and gaia languages_dev.json files. To add a locale, update the appropriate all-locales or languages_dev.json file.

If the b2g versions are changing against gecko trains (as they will every 6 weeks), you need to update this section.

How to adjust email notifications

The notify_config's are in each config file, like this.

This is a list-of-dictionaries. Each dictionary has a "to" email address.

If failure_only is set to True, no emails with a successful outcome will be sent to this address.

If skip_empty_messages is set to True, no emails without a message body will be sent (these should only be successful noop runs with no warnings).

Allowing both of these to be False will result in a lot of email, approximately one per minute per process. Just setting skip_empty_messages to True will send email per successful push, any sort of warning, or failures.

How to force the process to pull/bookmark/convert/push a repo, even if nothing's changed

There is a --no-check-incoming commandline option to the vcs_sync.py script.

This can also be set in the config file, as check_incoming set to True.

This can also be set in the config file per-repository, as check_incoming inside each of the specific conversion_repos where we want to skip the incoming check.

To force a push manually:

  • cd into project-branches jobs top level directory
  • wait for projects.lock to disappear (from cron job)
  • touch projects.lock # grab lock to avoid race with automation
    • or: touch my_user_name.lock; while ! ln my_user_name.lock projects.lock ; do sleep 10 ; done
    • wait for return of command prompt
  • get the command line from the run_*.sh file: grep python run_*.sh
  • add the '--no-check-incoming' option to the command
  • wait for processing to complete
    • NOTE: the log output is only to your screen. The usual summary, "projects.log" is only updated via the run_* script. (see also)
      • The official logs from the last run are in the "./logs/" directory.
      • Older logs are in the "./build/upload/logs/" directory.
  • rm 'projects.lock' # let normal processing resume

Where to find the keys

The keys are the same as the keys on vcs2vcs@gd{0..4}, and has been shared on Google Drive to various RelEngers... except the passphrase has been stripped from them: ssh-keygen -p -f FILE

How to move a VCS Sync process

  • pause the cron job
  • move the work dir (or rsync, if across machines).
  • If you're on a new machine, you need to make sure it's set up properly. (Should be easier once it's puppetized)
  • make sure the cron job script is pointing at the right location
  • restart the cron job

How to start a mozilla-central based repo conversion from scratch

  • First, you need a copy of initial3.tar.bz2. This is mozilla-central-with-cvs-history, already converted to git for you, up to Spring, 2013. Without this file, you will need to run the initial conversion with this config file (or beagle) and let it run for about a week.
    • The sha512 sum is 0a3243fe5a6c8ffa4e47131e0eb0243e1f5676ea3cacd535d11b424f8f601511130a2e941670950b63c0b00726dafa9bd30bf7c3d040752fce824158021ef014
    • This file exists on vcssync{1,2} at /opt/vcs2vcs/initial3.tar.bz2 , on github-sync{2,4} at /home/asasaki/initial3.tar.bz2 , and on Aki's laptop.
  • Second, you need a config file for the conversion type. This is probably covered with the beagle, gecko-git, and project-branches config files here.
  • Clone the appropriate mozharness repo + branch (see here, unless you're working off a separate development branch). This will go into the base_work_dir. For instance, if you want to use /opt/vcs2vcs as the base_work_dir, this will be /opt/vcs2vcs/mozharness .
  • Extract the contents of initial3.tar.bz2 into the appropriate location. For instance, if I want to use /opt/vcs2vcs/ as my base_work_dir, and I'm using this config file (which specifies "gecko-git" as my conversion_dir), I'll need to move some files around.
cd /opt/vcs2vcs
tar xjvf initial3.tar.bz2  # this creates conversion/beagle
mkdir -p build/conversion
# move the extracted directory from initial3.tar.bz2 to where
# the mozharness script expects the gecko-git conversion dir to be
mv conversion/beagle build/conversion/gecko-git
rmdir conversion
  • Run the conversion once if you want to make sure.
cd /opt/vcs2vcs
# For example,
python mozharness/scripts/vcs-sync/vcs_sync.py -c mozharness/configs/vcs-sync/CONFIGFILE.py  # [--no-push] [--no-upload] [--no-notify] if desired
    • If you want to avoid certain actions, you can specify that you want to skip them. --no-push, --no-upload, and --no-notify might be some options you want to use, depending on the situation. --help and --list-actions should be helpful as well.
  • Run in cron like the other boxen.

Troubleshooting

How to deal with non-ffwd

  1. Determine what happened. This can generally be found with 
    hg heads
    . If there are multiple heads for a branch, you'll need to follow the below instructions.
  2. Determine if you want to discard one of the heads, or merge them. (Do you want the changes from both heads, or do you want one head's changes to stick and discard the other one?)
    • If you want to merge the two heads, you can do so with hg merge, hg commit, and hg push
    • If we want to discard one,
# we're on good revision X
# we have old [bad] revision Y, which is a different head 
hg debugsetparents X Y; hg commit -m "merging X and Y via debugsetparents" # Create a new changeset that has parents X and Y
hg glog | more # see that the history has merged
hg diff -r X # see that the code is the same as revision X, no diffs
hg push

If you hit a relbranch issue you may need to follow these instructions instead.

How to deal with project branch reset

To some degree, the integration-gecko-projects repo should be self-healing. If the incoming changesets to a repo are incompatible (so an hg pull doesn't work), the script should blow away the source repo and reclone. The integration-gecko-projects target will shortly have force_push set to True, so it should be pushing with git push -f.

However, project branch resets may become an issue; time will tell.

We can probably:

and it'll clone a new copy of the branch, convert, and push.

If there are still issues, we can debug further to try to get it to work. A final nuke-the-site-from-orbit solution is documented here, but may take the repo down for a day or so.

How to deal with completely resetting gecko-projects

  1. Make the decision that this is the solution to the problem.
  2. Make sure people are aware this is going to happen.
  3. Stop the conversion of gecko-projects. This is currently on vcs2vcs@vcssync2, and is run via cron.
  4. Save a copy of https://raw.github.com/mozilla/integration-gecko-projects/master/README.md
  5. Delete the repo! This option is available in the "Danger Zone" area of https://github.com/mozilla/integration-gecko-projects/settings
  6. Recreate the repo. There's a "Create a new repo" icon in the top right. This will need the same name, should be public, and you want to initialize the repo with a README. (gecko-projects is the only repo where we want the README: essentially, we want an empty master branch because github doesn't allow you to delete the first populated branch in a repo.)
  7. Edit the README with the contents you downloaded in step 4.
  8. Start the conversion back up. You may only need to delete the build/target and build/stage_source/BAD_REPO directories. Or you may need to blow away the whole thing from orbit to be sure, in which case you should follow the docs for how to start a mozilla-central based repo conversion from scratch.
    • Note, just cloning the hg repos for the projects repo can take many many hours, so this will take the gecko-projects repo down for a long time.

How to deal with GECKO90_2011121217_RELBRANCH

We were getting email with the following failure: 

11:14:01    ERROR -  remote: error: denying non-fast-forward refs/heads/GECKO90_2011121217_RELBRANCH (you should pull first)
11:14:01    ERROR -   ! [remote rejected] GECKO90_2011121217_RELBRANCH -> GECKO90_2011121217_RELBRANCH (non-fast-forward)
11:14:01    ERROR -  error: failed to push some refs to '/opt/vcs2vcs/build/target/beagle/.git'
11:14:02    ERROR - Return code: 256
11:14:02    ERROR - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
11:14:02    ERROR - This was a test push that failed; not proceeding any further with mozilla-beta!

To debug: 

# vcs2vcs@vcssync1
cd /opt/vcs2vcs/build/target/beagle
git show GECKO90_2011121217_RELBRANCH  # this gave 1003ec79451969335008880ad82e305d93b89642
cd /opt/vcs2vcs/build/conversion/beagle
git show GECKO90_2011121217_RELBRANCH  # this gave fac7279c040d643fb4c35105fa85b9335ba2c2f9
git merge-base fac7279c040d643fb4c35105fa85b9335ba2c2f9 1003ec79451969335008880ad82e305d93b89642

The merge-base returned fac7279c040d643fb4c35105fa85b9335ba2c2f9. That means that fac727 is a parent to 1003ec. No sha divergence, only stale history.

I noticed that http://hg.mozilla.org/releases/mozilla-beta/rev/GECKO90_2011121217_RELBRANCH pointed at a9221b332d8a (dec 15) and http://hg.mozilla.org/releases/mozilla-release/rev/GECKO90_2011121217_RELBRANCH pointed at 4e309e63c279 (jan 3); Calendar must have done a release off the relbranch off mozilla-release. Since the branch name is shared across mozilla-beta and mozilla-release, converting the branch from mozilla-beta effectively does a non-fastforward push and is rejected.

To fix, I exported the two revisions from mozilla-release to mozilla-beta: 

cd mozilla-release
hg export -r 79350 > ../cal2
hg export -r 79349 > ../cal1
# Doublecheck to make sure those look good
cd ../mozilla-beta
hg pull -u
hg up -r GECKO90_2011121217_RELBRANCH
hg import ../cal1
hg ident  # sha matched m-r
hg import ../cal2
hg ident  # sha matched m-r
hg out
hg push

Then update https://wiki.mozilla.org/ReleaseEngineering/VCSSync/History#Relbranch_issues .

Hints

hg glog 0..REVISION
Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/VCSSync&oldid=1196481"