Sheriffing/How To/Getting started as a sheriff

From MozillaWiki
Jump to: navigation, search

So you want to be a Sheriff? Here's how to start.

Things You'll Need

Not all of these have to be ready on Day One, but eventually you'll need the following.

  • Join the Element chat for communicating with developers, fellow sheriffs, and others.
  • Clone of the Mercurial repositories, for committing changes to the repositories.
  • A properly configured mercurial setup. You can use ./mach bootstrap from your clone of the repository to run through a setup wizard. I recommend setting up the qbackout extension to aid in easily backing out patches.
  • LDAP account, for signing in to Treeherder and pushing commits to the source code repositories.
  • A modern browser, to access Treeherder, Bugzilla, and other Mozilla web tools.
  • A Bugzilla account: Bugzilla is Mozilla's primary bug tracker.
    • After the account got set up, Email Preferences, scroll down to User Watching, enter sheriffs@mozilla.bugs and click Submit Changes at the bottom. This pseudo email address/Bugzilla account gets used by developers to e.g. request backouts from the sheriff on duty and watching it will notify you.

Set up Mozilla SSH Key, mercurial and mozilla-unified repo, test push to try

Mozilla has a server for testing patches before they are pushed to the integration repos. Practically, every developer can test their patch without breaking the repo. You can test the functionality of a patch by creating the job in treeherder using Mercurial. For that the following need to be done:
IF YOU DO NOT HAVE A SSH KEY in https://login.mozilla.com/ssh_keys_index then:

  • Generate a new ssh key for Mozilla. Preferably, the key should be generated in a new Ubuntu virtual machine (VM).
  • After you created the VM User in Ubuntu, log in that account with putty and generate the private and public keys using the following command: ssh-keygen -t rsa -b 4096 -C "Mozilla key". The above command will generate id_rsa and id_rsa.pub in ~user/.ssh/
  • At Enter passphrase you need to type in a password which you will not forget!
  • After generating the key, the content of id_rsa.pub needs to be added here: https://login.mozilla.com/ssh_keys_index . If you already have a key, make sure the public one is in https://login.mozilla.com/ssh_keys_index and the private one in the new Ubuntu user, in id_rsa.
  • After creating/setting the key, create the following file ~user/.ssh/config and add the following lines (in the email section add your Mozilla email address):
    • Host hg.mozilla.org
    • User user@mozilla.com
    • IdentityFile name of they key file without the .pub (e.g. id_rsa_mozilla_2024-03-11)
  • For additional debug info, ssh -vvv hg.mozilla.org is often helpful. This should work once your request for level 1 commit access has been granted.
  • Configure the identity file to request the password once per session:
    • cd ~/.ssh
    • ssh-add name of they key file without the .pub (e.g. id_rsa_mozilla_2024-03-11)
Installing Mercurial

Mozilla recommends using the latest Mercurial version. We’ll need to install it. If you have any other version installed you can uninstall it with apt-get remove mercurial and install the new version using the following commands (with root):

  • apt-get install python-pip -y
  • pip install Mercurial

If installation fails, used the bootstrap.py install script provided by the mozilla source code directory:

This will also clone the source code repository.

Setting up the mozilla-unified repo

Now that you have Mercurial installed, we will make a clone of mozilla-unified.
Log out of root, remain on your new user account and run the following command in the home of your current user: hg clone --uncompressed https://hg.mozilla.org/mozilla-unified . Wait until the download is finished (about 5-10 min) and the mozilla-unified folder will be created.

Setting up Mercurial for Mozilla
  • After Mozilla-Unified has been downloaded, access the folder (cd mozilla-unified) and run the below command: ./mach mercurial-setup.
  • In the name and email sections, type your Mozilla account details, for all other questions just answer Y.
  • After you’ve finished the above setup, download .hgrc.txt file and save it in /home/user under the name .hgrc . Overwrite the file with the existing one and edit it: type your name and surname and your Mozilla user in the username section. Also, in the lower part of the file, type only your username.
  • Access the mozilla-unified folder and download autoland repo using this command: hg pull autoland
  • After the above command finished downloading autoland, edit .hg/firefoxtrees and delete all the lines except autoland, inbound, central, active esrs (check the ones available on Treeherder), beta and release.


Note: If only AUTOLAND is in that file, you need to run the following command: hg pull beta && hg pull central && hg pull inbound && hg pull esr
If you’ve come this far, you only need to run the following commands:

  • hg pull fxtrees
  • hg update <repo> //repo = autoland/inbound/central/esr/release/beta

Check the content of the .hgrc file in the home folder, it shall looks similar to this: 

[ui]
username = Your Name <yname@mozilla.com>
merge = internal:merge
editor = nano
interface = curses

[alias]
trymc = push -f ssh://hg.mozilla.org/try

[defaults]
commit = -v
diff = -U 8
qdiff = -U 8
qnew = -U

[diff]
git = 1
showfunc = 1
unified = 8

[extensions]
blackbox = 
bzexport = /home/user/.mozbuild/version-control-tools/hgext/bzexport
firefoxtree = /home/user/.mozbuild/version-control-tools/hgext/firefoxtree
hgext.churn = 
histedit = 
mozext = /home/user/.mozbuild/version-control-tools/hgext/mozext
progress = 
purge = 
push-to-try = /home/user/.mozbuild/version-control-tools/hgext/push-to-try
rebase = 
strip = 
transplant = 
absorb = 
# js-format = /home/user/.mozbuild/version-control-tools/hgext/js-format
# format-source = /home/user/.mozbuild/version-control-tools/hgext/format-source

[hooks]

[experimental]
clonebundles = true

[bugzilla]
username = yname@mozilla.com
Test push to try repo

After the setup is finished, run the following test command: ./mach try -b o -p linux -u mochitest-1 -t none
!after a change of ssh keys, if you hit the following error when pushing to try: Remote: sign_and_send_pubkey: signing failed: agent refused operation then you’ll need to delete the current ssh agent and add the new one using these commands:

  • ssh -add -d
  • ssh -add

Afterwards you can run ./mach try -b o -p linux -u mochitest-1 -t none
This command will create a push in treeherder, and if all goes well, that push will appear here: https://treeherder.mozilla.org/#/jobs?repo=try . The output of the test should look like this: 

./mach try -b o -p linux -u mochitest-1 -t none
Creating temporary commit for remote...
pushing to ssh://hg.mozilla.org/try
Enter passphrase for key '/home/arny/.ssh/id_rsa':
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 0 changes to 0 files (+1 heads)
remote: recorded push in pushlog
remote:
remote: View your change here:
remote:   https://hg.mozilla.org/try/rev/a9e732c8d4df66b2961f476652765ab451dcdeae
remote:
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=try&revision=a9e732c8d4df66b2961f476652765ab451dcdeae
remote: recorded changegroup in replication log in 0.008s
push complete
temporary commit removed, repository restored

The Sheriffs How-To

A lot of knowledge about what sheriffs do is built up by day-to-day experiences on the job. We've created a number of wiki pages documenting a lot of this information, indexed here.

Some of the more helpful (and up to date) pages:

  • Sheriffing/How:To:Treeherder - Explains some useful terminology, what each part of Treeherder's UI means/does, and how to use Treeherder to classify failures
  • Sheriffing/How:To:SheriffingFromUnifiedRepos - This is useful, though some of the referenced repositories are no longer sheriffed, and it assumes you have knowledge of a lot of Mozilla/mercurial terms.

Communication

Sheriffs hang out in the sheriffs channel on Matrix - also it's useful to be in the following Channels:

  • CI Automation: For questions about test automation.
  • developers: To find developers or e.g. provide status updates on longer tree closures
  • Firefox CI: Escalate e.g. issues with hardware backlog here - the ones in the 'Hardware' table of the backlog dashboard
  • release-drivers: Find the Release Managers (the people who approve beta uplifts) here and ask questions or escalate issues related to that
  • releaseduty: For issues with tasks belonging to a release, e.g. the 'UV' ones on beta etc.
  • releng: For general issues with the release build infrastructure
  • Taskcluster - best to check #taskclouster-cloudops on Slack before if the Taskcluster system got updated
  • treeherder: The website to manage build, test and performance results

Also join the sheriffs mailing list.

Sheriffed Repositories

Mozilla has dozens of different source code repositories for various projects and releases, but sheriffs only need to care about a few of them:

  • mozilla-central, the main development repository.
  • mozilla-inbound, one of mozilla-central's integration branches where new commits initially land for testing.
  • autoland, another integration branch for mozilla-central.
  • mozilla-beta, one of the release stabilization branches for code that has already gone through several weeks of testing.
  • mozilla-release, the official release branch, where Firefox code goes to become part of an official build of Firefox.
  • mozilla-esrXX, one or more branches set up to track patches for the current (and at times when the ESR releases overlap, the previous) version of Firefox ESR. (At the moment, this is mozilla-esr52.)

Once you're up to speed, it's best to leave a tab open to each of these repositories throughout your work day so you can track failures as they come in.

Sheriffing Tasks

Sheriffs do a lot of things throughout the day.

  • Watch the sheriffed repositories via Treeherder for failures.
    • Classify/Star known intermittent failures.
    • File tracking bugs for newly discovered intermittent failures.
    • Resolve issues with code that has landed. This can be done with some of the following:
      • Backout the bad code.
      • Ensure that developers land followup patches to fix issues that are discovered.
    • Monitor the repositories and tests for any issues related to infrastructure issues.
    • Mark repositories as being unable to accept commits from developers (also called "close the trees") via Treestatus, allowing you to fix issues without a bunch of new potentially-bad commits from landing in the interim.
  • Merge known-good commits from the mozilla-inbound and autoland repositories over to mozilla-central, and then back around to mozilla-inbound and autoland.
    • Once the actual merge is performed, you need to run the Bugherder tool against the merge on mozilla-central to get the information properly updated in Bugzilla. There's a link to Bugherder in the per-push menu item in Treeherder.
  • Landing approved patches that are marked as "checkin-needed" onto mozilla-inbound or autoland.
  • Backporting approved patches (also called "uplifting") from mozilla-central to one or more of the release stabilization branches (mozilla-beta, mozilla-release, mozilla-esr52, etc) to get the patch into users' hands sooner.

Permissions

The following permission have to be set for the mentioned actions:

  • Subscribe to sheriffs mailing list
  • commit access level 1 (scm_level_1 LDAP group), requested as documented here:
    • save job classifications
    • push to Try server (test patches, do beta simulations etc.)
  • treestatus_users membership with no expiration
    • change the status of the trees (e.g. close on test failures or infrastructure issues)
  • commit access level 3 (scm_level_3 LDAP group), requested as documented here:
    • push to mozilla-central etc. (not autoland or modify servo/, see below)
    • land patches from within the browser with Lando/Phabricator
    • retrigger jobs
    • request to run job on previous pushes where it might have been skipped ("Backfilling")
    • request to add selected jobs to a push
  • scm_allow_direct_push LDAP group
    • push directly to trees without using Phabricator/Lando to land backouts
  • scm_autoland LDAP group:
    • push directly to autoland (needed for merges and some backouts)
  • scm_servo_vendor LDAP group:
    • allow to changes files in the servo/ folder (needed for merges and some backouts)
  • sheriff LDAP group:
    • terminate machines running tests
    • cancel all jobs on non-Try pushes
    • manually request Nightlies (?)
  • is_staff set for user account by Treeherder admins:
    • request to run all missing jobs from menu for a push
    • request to run all missing Talos jobs from menu for a push
  • document edit access
  • Element chat
    • set as Moderator in sheriffs channel
    • add to sheriffs community (will show the sheriffing star next to messages from people in this group)
Retrieved from "https://wiki.mozilla.org/index.php?title=Sheriffing/How_To/Getting_started_as_a_sheriff&oldid=1250767"