Media/WebRTC/libwebrtc Update Process

This page describes the process for updating the in tree, vendored copy of libwebrtc using the fast-forward scripts.

Prerequisites

• NOTE: all steps below assume the user is in the bash shell.
• Create a new bug for the fast-forward update. It should depend on the previous fast-forward bug. Bug 1800920 is an example.
• Before each new merge cycle elm needs to be reset to Central.
  1. Ensure you have filed a bug for the next merge cycle, and further ensure that the next merge cycle bug has the previous merge cycle bug as a dependency.
  2. File a bug as a clone of Bug 1822294 and add your new bug as dependency to the next merge cycle bug.
  3. Wait for the repo admins to reset the repository.
  4. Celebrate
• Builds are done after vendoring in each commit. A minimal, default .mozconfig is provided during the prep_repo step below.
• Running the fast-forward process from macOS and Linux is supported. For more information on macOS, see here and here. TL;DR: If running from macOS, avoid Xcode and git while scripts are running, and you may also need to run this command ahead of time:

git config --global core.trustctime false

• Starting with a fresh clone of elm is recommended for each new chromium milestone.

hg clone --stream ssh://hg.mozilla.org/projects/elm

• Verify the most recent commit is the .arcconfig update. If the most recent commit doesn't look like "Bug xxx - FLOAT - REPO-elm - update .arcconfig repo callsign", run the following commands:

 ./mach python dom/media/webrtc/third_party_build/restore_elm_arcconfig.py
 hg push -r tip

Update steps

• Start from the elm checkout

cd elm

• Update the default config file for the next upstream milestone

bash dom/media/webrtc/third_party_build/update_default_config.sh

• Prepare the git repository - Running prep-repo.sh will display instructions on how to add recent moz-central changes made in third_party/libwebrtc to the top of the patch stack in our git repository. This is expected and necessary for successfully vendoring changes into third_party/libwebrtc.

bash dom/media/webrtc/third_party_build/prep_repo.sh

• Prepare no-op tracking files for cherry-pick commits

bash dom/media/webrtc/third_party_build/build_no_op_commits.sh

• Save the newly updated patch-stack - run this in a bash shell

(source dom/media/webrtc/third_party_build/use_config_env.sh ; \
 ./mach python dom/media/webrtc/third_party_build/save_patch_stack.py \
 --repo-path $MOZ_LIBWEBRTC_SRC \
 --target-branch-head $MOZ_TARGET_UPSTREAM_BRANCH_HEAD \
 --separate-commit-bug-number $MOZ_FASTFORWARD_BUG
)

• bootstrap and sanity build

./mach --no-interactive bootstrap --application-choice=browser && ./mach build

• Run loop-ff.sh

bash dom/media/webrtc/third_party_build/loop-ff.sh

• Push to elm - after the loop-ff.sh script completes with the message ===loop-ff=== Completed fast-foward to {short-sha}, run the following step to push the new version of libwebrtc to elm:

hg push -r tip

Operational Notes

Reacting to loop-ff.sh

Three main types of errors will cause loop-ff.sh to exit: the first is a git rebase conflict, the second is a failure to regenerate build files (most often due to BUILD.gn changes or missing/added files), and the third is a build error due to API changes in upstream code.

• When fixing rebase errors or making other changes to fix build issues in moz-libwebrtc (any changes under third_party/libwebrtc), mercurial commit messages follow the pattern:

Bug {current-fast-forward-bug-number} - (fix-{upstream-sha}) {description}

# after following all the other instructions from loop-ff.sh (running the
# generation script for moz.build files, committing possible moz.build 
# changes and doing a build), run the following command:
(source dom/media/webrtc/third_party_build/use_config_env.sh ; 
 ./mach python dom/media/webrtc/third_party_build/save_patch_stack.py \
  --repo-path $MOZ_LIBWEBRTC_SRC \
  --target-branch-head $MOZ_TARGET_UPSTREAM_BRANCH_HEAD )

• If instructed to run commit-build-file-changes.sh, its auto-generated mercurial commit messages follow the pattern:

Bug {current-fast-forward-bug-number} - (fix-{upstream-sha}) {description} - moz.build file updates

• When making changes to fix Mozilla code (code outside of third_party/libwebrtc) in response to changes upstream, mercurial commit messages follow the pattern:

Bug {current-fast-forward-bug-number} (MOZ) - {description}

If multiple kinds of changes are needed to fix build errors, commit hygiene is a must: (MOZ) patches contain changes to files outside of third_party/libwebrtc, (fix-xxx) commits contain changes to third_party/libwebrtc files, and finally "(fix-xxx) ... - moz.build file updates" contain only third_party/libwebrtc moz.build changes.

General tips

• Stopping the loop-ff.sh manually is best done when in the Test build phase using Ctrl-c.
• Rather than waiting until the end of the process update process, you may push to elm to confirm build fixes, or for others to follow along. It is best to push a commit that successfully builds, but this isn't a hard and fast rule. Sometimes, it may be helpful to push a commit that is broken to get help from others, or when the fixes are complicated enough that you feel a review is necessary.
• If loop-ff.sh reveals an issue that is best dealt with in a followup, one can commit a temporary patch marked with REPO-elm in the first line of its description, i.e. the "title". The temporary commit allows loop-ff.sh to continue, whereas REPO-elm makes sure the commit cannot be pushed to any other mozilla repo than elm (and try).
• While loop-ff.sh is running, in a separate terminal window it may be helpful to see the overall progress by running:

tail -f .moz-fast-forward/logs/log-loop-ff.txt | grep loop-ff

• GYP generated Java files
• If instructed to run extract-for-git.py, note that loop-ff.sh has more context than verify_vendoring.sh and offers information that is relevant: Specifically, the guidance about tip::tip:

   If you have previously run loop-ff.sh successfully, there may be a new
   change to third_party/libwebrtc that should be extracted and added to
   the patch stack in github.  It may be as easy as running:
       ./mach python dom/media/webrtc/third_party_build/extract-for-git.py tip::tip
       mv mailbox.patch /Users/jan-ivar/moz/elm/.moz-fast-forward/moz-libwebrtc
       (cd /Users/jan-ivar/moz/elm/.moz-fast-forward/moz-libwebrtc && \
        git am mailbox.patch)

Replace tip here with your (fix-xxx) mercurial patch, if it is not at tip (e.g. if it is followed by (fix-xxx) - moz.build file updates).

Recovering from errors

• If a particularly complicated rebase conflict is encountered, it is possible to start over the most recent vendoring step by resetting some state and restoring the patch stack. Run these commands in a bash shell:

rm .moz-fast-forward/*.resume # resets the resume logic in loop-ff.sh
hg revert third_party/libwebrtc && hg purge third_party/libwebrtc
./mach python dom/media/webrtc/third_party_build/restore_patch_stack.py --repo-path .moz-fast-forward/moz-libwebrtc

Review

• Simple changes don't require individual review, but if you'd feel better about getting reviewed, push the vendored commit to elm (which may break all the builds). Then put the patch up for review. After r+, land it using Lando to elm.

Waiting to land

Daily rebase on elm (after completing loop-ff process)

Periodically, ideally daily to minimize surprises from mozilla-central, rebasing elm onto the latest mozilla-central is recommended after completing the update process. Rebases should be completed until the request for merging elm to moz-central is made.

Historical note: we use a script to rebase because 1) it regenerates the moz.build files on the commits where those are changed. This avoids the typically massive rebase conflict if anyone has touched the build files. 2) it exports/imports each commit individually to limit rebase conflicts on individual patches, making it easier to deal with issues.

bash dom/media/webrtc/third_party_build/elm_rebase.sh

If one would like to make changes to the patch ordering during the rebase operation (say, for example, a fix-commit was necessary but is separated from the initial upstream patch), use the following command line:

STOP_FOR_REORDER=1 bash dom/media/webrtc/third_party_build/elm_rebase.sh

Requesting merge from elm to mozilla-central

Typically, immediately after the code-freeze has lifted, a request is made for the sheriffs to merge elm to mozilla-central. This done via email, recently to Sebastian Hengst.

Post-merge Steps

• After the merge from elm to mozilla-central is complete, the moz-libwebrtc github repo should be updated with the new branch of the commit stack for the new release. There is script that will make the proper branch name and do the push:

bash dom/media/webrtc/third_party_build/push_official_branch.sh