Media/WebRTC/libwebrtc Update Process/automation plan
Note: If you are trying to do a fast-forward update please go here.
Note: If you are trying to do a fast-forward update please go here.
Note: If you are trying to do a fast-forward update please go here.
This describes the rough plan to document and automate the process to "fast-forward" through webrtc.org libwebrtc change-sets to bring Firefox's libwebrtc version as close as possible to Chrome. This page will be updated as new information is discovered.
Contents
- 1 Plan Overview
- 2 Findings
- 2.1 current third_party/libwebrtc directories and status
- 2.2 verify generation/build with extra, unused directories
- 2.3 verify current vendoring scripts cmd-line params
- 2.4 build instructions for moz-libwebrtc
- 2.5 remove dead/unused files
- 2.6 patches after vendoring moz-libwebrtc into moz-central
- 2.7 patches on top of upstream github moz-libwebrtc commit
- 3 Keeping transitive dependencies updated
- 4 Open Questions
- 5 Moving third_party/libwebrtc/third_party/* to third_party/*
Plan Overview
- repo tasks
- - update moz-libwebrtc fork to latest webrtc.org
git checkout master git remote add upstream https://webrtc.googlesource.com/src git fetch upstream git merge upstream/master # fast forward merge git push origin master
- - update elm to latest moz-central
- verify current third_party/libwebrtc directories
- verify generation/build with extra, unused directories from stock moz-libwebrtc in place
- verify current vendoring scripts cmd-line params
- - check moz-central a93c2d1df066 - Bug 1654112 - Vendor libwebrtc for rel86
- - check moz-central d1df4970f4f0 - Bug 1654112 - Vendor libwebrtc/third_party for rel86
- - check moz-central 587c40771588 - Bug 1654112 - Vendor libwebrtc/build for rel86
- verify build instructions for moz-libwebrtc
- remove dead/unused files in third_party/libwebrtc
- identify patch set for changes after vendored moz-libwebrtc (587c40771588)
- identify patch set for changes to upstream github revision (edd7f9e94d)
- Manually identify how to fast-forward one changeset
- - move one changeset forward in webrtc.org history
- - verify local build in moz-libwebrtc
- - apply moz-libwebrtc-to-upstream patches, verify local build
- - vendor moz-libwebrtc
- - apply moz-central patches
- - full try run
- write automation scripts "Manually identify how to fast-forward one changeset"
- investigate updatebot
Findings
current third_party/libwebrtc directories and status
- Additional vendoring
build (with some changes during vendoring) third_party
- Mozilla added
README.mozilla - generated during vendoring process moz.build - generated by generate-gn-build-files.sh webrtc_gn - generated by generate-gn-build-files.sh X11 - imported later by Byron in c91f12b557a1 D113830 tools/grit - imported later by Nico in 3cce5e6938f0 D114027
- Mozilla vendored after a93c2d1df066
- - Both of the following were inadvertently vendored from moz-libwebrtc commit bd4a718667. The only meaningful difference is in sdk/objc/helpers/RTCDispatcher.{h|m}, but these files are not currently used.
sdk/BUILD.gn - imported later by Byron in 9314046d89eb D105015 sdk/objc - imported later by Byron in 9314046d89eb D105015
- - We don't build the sdk directory (see here).
- - Only sdk:helpers_objc is used (see here)
- - From sdk/objc/helpers, only sdk/objc/helpers/scoped_cftyperef.h is used:
third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_iosurface.h third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_provider.h third_party/libwebrtc/modules/desktop_capture/mac/screen_capturer_mac.mm third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_cgimage.h
- Untouched from Bug 1665166 - Move media/webrtc/trunk/* to third-party/libwebrtc
README.md (should have been vendored) DEPS (should have been vendored) google_apis (is this referenced/used by current code?) tools/clang (is this referenced/used by current code?) chromium_deps dummy_file.txt net Makefile.old peerconnection_client.target.mk peerconnection.Makefile
verify generation/build with extra, unused directories
- Add missing directories
cp -r ~/git-checkouts/moz-libwebrtc/test ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/pc ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/resources ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/docs ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/tools_webrtc ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/examples ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/p2p ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/rtc_tools ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/data ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/style-guide ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/stats ~/mozilla/moz-central/third_party/libwebrtc/ cp -r ~/git-checkouts/moz-libwebrtc/media/sctp ~/mozilla/moz-central/third_party/libwebrtc/media
- local build file generation produces no build modified build files
- Full build results here using ./mach try fuzzy -q 'build-'
- Full linux test results here using ./mach try fuzzy -q 'linux-'
verify current vendoring scripts cmd-line params
- Vendor libwebrtc (a93c2d1df066)
- - The following commands produce a93c2d1df066 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit: File:Vendor-fixup.patch.zip
cd dom/media/webrtc/third_party_build python3 vendor-libwebrtc.py \ --from-github https://github.com/mozilla/libwebrtc \ --commit 149d693483e9055f574d9d65b01fe75a186b654b \ libwebrtc
- Vendor libwebrtc/third_party (d1df4970f4f0)
cd dom/media/webrtc/third_party_build python3 vendor-libwebrtc.py \ --from-googlesource \ --commit 5dc5a4a45df9592baa8e8c5f896006d9193d8e45 \ third_party
- Vendor libwebrtc/build (587c40771588)
- - The following commands produce 587c40771588 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit: File:Vendor-build-fixup.patch.zip
cd dom/media/webrtc/third_party_build python3 vendor-libwebrtc.py \ --from-googlesource \ --commit ac22062b465485d3e1196ae9c506b3617cc16fb5 \ build
build instructions for moz-libwebrtc
Note: These instructions do not currently work for Ubuntu 22.04, but are confirmed to work on 20.04.
- for a freshly built Ubuntu machine, you'll need to do these prerequisite steps:
sudo apt update sudo apt install python2 curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py python2 get-pip.py
- verify build for webrtc.org
- - Note: we can build as far forward as eacbd972ab (HEAD, branch-heads/4287, branch-heads/4286, branch-heads/4285, branch-heads/4284) without needing to gsync
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git export DEPOT_TOOLS=`pwd`/depot_tools (cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 ) export PATH=$DEPOT_TOOLS:$PATH ; \ export DEPOT_TOOLS_UPDATE=0 ; \ export DEPOT_TOOLS_WIN_TOOLCHAIN=0 mkdir webrtc-checkout cd webrtc-checkout fetch --nohooks webrtc # populates .gclient and .gclient_entries # on fresh Ubuntu install, do: ./src/build/install-build-deps.sh (cd src && git checkout edd7f9e94d) gclient sync cd src gn gen out/Default ninja -C out/Default ./out/Default/rtc_media_unittests # quick smoke test
- verify build for moz-libwebrtc
- - Use edd7f9e94d because some our patches stacked on top of this upstream revision are specific to building in the moz-central tree.
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git export DEPOT_TOOLS=`pwd`/depot_tools (cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 ) export PATH=$DEPOT_TOOLS:$PATH ; \ export DEPOT_TOOLS_UPDATE=0 ; \ export DEPOT_TOOLS_WIN_TOOLCHAIN=0 mkdir moz-libwebrtc-checkout cd moz-libwebrtc-checkout gclient config https://github.com/mozilla/libwebrtc --name src --unmanaged gclient sync -D --force --reset --with_branch_heads # ~30 min cd src git checkout edd7f9e94d gn gen out/Default ninja -C out/Default ./out/Default/rtc_media_unittests # quick smoke test
remove dead/unused files
- Successfully removed:
chromium_deps dummy_file.txt net Makefile.old peerconnection_client.target.mk peerconnection.Makefile
tools/clang
patches after vendoring moz-libwebrtc into moz-central
hg diff -r 587c40771588 -r central \ --exclude '**/moz.build' \ --exclude 'third_party/libwebrtc/X11' \ --exclude 'third_party/libwebrtc/sdk/BUILD.gn' \ --exclude 'third_party/libwebrtc/sdk/objc' \ --exclude 'third_party/libwebrtc/tools/grit' \ --exclude 'third_party/libwebrtc/build' \ --exclude 'third_party/libwebrtc/third_party' \ third_party/libwebrtc
- - While the above command works, when used within the fast foward script it requires reapplication of fixes for merge conflicts with each new advance in the webrtc.org changesets. It makes more sense to reuse the extract-for-git.py script to build a patch set of changes we can apply on top of our current patch stack residing in moz-libwebrtc. That command looks like:
cd dom/media/webrtc/third_party_build && \ python3 extract-for-git.py 587c40771588::2b187d932ca1
- - where
587c40771588
is the moz-central vendoring commit and2b187d932ca1
is the current tip of moz-central - - extract-for-git produces a file named
mailbox.patch
which can be applied in moz-libwebrtc with:
git am mailbox.patch
- - Note, fixes were required during the
git am
process. These were dealing with a couple places were we imported directories into moz-central from webrtc.org after the vendoring commit, and a couple cases of Windows vs Linux line endings causing conflicts.
patches on top of upstream github moz-libwebrtc commit
- easiest way
git diff -r edd7f9e94d -r mozilla-modifications-rel86
- more useful for rebasing on top of upstream commits
git branch mozpatches edd7f9e94d git checkout mozpatches git format-patch -k edd7f9e94d..mozilla-modifications-rel86 git am *.patch # applies to branch mozpatches
- in both cases, due to an anomaly during the previous vendoring operation an additional patch must be applied to match the vendored commit in moz-central
curl -o vendor-fixup.patch.zip https://wiki.mozilla.org/images/2/29/Vendor-fixup.patch.zip unzip vendor-fixup.patch.zip (cd third_party/libwebrtc && patch -p1 < vendor-fixup.patch)
Keeping transitive dependencies updated
- we will use updatebot to keep our dependencies updated. This may happen after an initial manual update.
- libyuv
- libvpx
- pipewire
Open Questions
- How to see release branches in webrtc.org?
Add "fetch = +refs/branch-heads/*:refs/remotes/branch-heads/*" to .git/config and run 'git pull'.
- How to find versions of webrtc.org that ship with Chrome?
If you have acquired webrtc.org/src using gclient sync (instructions), there are branches listed that look like 'branch-heads/nnnn'. You can cross-reference the Chrome release and find the webrtc.org release here: https://chromiumdash.appspot.com/branches example: Chrome's 98 release uses webrtc.org 4758, which is sha a6b138d6b4
- Do we want to advance commit-by-commit?
I think we may need to do this to avoid large-scale breakage and make it easier to fix issues.
- Do we want to manually advance the process as quickly as possible, and then modify our scripts to start taking advantage of update-bot?
This allows a more finely tuned process to help us get caught up, but pushes out the integration with update-bot.
- How/when do we start integrating webrtc.org's unittests into our try runs?
This may need to happen in our tree (probably post-vendoring) because the process of getting the webrtc tree buildable is a long process requiring lots of network.
- Do we want to add our fixup-patch to the moz-libwebrtc branch and move the branch forward in the official repo to make it more clear what was previously vendored?
Moving third_party/libwebrtc/third_party/* to third_party/*
Motivation
(lib)webrtc and Firefox share a number of dependencies. Both projects expect that their dependencies will live under a `third_party` in their respective project roots. As a dependency of libwebrtc, this presents an organizational problem. Additionally there are libraries that are expected to exist in the deployment environments. This can pose a problem where Firefox supports systems with older versions of those libraries than libwebrtc does. Resolving this cluster of issues would directly impact a number of our WebRTC team Q4 goals.
- Reduce the number of libwebrtc patches, with an emphasis on build patches
- Reduce the merge process complexity
- Reduce our response time to security issues
- Uplifting patches
Additionally, resolving these issues would assist with broader product goals.
- Improve performance
- Reduce build time
Patch Reduction
We carry a number of libwebrtc build system patches. These patches create numerous merge conflicts each merge cycle. Problematically, Google is unable to perform CI builds with our build_with_mozilla flag. They have expressed interest in removing this flag. In order to remove this flag entirely we need to eliminate our build patches. We could then express our build as a series of feature flags within their build system. Perhaps they would be willing to run that flag set in their CI.
Merge Complexity
Updating the dependencies is tied directly to the merge process. Decoupling can be achieved by moving the dependencies into `third_party` and placing them under control of `updatebot`. This would create a smaller merge with less risk.
Approach
We will need to employ several strategies in order to create a robust build environment.
- We will vendor the libraries that libwebrtc requires, but are not provided on our older supported platforms. These dependencies can be kept up to date with `updatebot`.
- We will move vendored shared dependencies from `libwebrtc/third_party` into `third_party`.
As simple as this plan sounds it is complicated by mixed build systems and the current method of importing dependencies into libwebrtc. Eliminating duplicate dependencies also has the risk of running into incompatible version expectations by dependency consumers.
Notes
- (lib)webrtc dependency pinning
- Chromium Dependencies
- gclient setup
- Nika's bazel hack
- (lib)drm
- libgmb
- abseil-cpp
- libyuv
- libvpx
- libdav1d
- libaom
- Linux shared library intricacies
- dlopen
- Use in FireFox https://searchfox.org/mozilla-central/search?q=dlopen%28&path=&case=true®exp=false
- Use in Chromium https://source.chromium.org/search?q=dlopen
- Loading pipewire https://searchfox.org/mozilla-central/rev/0a2eba79c24300ce0539f91c1bebac2e75264e58/third_party/libwebrtc/modules/desktop_capture/linux/wayland/moz_base_capturer_pipewire.cc#32
- https://linux.die.net/man/3/dlopen
- Ubuntu 18.04 LTS support
- RedHat modifications
- Vendoring headers
- Finding versions of loaded libraries