EngineeringProductivity/Projects/Conduit
Contents
Project Definition
Project Conduit is a loosely coupled, extensible set of services that will provide automation and visualization of the main Firefox engineering processes, from submitting changesets for review to landing them. It is an evolution and course correction of MozReview.
The central concept of Conduit is the organization of our systems around commits. Commits are discrete units of work that exist from the initial submission of code changes from a developer for feedback, testing, and review, up to landing on integration branches and from there moving to mozilla-central and release branches. Since a commit is the final form of a code change, including metadata like date, author, summary, etc., it is easiest to work with them throughout the whole system, copying and transforming them as needed.
Conduit will include a code-review tool (Phabricator) with some custom extensions, and various services relating to submission, analysis, testing, and landing of commits.
See the Conduit road map.
See also our design docs, which may be moved to https://readthedocs.org at some point.
Tracking
Conduit is being developed in fortnightly sprints. See our Trello board.
mcote also posts updates and notes to his blog.
Code
Code for Conduit services is on GitHub in the mozilla-conduit org. Our Phabricator extensions are located in their own repo, as are our Phabricator configuration files. There is also a BMO extension for integrating with Phabricator.
Bugs and Communication
Bugs should be filed in the Conduit product on BMO.
Announcements and discussion are in the mozilla.tools newsgroup/mailing list/Google group and mozilla.dev.version-control newsgroup/mailing list/Google group.
Discussion also occurs in the #conduit channel on irc.mozilla.org and on Slack.
The Commit-based Approach
Autoland
Authorized developers should have a way to land any commits, their own or others', with the press of a button. This system should
- support Mercurial
- preserve the original commits' author and message
- retry if the destination repository is closed or unavailable (e.g. trees are closed)
- have a permission model that allows authorized users to use autoland even if they can't push directly to the destination repo
- this allows us to restrict direct-landing permission, and hence keys, to a small subset of people
- allow different sources and destinations (staging to try, staging to mozilla-central, uplifts from mozilla-central to beta to release)
Note that this system is made much simpler by having a staging repository and transplanting commits rather than applying diffs for a number of reasons:
- history is preserved so commits can be sent to try for testing without rebasing first
- all metadata is already in the staged commits (author, commit message, etc.).
- this is particularly important in our workflow because some developers use different hg author names/emails from what they use in Bugzilla
- a stack of commits can be landed at once
Furthermore, an advanced Autoland service can provide tighter integration with GitHub-hosted repositories. This is particularly important for some of the Quantum work, which requires synchronization of Servo and Gecko code.
Tracking and Visualization
In the Firefox development and release process, code changes traditionally follow this process:
- Developer pushes patch to Bugzilla
- Developer may push patch to Try for test results, which may or may not be posted to or linked from the bug
- After the patch gets an r+, developer commits and pushes to mozilla-inbound
- Sheriff merges changesets, in bulk, to the mozilla-central branch.
- A Nightly build is created which contains the changeset, amongst many others.
- The changeset is merged, along with other unrelated changesets, to the aurora branch.
- An Aurora build is created which contains the changeset, amongst many others.
- Repeat steps 5-6 for beta and release.
At any point, the changeset may also be backed out of a particular branch and later reapplied (in a modified form).
There is currently no straightforward way to track or see work "move" across the branches and builds. The combination of a central commit index plus an integrated Autoland service would provide this view and would be of use to developers and release managers.
Static Analysis and Other Reports
With a microservice architecture and a commit index, adding a new analysis step, either manually or automatically initiated, becomes straightforward. Commits can be easily pulled down to a testing server straight from the staging repo and results either posted to the bug or code-review tool, or pulled into the commit index for visualization.
Microcommits
Having a push-based approach (i.e. a staging/review repository) makes it simpler to break up a large piece of work into a series of small commits. Microcommits improve quality and reduce review time by shrinking the scope of each change. Furthermore, Mercurial's history-editing abilities mean that history is not lost when commits are edited or rebased (obsolete commits do not automatically get deleted) and that force-pushes are not required.
The flexibility of a push-based approach also allows other types of workflows, such as squashing multiple local changesets into a single changeset for review.
Loose Coupling
By separating core process automation from the work to integrate such automation with code-review and other tools, not only should it be possible to replace a tool without major effort, but improvements and bug fixes to individual services should be much easier.
High-Level Design
Conduit is designed around multiple small services, with the exception of necessarily larger tools, for example issue trackers and code-review apps. Services will generally have their own UIs, moving away from our previous model in which automation interfaces have been integrated directly into the larger tools. Standing up separate UIs for individual services, such as autoland, allows them to be used independent of bug-tracking and code-review tools and also speeds up development, freed from the confines of an extension system.
Initial services will include autoland and a commit index. Both functions currently exist but are tightly coupled with Review Board, with much of the code living in the MozReview Review Board extension. These will be split off into small, independent applications.
Alternatives
The other clear alternative is the path MozReview was previously on: continuing to develop automation services partially within Review Board, and continuing to improve core Review Board features at the same time. This has been unsatisfactory for the reasons expounded above.
Another alternative is to use a full tool suite that implements the same level of changeset-based automation that we have (push-to-review, autoland) and that we are working on or planning for (conflict detection, static analysis, etc.). Solutions in this area are few and far between and often designed with smaller applications in mind. GitHub can be seen as one such alternative, though it lacks critical features like support for confidential issues and patches and has clear integration path with our testing and CI systems. We are planning some support for GitHub-based workflows, but it cannot be a full replacement for our processes. Similarly, suites like Phabricator offer compelling issue-tracking and code-review tools that we can use in Conduit, but they lack an autoland feature that meets our requirements, as well as the flexibility offered by a repository-based system.
In general, Firefox is a sufficiently unique, complex, and highly scaled project that it is highly unlikely that there exists any off-the-shelf solution that can fully automate our engineering processes. However we are always exploring ways that we can avoid duplicating effort.
