ReleaseEngineering
Contents
- 1 Quick Links For You
- 2 Quick Links For Us
- 3 About Us
- 4 Contacting Release Engineering
- 5 Contributing
- 6 Team
- 7 Releng Offerings
- 8 Development
- 9 Operations
- 10 Releng Applications
- 11 Mozilla Applications
- 12 External Third Party Dependencies
- 13 Deprecated
- 14 Subpages
- 15 Run the tasks to staging pools (OSX/Linux)
Quick Links For You
Quick Links For Us
About Us
We ship Firefox to the world!
Release Engineering does this by:
- Developing world-class, scalable, open, secure, and maintainable infrastructure for application builds and tests, on a diverse set of desktop and mobile platforms
- Building fast and reliable pipelines to transform source code into shipped products
Contacting Release Engineering
Our team is located on 2 continents and in 5 time zones. This allows us to provide better service, especially if you contact us using one of the following methods:
- To request a new service, please file a bug
- To ask a general question, ping in #firefox-ci on Matrix, or email release@mozilla.com
- To request status on an existing request, set the "needinfo" flag in the bug.
Our team works normal business hours for our respective time zones. If you need to escalate an issue that is both urgent and important, use the following methods:
- Use #firefox-ci on Matrix or email release@mmozilla.com, starting your message with 'URGENT:'. If no response in 15 minutes, then
- Be sure the bug severity is set to "S1"
- Call Johan Lorenzo (Paris time).
If you get voice mail, please leave your callback number.
Contributing
If you are interested in getting involved with Release Engineering, please see our contributing page! We'd love to hear from you.
Team
Mozilla's Release Engineering:
- twitter: @MozRelEng
- blogs: Planet Release Engineering
- calendar: Google Calendar
- mailing list: release-engineering@lists.mozilla.org
Name | Profile | Blog | |
Ben Hearsum | bhearsum | @bhearsum | Ben's blog |
Interns
Releng Offerings
As mentioned above, Releng offer pipelines to transform source code into shipped products. The following sections break down how that happens. Each section can be understood as both isolated and combined parts.
Initiating a Release
To start a release, Release Management submit a form in Ship It with what product they would like to release, what revision to use, and a number of other inputs.
Configuring & Scheduling
Ship It then creates Taskcluster tasks that are needed to take a builds and release them to users. These tasks make up the release graph that are derived from templates within taskgraph and are submitted via action tasks.
Signing
The signing of build artifacts and binaries. Some features of this:
- signing is done through a client/server model
- integrated with Taskcluster thanks to signingscript in conjunction with scriptworker, a securely managed Taskcluster worker type. signingscript tasks are part of the task graph that is generated from Ship It
- keys are generated with modern standards
- keys are secured by our infra and rotated regularly
- coming soon: self served signing through exposed API
Uploading
As important release artifacts are created, they are then uploaded to archive.m.o where they can served from and live permanently for historical or longstanding support reasons.
Like signing, uploading is integrated with Taskcluster thanks to beetmoverscript in conjunction with scriptworker, a securely managed Taskcluster worker type. signingscript tasks are part of the task graph that is generated from Ship It
Install & Update Tests
Like product tests, releases likewise need tests. This ensures that initial downloads install correctly on all supported platforms and updates work as expected, all before actually releasing. Right now this is done through our Update and Final Verify tests. Which, like all other sections are part of the release graph.
TODO: link to documentation and source of these
Shipping & Updates
Finally, we release the product via both initial downloads and as new updates.
For Desktop Firefox:
- Initial installs are served via updating the download page, mozilla.com (Bedrock) and updating Bouncer, Mozilla's mirroring and redirection server.
- Updates are handled by our own application, Balrog. Balrog is our server side component of the update system
- this is integrated with Taskcluster thanks to balrogscript in conjunction with scriptworker, a securely managed Taskcluster worker type. balrogscript tasks are part of the task graph that is generated from Ship It
For Android Products, including Firefox:
- releases are submitted to the Play Store
- this is integrated with Taskcluster thanks to pushapkscript in conjunction with scriptworker, a securely managed Taskcluster worker type. pushapkscript tasks are part of the task graph that is generated from Ship It
Development
Best Practices
- Python Standards - Our Python coding standards
- Development Best Practices - read this when you're starting something new
- Deployment - How we deploy our tools, and how do we update them.
- Wiki Guidelines - how to wiki garden for fun and profit
Bugzilla
Operations
Release Engineering has historically been involved with the Build and Release pipeline of a product. As part of that, we have some responsibility to help ensure that the build (continuous integration) and release (deployment) pipeline infrastructure is in good running health. We have two functional groups that monitor and respond to any operational issues that come up.
Buildduty
Buildduty provides support for Firefox's build and test infrastructure
The role of Buildduty has grown beyond Release Engineering's scope. There is a dedicated team of 6 people, providing 24/7 support of Mozilla's build and test infrastructure.
See the Buildduty home page and manifesto for more information.
CI Duty
CI Duty provides support for Firefox's release infrastructure and is currently handled by Releng
See the CIDuty for more information.
Releng Applications
- Applications - Various applications and services that RelEng provides
- TODO update application list with: Services, Scriptworker, ChainOfTrust
Mozilla Applications
- Taskcluster - our task execution tool for providing continuous integration and release pipelines
- Treeherder - aggregates all the build, test, and release tasks as a reporting dashboard
- PuppetAgain - Release Operations tool and puppet configs for configurating build and release machines. Both for workers and hosting applications
- TODO - add Release Operations replacement for Slaveapi: Controller
External Third Party Dependencies
- AWS
- Heroku
- Papertrail
- Nagios
- Sentry
Deprecated
below pages should be deleted or be given a "warning" outdated box where appropriate.
VCS Sync
Releases
- Release Checklist and Build Notes
- Releng Post Mortem Meetings
- What is releaseduty?
- Release Promotion
- Update Generation
Platform Support
Tryserver
Testing
Subpages
- Presentations
- How Tos
- Releng Docs
- Special:PrefixIndex/ReleaseEngineering/ - all subpages of ReleaseEngineering in the wiki
- ReleaseEngineering/Obsolete Pages
- day 1 checklist
- day -1 checklist
Run the tasks to staging pools (OSX/Linux)
- Clone mozilla-central to local machine: hg clone ssh://hg.mozilla.org/mozilla-central
- Add the following changes to taskcluster/taskgraph/transforms/task.py file:
diff --git a/taskcluster/taskgraph/transforms/task.py b/taskcluster/taskgraph/transforms/task.py --- a/taskcluster/taskgraph/transforms/task.py +++ b/taskcluster/taskgraph/transforms/task.py @@ -1717,16 +1717,48 @@ def add_index_routes(config, tasks): index_type = index.get('type', 'generic') task = index_builders[index_type](config, task) del task['index'] yield task @transforms.add +def set_worker_type(config, tasks): + for task in tasks: + level = str(config.params['level']) + + provisioner_id, worker_type = get_worker_type( + config.graph_config, + task['worker-type'], + level, + ) + task['worker-type'] = '/'.join([provisioner_id, worker_type]) + yield task + + +@transforms.add +def use_beta_workertypes(config, tasks): + """ + This transform uses a dictionary of worker type names to worker type names + in order to map one set of worker types to another set. This is typically + useful when you want to run a try push against a staging set of worker + types, that you want to test before rolling them out to production. + """ + find_replace_dict = { + "releng-hardware/gecko-t-osx-1014": "releng-hardware/gecko-t-osx-1014-staging", + "releng-hardware/gecko-t-linux-talos": "releng-hardware/gecko-t-linux-talos-b" + } + for task in tasks: + if task['worker-type'] in find_replace_dict: + task['worker-type'] = find_replace_dict[task['worker-type']] + yield task + + +@transforms.add def build_task(config, tasks): for task in tasks: level = str(config.params['level']) provisioner_id, worker_type = get_worker_type( config.graph_config, task['worker-type'], level,
- Commit the changes using the following commit message: hg commit -m "<commit_message>; try: -b do -p macosx64 -u all -t none". The <commit_message> will be a message that will describe your action, like: Run all tests on OSX staging workers.
- To run different tests or on a different platform, use https://mozilla-releng.net/trychooser/
- Do _not_ run unnecessay tests or you may end up on the hiscore board.
- Push the changes into try repository: hg push -f -r . ssh://hg.mozilla.org/try
- You will receive an email like this:
Thank you for your try submission. It's the best! Results will be displayed on Treeherder as they come in: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6657878a04024803ce0aae0601627810b2d8d95e It looks like this submission has talos jobs. You can compare the performance of your push against a baseline revision here: https://treeherder.mozilla.org/perf.html#/comparechooser?newProject=try&newRevision=6657878a04024803ce0aae0601627810b2d8d95e Once completed, all build and log artifacts will be available through the Treeherder URL above. All Taskcluster based artifacts can also be found in the Taskcluster Artifact Index: e.g. https://tools.taskcluster.net/index/artifacts/gecko.v2.try.revision.6657878a04024803ce0aae0601627810b2d8d95e Summary: Test linux generic-worker; try: -b do -p linux64 -u none -t chromez-e10s
Notes
- A useful link for try syntax: https://mozilla-releng.net/trychooser/
- To see the hosts from the staging pool:
* OSX hosts: https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010-beta * Linux hosts: https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-linux-talos-b