ReleaseEngineering/Maintenance
This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.
ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our master buildbots.
Contents
Relevant repositories
Non-releng repositories involved in the CI
Treeherder
- Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
- [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)
Mozharness deployments
Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.
Puppet deployments
Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.
Please note if you are applying a puppet change, unless you have good reason not to do so please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land en masse in production. It can be very difficult to untangle multiple puppet changes that arrive together.
The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.
Reconfigs / Deployments
This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)
Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.
Older changes are archived in: 2016 | 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009
| Outcome | When | Bug #(s) - Description(s) |
| in production | 2017-09-21 07:51 PT |
|
| in production | 2017-09-14 03:55 PT |
|
| in production | 2017-09-06 10:14 PT |
|
| in production | 2017-08-31 06:50 PT |
|
| in production | 2017-08-06 22:46 PT |
|
| in production | 2017-08-04 03:42 PT |
|
| in production | 2017-08-02 00:27 PT |
|
| in production | 2017-07-31 04:12 PT |
|
| in production | 2017-06-12 03:31 PT |
|
| in production | 2017-06-05 06:32 PT |
|
| in production | 2017-06-05 02:12 PT |
|
| in production | 2017-05-11 06:12 PT |
|
| in production | 2017-03-16 07:25 PT |
|
| in production | 2017-03-10 15:02 PT |
|
| in production | 2017-03-10 10:11 PT |
|
| in production | 2017-03-09 13:18 PT |
|
| in production | 2017-03-06 11:57 PT |
|
| in production | 2017-03-06 11:49 PT |
|
| in production | 2017-03-03 13:50 PT |
|
| in production | 2017-03-03 12:53 PT |
|
| in production | 2017-03-01 11:50 PT |
|
| in production | 2017-02-27 19:34 PT |
|
| in production | 2017-02-27 18:50 PT |
|
| in production | 2017-02-27 13:40 PT |
|
| in production | 2017-02-27 11:56 PT |
|
| in production | 2017-02-24 13:02 PT |
|
| in production | 2017-02-23 06:27 PT |
|
| in production | 2017-02-17 09:50 PT |
|
| in production | 2017-02-16 13:27 PT |
|
| in production | 2017-02-15 12:50 PT |
|
| in production | 2017-02-08 13:10 PT |
|
| in production | 2017-02-08 12:19 PT |
|
| in production | 2017-01-23 03:51 PT |
|
| in production | 2017-01-05 04:33 PT |
|
