Release Management/Release Process Checklist Documentation

From MozillaWiki
Jump to: navigation, search

The goal of this page is to document the Release Process Checklist being used by Firefox Release Managers to track each release throughout the cycle. Any changes to this documentation or the checklist should be reflected in both documents.

firefox-ios beta merge day steps documentation snapshot: https://github.com/mozilla-mobile/firefox-ios/wiki/Release-Checklist/2a0be3dc61a5794ce2f8c69d475cc4ad6efd256c application services beta merge day steps documentation snapshot: https://github.com/mozilla/application-services/commit/33ea53616a4d35989059bde593a68dc424a9b95e

Contents

Nightly Checklist

Given the nature of how nightly builds are created and shipped, the role of the release manager during this phase of the cycle skews much more heavily to the monitoring aspect rather than release mechanics.

Prior to the start of the Nightly cycle, the following tasks need to be performed before Merge Day

  1. Create a Release Checklist before the cycle starts.
  2. Create the Milestones Document for the Release Checklist.
    • Use the sheet linked in the Release Checklist.
  3. Create a Firefox Retrospective Notes document.
    • Use the template from Google drive.
    • It is useful to document during the release cycle.
  4. Create a folder in the Release Tracking Google Drive for the release.
    • Move the Release Checklist and Firefox Retrospective Notes to this folder.
  5. Add a Release Tracking Sheet link to Release Owners wiki
    • Add the link to the date in the Release Date column.
  6. Ensure that Bugzilla is bumped to the Nightly version, see BMO/new-version
    • Note: This is performed on the Friday before Merge Day.
    • Find the bug that tracks the bump.
    1. Update the Release Milestones in Bugzilla
    2. Got to https://bugzilla.mozilla.org/admin/new_release
      • The products and milesontes are automatically completed
    3. Click [ Submit ]
    4. Update the Release Tracking flags in Bugzilla
    5. Go to Bugzilla > Administration > Release Tracking Flags
    6. Add the cf_tracking_firefox and cf_status_firefox for the Nightly release
      • Example: If 97 is becoming the Nightly, then add cf_tracking_firefox97 and cf_status_firefox97.
    7. Deactivate the oldest cf_tracking_firefox flag.
      • Example: If 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then deactivate cf_status_firefox94.
    8. Edit the current cf_tracking_firefox_esr tracking flag to add the new release as a Value and deactivate the oldest release.
      • Example: If esr is based on 91, 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then edit cf_tracking_firefox_esr91 to add a Value 97+ set to mozilla-next-drivers and deactivate the Value 94+
    9. Add the cf_tracking_thunderbird and cf_tracking_thunderbird for the Nightly release.
      • Example: If 97 is becoming the Nightly, then add cf_tracking_thunderbird97 and cf_tracking_thunderbird97
    10. Deactivate the oldest cf_tracking_thunderbird flag.
      • Example: If 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then deactivate cf_tracking_thunderbird94.
    11. Edit the current cf_tracking_thunderbird_esr tracking flag to add the new release as a Value and deactivate the oldest release.
      • Example: If esr is based on 91, 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then edit cf_tracking_firefox_esr91 to add a Value 97+ set to mozilla-next-drivers and deactivate the Value 94+.
    12. Add a comment on the bug that the tracking flags were updated.
  7. Create a release notes skeleton in Nucleus.
    1. Click Admin Interface
    2. Click the Releases.
    3. Select the previous Nightly release notes (enable the checkbox beside the release).
    4. At the top of the page, expand the Action dropdown
    5. Select Copy releases.
    6. Click Go.
    7. Select the copy.
    8. In the Version text box edit the version to the current Nightly version.
      • Example: 99.0a1
    9. Set the Release Date to match when Central was bumped to Nightly
    10. Keep the nightly-only notes and remove the release notes from the previous release.
    11. For additional information on the Release Notes process see The Firefox release notes process wiki page.
  8. Update the bugs with nightly+ release note tracking.
    1. Remove nightly+ from any bugs older than 3 releases.
      • Add a comment as a reminder to add a release note request when the feature rides the train.
    2. Remove nightly+ from any bugs that are riding the train.
      • If the same bug is tracking the final release note then set the appropriate release in the relnote track
      • If a different bug is tracking the final release note then add a comment mentioning which bug is tracking the release note.
    3. See The Firefox release notes process wiki page for additional information.
  9. Update the Google calendar Releases Scheduling.
    1. Open the following URL and change the version to the Nightly release.
    2. Download the generated ical file.
    3. Review locally on a calendar app of your choice
    4. Open Google Calendar.
    5. Click + beside Other Calendars.
    6. Select Import.
    7. Select the ical file previously downloaded.
    8. Select the Releases Scheduling calendar.
    9. Click Import.
    10. Please Note: Importing the ical to Google Calendar is a one-shot operation. If you notice incorrect info or dates after importing, correct them directly in Google Calendar. Importing the ical file again will cause duplication.
  10. Ensure the REO (Regression Engineering Owner) is listed on Release Owners.
    • If required check the Google Sheet REO Rotation by Director.
    • Update the wiki with information from the spreadsheet.
      • Ensure that the REO and RelEng owners are listed.
      • Ensure the dates are accurate.
    • If required reach out to the Director to find out who the REO is.
    • Ensure they are invited to the Weekly Regression Triage.
  11. Ensure the Release Engineer on Release Duty is listed on Release Owners.
  12. Ensure the Release Owners wiki is up to date
    • The REO and Release Engineer on Release Duty should be listed

At the start of the Nightly cycle, the following tasks need to be performed on Merge Day

  1. Verify that Central was bumped to the Nightly version.
    • Check version_display.txt
    • Please Note: This is performed on Merge Day of the release going to beta, if it is incorrect talk to the RelMan on duty for the Beta cycle.
  2. Verify that the firefox-android version was updated.
    • Chheck version.txt should be xxx.0a1, where xxx matches the nightly version of desktop.
    • If the version was not updated, then sync with release engineering.
  3. Make the release notes public in Nucleus
  4. Verify the application services version was updated
    • The version in version.txt should be xxx.0a1, where xxx matches the nightly version of desktop.
    • If the version was not updated then talk to the RelMan on duty for the Beta cycle.
  5. Verify the firefox_versions.json is correct.
    • Please Note: Usually this json update is performed the day after merge day
    • Please Note: fx trains and various scripts in other tools use this data.
    • Verify the following are correct:
      • FIREFOX_NIGHTLY, NEXT_MERGE_DATE, NEXT_SOFTFREEZE_DATE
    • If these are incorrect, contact the RelEng on duty via the #release-duty channel in Matrix. It is possible a PR was missed during the RelEng merge day steps.
    • Please Note: There are also date changes that are monitored by autonag. For example, if Template:NextReleaseDate was not updated an autonag email will be sent.
  6. Verify the mobile_versions.json is correct
    • Verify the following are correct:
      • alpha_version, nightly_version
    • If these are incorrect, contact the RelEng on duty via the #release-duty channel in Matrix.

On a daily (or thereabouts) basis, the following items should be monitored during the Nightly cycle

  1. Review Pending tracking-firefoxXX requests
    • Example Filter
    • Ensure that the tickets are progressing, follow up as required.
  2. Review Open tracking-firefoxXX+ and blocking bugs
    • Example Filter
    • Ensure that the tickets are progressing, follow up as required.
  3. Review incoming regression bugs
    • See the REO section in the BugDash
    • Ensure that new tickets are prioritized.
    • Ensure that carry-over tickets are still on the team’s radar.
    • Review with the REO in the weekly regression meeting.
  4. Review stability rates and reported crash spikes
    • Leverage information from the automated emails, monitor top crashes in Crash Stats, and monitor Stability & Release Monitoring Looker Dashboard
    • Ensure that there are bugs tracking stability rates and crash spikes.
    • Ensure these tickets are tracked and prioritized.
    • Please Note: If ever there are stats/bugs that cannot be explained/identified, then reach out to the team for assistance on the #stability Matrix channel, it may be an issue with the tooling issue.
  5. Review the Release Notes requests for Nightly Tickets.

On a weekly (or thereabouts) basis, the following items should be monitored during the Nightly cycle

  1. Review the tracking Mana page for the release.
    • The tracking page can be found as a child of Firefox Release Tracking
    • Ensure that features targeting the release are on track, reach out as needed if any are slipping.
  2. Update the Weekly Release Tracking Digest document, as requested on #fx-weekly-digest.
    • Update the Release Milestones (Details) section to add Status (On Track, Delayed), Ship Date, and additional details depending on where the nightly is in the cycle for example coming up to the Soft Freeze.
    • Review the dates in the Release Milestones section and update the status.

As available, the following items should be monitored during the Nightly cycle

  1. Review and land mozilla-central repo-update HSTS HPKP remote-settings tld-suffixes patches.
    • Every Monday and Thursday revisions are automatically created for review in Phabricator.
    • Monitor email notifications sent by Phabricator: “No Bug, mozilla-central repo-update HSTS HPKP remote-settings tld-suffixes”
    • Review and land the patch to autoland.
  2. Review and land android nightly application-services version update bumps to mozilla-central
    • An automated nightly application-services build is triggered daily. The build will only run if commits merged to main since the previous nightly build.
    • update-bot creates a patch to bump the version for android in mozilla-central, runs a Try build, and sets the release-mangers group as reviewer.
    • Monitor email notifications sent by Phabricator: Update android nightly application-services version bump
    • Review and land the patch to autoland.
    • If the Try build failed:
  3. Review Test Plans for features targeting the release. QA shares test plans via email. For example:
    • Become familiar with how the feature works.
    • Understand how the team ensures it is of sufficient quality to ship in that release.
    • Provide feedback on the risk analysis and mitigations put in place.
  4. Review mid-Nightly and pre-Beta testing reports QA shares mid-Nightly and pre-Beta testing reports via email. For example:
    • If there are any red or yellow reports review the recommendations/plan and the bugs mentioned in the report.
    • If QA flagged that the feature is behind a pref verify that this is expected.

At the end of the Nightly cycle, the following tasks need to be performed prior to and during Merge Day

  1. Send Nightly Soft Code Freeze reminder to dev-platform and firefox-dev.
    • Send this at the start of the week before the Soft Code Freeze.
    • Remind developers that the window for landing riskier fixes is coming to a close until after the version bump.
    • Example email
  2. Prepare the Beta release notes in Nucleus
    1. Click Admin Interface
    2. Click the Releases
    3. Select the Nightly release notes (enable the checkbox beside the release)
    4. At the top of the page, expand the Action dropdown
    5. Select Copy release
    6. Click Go
    7. In the Channel dropdown select Beta
    8. In the Version text box edit the version to the Beta version
      • Example: 96.0beta
    9. Replace the content in the Text box with content from a previous beta release
    10. Set the Release Date to the date when the first Beta is released
    11. Ensure to remove any notes that are only applicable to nightly
  3. Sync with RelEng to confirm the RelEng on duty for the Central to Beta merge.
    • Sync on Friday before Merge Day
    • Use the #releaseduty channel on Matrix.
    • Align they are available for support if needed.
  4. Sync with AppServices team to confirm who will create the new Swift release on Monday
  5. Create checklist(s) for the new ESR cycle
    • Create a checklist for the relevant ESR that corresponds to the Firefox mainline release
    • See ESR calendar
  6. Perform the mozilla-central->mozilla-beta no-op trial run
  7. Create a Desktop Beta Checklist and Mobile Beta Checklist for beta 1
    • Use the b1 macro


Beta Checklist

Once a release moves to the Beta cycle, the daily tasks performed during the Nightly cycle will continue to be carried out as new bug reports come in from a wider audience and new features move through the QA cycle towards shipping. There is also an added triage step during Beta - monitoring the “missed uplifts” email or queries to find issues fixed in Nightly that still affect Beta.

The following tasks need to be performed on Merge Day at the start of the Beta cycle

  1. Sync with the Sheriff team to confirm Central is in a good state so that the Merge Day email can be sent.
    • Sync on Merge Day morning
    • Use the #sheriffs channel on Matrix.
    • Check the Beta simulations document for the release, this document is emailed to Release Management at the start of the night cycle.
    • Ensure that central does not currently have any known severe regressions or performance issues that would be carried into beta.
  2. Notify the firefox-ios dev team on #firefox-ios-releases that today is Merge Day
    • Let the team know roughly when you plan on branching
    • Ask them to let you know if there are any urgent pending PRs
  3. Perform the Release Merge Day - part II steps
  4. application-services: Create an application-services release branch with the format release-vXXX off of the main branch
  5. application-services: Update version.txt and the CHANGELOG.md in the release-vXXX branch
    1. In version.txt, update the version from XXX.0a1 to XXX.0.
    2. In CHANGELOG.md, change In progress to _YYYY-MM-DD_ to match the Merge Day date and add a URL to the release version change log
    3. Create a commit named 'Cut release vXXX.0`
    4. Create a pull request and merge to the release branch, see example PR
  6. application-services: Update the application-services version from XXX to YYY and CHANGELOG.md in the main branch
    1. In version.txt, bump the version from XXX.0a1 to YYY.0a1
    2. In CHANGELOG.md, bump the in progress version from XXX.0 to YYY.0, add a header for the previous release version, and add a URL to the previous release version change log
    3. Create a commit named 'Start release vYYY.0`
    4. Create a pull request and merge to the main branch, see example PR
  7. application-services: Create a new Application Services release in Ship-It
  8. application-services: Start the Application Services build in Ship-It via Promote
  9. application-services: Ship the Application Services build in Ship-It
  10. application-services: Tag the release in the Application Services repo
    1. See example tag
  11. application-services: Notify the application-services dev team on #application-services-eng that the release branch is ready, the version bumped was merged, and the release is ready
    • They will perform the rust-components-swift release steps
  12. firefox-ios: Create a firefox-ios release branch with the format release/vxxx off of the main branch
    • Create a branch name with the format release/vxxx off of the main branch through the GitHub UI
  13. firefox-ios: Update the firefox-ios version from the previous nightly version to the next nightly version in the main branch
    • Use the ios-version-bump.py script to update the Firefox iOS and Focus iOS version in main branch.
  14. firefox-ios: Notify the firefox-ios dev team on #firefox-ios-releases that the release branch is ready and the version bumped was merged
  15. firefox-ios: Trigger the Firefox: import translations GitHub action
    • Use the release/v[beta_version] as the target branch
    • See documentation here
  16. firefox-ios: Pin the Application Services version in the firefox-ios release branch
  17. firefox-ios: Change the firefox-ios branch in the scheduled Bitrise beta build to release/vxxx
    • View the firefox-ios app in BitRise
    • Expand the Scheduled list
    • Click the three dotted line button on the right to update the configuration for SPM_Deploy_Prod_Beta
    • Select Edit configuration
    • Change the Branch to release/vxxx
    • The Days should already be selected (Tues, Thu, Sun)
    • Save by clicking Schedule Build button at the button
    • Set the time for 9:00 PM UTC
    • Click the three dotted line button again and click Start schedule to enable the schedule
      • The workflow will then trigger based on the schedule
      • Ensure it is enabled by confirming that the branch name turned blue and a calendar icon appeared on the schedule.
  18. firefox-ios: Trigger the firefox-ios SPM_Deploy_Prod_Beta Bitrise build workflow for Firefox iOS
  19. firefox-ios: Ensure the firefox-ios build is available in Testflight
  20. firefox-ios: Add the Internal Group to the firefox-ios TestFlight build to start the review
  21. firefox-ios: Trigger the firefox-ios focus_SPM_Beta Bitrise build workflow for Focus iOS
  22. firefox-ios: Ensure the focus-ios build is available in Testflight
  23. Switch to Application Services in Firefox Android to Release
    • Change the Version and Channel in mobile/android/android-components/plugins/dependencies/src/main/java/ApplicationServices.kt
    • Push directly mozilla-beta
  24. Send the merge complete email to release-signoff and release-drivers
  25. Announce the soft freeze is over
    • Reply back to the soft code freeze announcement email
    • "The version bump to xxx has now landed on mozilla-central and the soft freeze period is over."
  26. Review any pending uplifts that may be required before proceeding with a beta 1 build
    • This should not be a common occurrence, however, an example would be a crash fix that may have landed in central post-merge.
  27. Set up Desktop and DevEdition builds in Ship-It
    1. Connect to the VPN
    2. Log into Ship-It
    3. Click Releases dropdown
    4. Click Firefox dropdown
    5. Click New
      • New shipit.png
    6. For Desktop complete as follows and click Create Release:
      • Please note: Revision should be the tip. The Partial updates field is automatic.
      • Screen Shot 2022-05-31 at 3.38.42 PM.png
    7. For DevEdition complete as follows and click Create Release
      • Please note: Revision should be the tip. The Partial updates field is automatic.
      • Screen Shot 2022-05-31 at 3.40.59 PM.png
  28. Start the Desktop and DevEdition in Ship-it via the Promote Task
    1. View the Pending Releases in Ship-It
    2. Click the Promote Task for Desktop and continue through the pop-ups
    3. Click the Promote Task for DevEditon and continue through the pop-ups
  29. Confirm the Desktop and DevEdition builds have started
    • Taskcluster email: “[desktop] Build of firefox xxx.0b1 build 1”
    • Taskcluster email: “[desktop] Build of devedition xxx.0b1 build 1”
  30. Confirm notification sent when the Desktop and DevEdition builds finish
    • Taskcluser email: “firefox xxx.0b1 build1/mozilla-beta is in the candidates directory”
    • Taskcluser email: “devedition xxx.0b1 build1/mozilla-beta is in the candidates directory”
  31. Schedule push to Desktop and DevEditon CDN from Ship-It via Push
    1. Click Push
      • Screen Shot 2022-05-31 at 3.42.21 PM.png
    2. Click Schedule
      • Screen Shot 2022-05-31 at 3.44.18 PM.png
  32. Confirm notification sent when the Desktop and DevEditon CDN push finishes
    • Taskcluser email: “firefox xxx.0b1 build1/mozilla-beta has been pushed to cdntest”
    • Taskcluser email: “devedition xxx.0b1 build1/mozilla-beta has been pushed to cdntest”
  33. Create Firefox Android (AC, Fenix, Focus) release in Ship-It
  34. Start Firefox Android (AC, Fenix, Focus) release in Ship-It via Promote
    • Please note: you can only start a ship-it build on a revision that has a completed decision task
  35. Confirm notification sent when the Firefox Android (AC, Fenix, Focus) builds finish
    • Taskcluster email: “firefox-android XXX.0b# build1/mozilla-beta is in the candidates directory"
  36. Push Firefox Android (AC, Fenix, Focus) release in Ship-It via Push
  37. Confirm notification sent when the Firefox Android (AC, Fenix, Focus) push finishes
    • Taskcluster email: “Focus/Fenix XXXb#-build1 has been pushed to the closed testing track on Google Play"
  38. Ship Firefox Android (AC, Fenix, Focus) from Ship-It via Ship
  39. Turn on managed publishing in the Play Console for Fenix.
    • Fenix beta 1 only rolls out once QA sign-off.
    • This allows you to submit an app for review but control when it is published.
  40. Manually create a Fenix release on the Production track @ 25% rollout and submit for review
    • If an app is still pending review on a closed track then you cannot "promote" it to a production track.
    • You can manually create the release by selecting the bundle that was pushed to the Play Console.
    • Don't forget to edit the release name, for example change it to 101.0b1.
    • See Prepare and roll out a release for more information.
  41. Turn on managed publishing in the Play Console for Focus.
  42. Manually create a Focus release on the closed testing track (Foxfooding) @ 25% rollout and submit for review
  43. Monitor for QA sign-off on desktop functional testing before proceeding with Step 23.
    • Please Note: Desktop Build Validation sign-off is usually provided the day after Merge Day.
    • QA will post a message to the #qa-coordination channel in Slack when they complete functional testing.
    • Build validation testing is tracked here
  44. Once QA has signed off, push Desktop and DevEdition to Beta in Ship-It via Ship.
  45. Verify that the Balrog rule changes are live and correctly set.
  46. Activate automated betas in Ship-It
    1. At the bottom of the page, click the red x beside Firefox Desktop: Beta
      • Screen Shot 2022-05-31 at 3.51.31 PM.png
    2. Click Enable
      • Screen Shot 2022-06-01 at 9.47.26 AM.png
    3. Perform the same actions for Firefox Developer Edition: Beta and Firefox Android (Android-Components, Fenix, Focus): Beta
    • Please Note: Automated Betas are built at 13:00 UTC every Monday, Wednesday, and Friday as determined here.
  47. Make the Beta release notes public in Nucleus
    • Enable the Is Public checkbox and save
      • Screen Shot 2022-06-01 at 9.49.18 AM.png
  48. Share a link to the Beta release notes and the release notes draft document in #release-notes Slack channel
  49. Email release-notes with a link to the draft document and the submission deadline.
  50. Monitor for QA sign-off on desktop update testing
    • QA will post a message to the #qa-coordination channel in Slack when they complete update testing.
  51. Monitor for QA sign-off on Fenix/Focus build validation before proceeding with Step 31.
    • Fenix build validation testing is tracked here
    • Focus build validation testing is tracked here
    • Please Note: Build Validation sign-off is usually provided the day after Fenix/Focus builds are produced.
    • Focus and Fenix QA sign-off are sent via #qa-coordination Slack Channel
  52. Turn off managed publishing in the Play Console for Fenix
    • This allows the app to publish automatically once it passes review.
  53. Turn off managed publishing in the Play Console for Focus
  54. Monitor for QA sign-off on Firefox iOS build validation
    • Build validation testing is tracked here
    • Once QA has signed off, add the External Beta Testers group to the Firefox iOS TestFlight Build
  55. Monitor for QA sign-off on Focus iOS build validation
    • Build validation testing is tracked here
  56. Verify the mobile_versions.json is correct
    • Verify the beta_version is correct
    • If incorrect, contact the RelEng on duty via the #release-duty channel in Matrix.

The following tasks need to be performed daily during the Beta cycle

  1. Monitor Uplift Requests
    • Example Filter and Phabricator Dashboard
    • Accept or reject the beta uplift requests.
    • See Uplift Rules for guidelines
    • Please Note: During the RC week, also monitor release uplift requests.
    • Please Note: When pushing uplifts be conscious of timing around the automated beta builds. Pushing an uplift will trigger CI, the automated beta build will not trigger until the CI completes.
    • Please Note: If an uplift contains string changes, then l10n approval is required and the commit message must end with l10n=approver_name. Add a ni on the bug for approval before taking the uplift.
  2. Monitor Pending tracking-firefoxXX requests
    • Example Filter
    • Ensure that the tickets are progressing, and follow up as required.
  3. Monitor Open tracking-firefoxXX+ and blocking bugs
    • Example Filter
    • Ensure that the tickets are progressing, and follow up as required.
  4. Monitor Newly-filed regression bugs
    • New regressions bugs can be found here
    • Review with the REO - ensure that new tickets are prioritized and ensure that carry-over tickets are still on the team’s radar
  5. Monitor stability rates and reported crash spikes
  6. Monitor the Release Notes requests for Beta Tickets.
    • Example Filter
    • Check through tickets in the release that are tagged for release notes. Review and add these notes to the Beta Release Notes in Nucleus.

The following tasks need to be performed for each beta build (Monday, Wednesday, Friday) during the Beta cycle

Please Note: The Beta build pipeline is also monitored by the Sheriffs, if there is an intermittent test failure they will rerun the failed job. If there is a permanent failure or an issue in the build infrastructure, then it may require canceling the build. Discuss in the #releaseduty Matrix channel. If the decision is made to cancel, then view the pending release in ship-it and click cancel. To create a new build, follow the same steps as outlined in One Time - Start of Beta Cycle to produce build 2 of the current beta release.

  1. Create a tab in the release tracking sheet “xxx.0bx”
    • Use the relevant beta macro
    • Please note: After all the beta steps are complete, the tab can be hidden in order to minimize clutter.
  2. Review tracking-firefoxXX+ bugs and approval requests.
    • This step is performed as part of the daily activities, tracked in the checklist for confirmation.
  3. Verify all approved bugs landed on mozilla-beta.
    • This step is performed as part of the daily activities, tracked in the checklist for confirmation.
  4. Ensure the Firefox iOS build is available in Testflight
  5. Confirm the Desktop and DevEdition builds have started.
    • Taskcluster email: “[desktop] Build of firefox xx.0bx build 1”
    • Taskcluster email: “[desktop] Build of devedition xx.0bx build 1”
    • Please Note: The automated beta builds schedule is determined here.
  6. Confirm notification sent when the Desktop and DevEdition builds finish.
    • Taskcluster email: “firefox xx.0bx build1/mozilla-beta is in the candidates directory”
    • Taskcluster email: “devedition xx.0bx build1/mozilla-beta is in the candidates directory”
  7. Confirm the Firefox Android (AC, Fenix, Focus) builds have started
    • Taskcluster email: “Build of firefox-android xx.0bx build 1"
  8. Confirm notification sent when the Firefox Android (AC, Fenix, Focus) builds finish
    • Taskcluster email: “firefox-android xx.0bx build1/mozilla-beta is in the candidates directory"
  9. Push Firefox Android (AC, Fenix, Focus) release in Ship-It via Push
    • Wait until the previous beta has passed review before pushing.
    • Pushing a new release to the Alpha track will reset the clock on the previous beta version review.
  10. Confirm notification sent when the Firefox Android (AC, Fenix, Focus) push finishes
    • Focus/Fenix XXXb#-build1 has been pushed to the closed testing track on Google Play
  11. Ship Firefox Android (AC, Fenix, Focus) from Ship-It via Ship
  12. Beta 2 and Beta 3 only - Bump the previous beta version to 100%
    • To create a new release any staged releases must have rolled out to 100%
    • For Beta 2, check if Fenix/Focus nightly has any stability concerns that could also affect beta and then bump Fenix beta 1 to 100%.
  13. Manually create a Fenix release on the Production track @ 50% rollout for b2 and @ 99% rollout for b3+
  14. Manually create a Focus release on the closed testing track (Foxfooding) @ 50% rollout for b2 and @ 99% rollout for b3+
  15. Confirm notification sent when the Desktop and DevEditon CDN push finishes.
    • Taskcluster email: “firefox xx.0bx build1/mozilla-beta has been pushed to cdntest”
    • Taskcluster email: “devedition xx.0bx build1/mozilla-beta has been pushed to cdntest”
  16. Verify that the Desktop and DevEdition Balrog rule changes are live and correctly set.
  17. Monitor for QA sign-off on Desktop and DevEdition build validation for both update/functional testing.
    • QA will post a message to the #qa-coordination channel in Slack when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
  18. Monitor for QA sign-off on Fenix/Focus build validation before proceeding with bumping the Fenix/Focus rollout.
    • Focus and Fenix QA sign-off are sent via #qa-coordination Slack Channel.
    • Please Note: Build Validation sign-off is usually provided the day after Fenix/Focus builds are produced. If the sign-off is yellow/red, then follow-up on the #mobile-android-team Slack channel. Ensure a plan is in place or a decision is made to block/proceed with the release.
    • For b3+ once QA has signed off, roll out Fenix to the production track in Google Play to 100%.
      • Reply to the QA sign-off Slack message and include the rollout percentage.
    • For b3+, once QA has signed off, roll out Focus to the Closed testing - Foxfooding track in Google Play to 100%.
      • Reply to the QA sign-off Slack message and include the rollout percentage.
  19. Monitor for QA sign-off on Firefox iOS build validation before proceeding with adding the External Testers group in TestFlight.
    • Firefox iOS QA sign-off are sent via #qa-coordination Slack Channel.
    • Please Note: Build Validation sign-off is usually provided the day after Firefox iOS builds are produced. If the sign-off is yellow/red, then follow-up on the #firefox-ios-releases Slack channel. Ensure a plan is in place or a decision is made to block/proceed with the release.
    • Once QA has signed off add the External Testers group to Firefox iOS.
      • Reply to the QA sign-off Slack message.

As available, the following items should be monitored during the Beta cycle

  1. QA shares beta testing and pre-release testing reports via email. Review the test reports for the release. For example:
    • If there are any red or yellow reports, review the recommendations/plan and the bugs mentioned in the report. For example:
      • Track the bug reported as blockers.
      • If there is no clear plan to address the report, follow up to ensure that engineering is reviewing.
    • If QA mentions the feature is behind a pref verify that this is expected.
  2. Phabricator patches for remote-settings updates are generated automatically. Typically this is every Monday and Thursday. These patches must be included in the beta uplifts.
    • Monitor email notifications sent by Phabricator: “No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes”
    • Review the and land patch.
  3. Use the Mana Beta Quality Metrics Page to review the Beta Quality Metrics.
    • The data is embedded from this wiki page.
    • The data is currently added manually.
      • The data is stored automatically every day in bugzilla using New Charts
      • To extract the data, run the report by adding the next needed dates and then exporting to csv. (The queries are numbered to make this step easier)
      • This is then manually pasted into the 'Data from Queries' tab.
      • Please Note: the goal is to automate this.
    • Take a look at the metric/charts over time to ensure the value is stable.
      • Please Note: Ensure the data is updated prior to the Tuesday channel meeting.
  4. Update the Weekly Digest document, as requested on the #fx-weekly-digest Slack channel.
    • Update the Release Milestones (Details) section to add Status (On Track, Delayed), Ship Date, information on the current beta release for both Desktop/DeveEdition and Fenix/Focus, information on where we are in the Beta cycle for example when coming up to end of early beta.
    • Review the Release Milestones section dates and update the status.
    • Update the Quality Metrics section to add information on the number of outstanding regressions, if there are any release blockers, and an overview on the trend of the Beta Quality Metrics metrics.

The following needs to be performed at the End of Early Beta

Please Note: This is performed on the Friday at the midday point of the beta cycle, see the Releases Scheduling google calendar. This is typically after beta 6 has been released. This should be done in before pushing any other uplifts to beta after beta 6.

  1. Go to Treeherder.
  2. Select the beta repo.
  3. On the latest push, click on the down arrow at the top right corner.
  4. Select “Custom push action…”
  5. Choose merge-automation from the dropdown.
  6. Insert the following payload and click trigger.
force-dry-run: true
behavior: early-to-late-beta
push: true

After the job runs, verify that the commit was pushed to the mozilla-beta repo, see example.


The following needs to be performed at the end of the Beta Cycle

  1. Perform the mozilla-beta->mozilla-release migration no-op trial run
  2. Disable automated beta builds in Ship-it.
  3. Disable the firefox-ios firefox-ios SPM_Deploy_Prod_Beta BitRise scheduled workflow.
    • Expand the Scheduled section in the firefox-ios on BitRise
    • Pause the schedule for the SPM_Deploy_Prod_Beta that runs "Every Tue, Thu, Sun @ 20:00"
  4. Please Note: Monitor beta/release uplift requests between the final beta build and the beta to release merge. It is possible that a late uplift request may come in for a regression/stability fix, evaluate if this is a blocker for the RC and if it should be uplifted to beta before the beta to release merge and RC build.

RC Checklist

Once a release moves to the Release cycle, preparations for the release candidates begin. Until go-live, the daily tasks are similar to the beta cycle in terms of monitoring and if needed create a new release candidate.

The following tasks need to be performed at the start of RC week (week before go-live) for Desktop/Android

  1. Review tracking-firefoxXXX+ bugs and approval requests
    • Verify that all approved uplift requests were uplifted to beta
  2. Verify all approved bugs landed on mozilla-release
  3. Perform the Release Merge Day - part I steps
  4. Send the merge complete email to release-drivers
  5. Example email
  6. Once the merge is complete, verify that the Treeherder tests green/starred
  7. Set up Desktop build in Ship-It.
    1. Connect to the VPN.
    2. Log into Ship-It.
    3. Click New Release.
      • Shit-it new release.png
    4. Complete as follows and click Create Release:
      • Screen Shot 2022-06-07 at 9.26.10 AM.png
      • Please note: Revision should be the tip. The Partial updates field is automatic.
    5. Click Submit.
      • Screen Shot 2022-06-07 at 9.27.24 AM.png
  8. Start the Desktop build from Ship-It via Promote RC.
    • Screenshot 2022-06-07 at 09-28-44 RelMan Activities Overview.png
    • Click the Schedule button.
      • Screenshot 2022-06-07 at 09-31-56 RelMan Activities Overview.png
  9. Confirm Desktop build has started.
    • Taskcluster email: “[desktop] Build of firefox xx.0 build 1”
  10. Set up Firefox Android (AC, Fenix, Focus) release in Ship-It.
    • Follow the same steps as creating a beta build.
  11. Start Firefox Android (AC, Fenix, Focus) release in Ship-It via Promote.
  12. Confirm notification sent when Firefox Android builds finish.
  13. Push Firefox Android (AC, Fenix, Focus) release in Ship-It via Push.
  14. Confirm notification sent when the Firefox Android (AC, Fenix, Focus) push finishes.
  15. Turn on managed publishing in the Play Console for Fenix/Focus/Klar.
  16. Manually create a Fenix/Focus/Klar release on the Production track with a 5% rollout.
  17. Submit the Fenix/Focus/Klar release for review.
  18. Push Desktop to Beta at 100% from Ship-It via Ship RC.
    • Screenshot 2022-06-07 at 09-32-46 RelMan Activities Overview.png
    • Click the Schedule button.
      • Screenshot 2022-06-07 at 10-28-30 RelMan Activities Overview.png
  19. Verify that the Balrog rule changes are live.
    1. View the Balrog Rules for Firefox Beta.
    2. Verify that the Background Rate for Firefox-xx.0-build1 is 100.
      • Screenshot 2022-06-07 at 10-39-15 RelMan Activities Overview.png
      • Please Note: The Fallback Mapping is the previous beta release.
  20. Email release-signoff with a confirmation that updates are live.
  21. Verify the package in the Microsoft Store submission is the correct version.
    • Please Note: The submission is automatically created as part of the RC build pipeline.
    • Verify that correct package was uploaded, the version displayed in the submission matches the release version
  22. Submit the Desktop RC build to the Microsoft Store for review
    • Please Note: The Microsoft Store review can take several days, it’s recommended to submit for review at the end of RC week. If there is a blocker found later, you can always resubmit.
    1. Navigate to the Microsoft Partner Center
    2. Select Firefox from Windows & Xbox > Overview
    3. Select the pending submission
    4. In Packages enable gradual rollout and set to 25%
      • Ensure to enable the option to provide the newest package when users manually check for updates.
    5. In Submission Options set to Automatically Publish on the go-live date and time
    6. Submit to the store for review
    • Please see App submissions documentation for additional information on the Microsoft Store submission process.


The following tasks need to be performed at the start of RC week (week before go-live) for iOS

Firefox/Focus/Klar iOS

  1. Check if there are any missing backports from the previous release.
    • Ensure that any backports from the dot releases on the previous version were also backported.
  2. Check if there are any pending PRs for the firefox-ios release branch.
    • firefox-ios
    • Example Filter: is:pr is:open base:release/v115
  3. Sync with the iOS team for Hard Freeze.
  4. Run the Firefox:import translations (strings sync) action, set the target branch to the release branch (ex: release/v118).
    • See documentation here.
  5. Trigger the firefox-ios SPM_Deploy_Prod_Beta bitrise build workflow for the release branch.
  6. Trigger the firefox-ios focus_Release bitrise build workflow for the release branch.
  7. Verify that Firefox iOS release is available in TestFlight.
  8. Verify the Focus/Klar release is available in TestFlight.

As available, the following should be monitored/performed during RC week for Desktop/Android

  1. Phabricator patches for remote-settings updates are generated automatically. Typically this is every Monday and Thursday. These patches must be included in release/esr uplifts.
    • Monitor email notifications sent by Phabricator: “No Bug, mozilla-release repo-update HSTS HPKP remote-settings tld-suffixes”/“No Bug, mozilla-esr91 repo-update HSTS HPKP remote-settings tld-suffixes”
    • Review and land the patch.
  2. Update the Weekly Digest document, as requested on #fx-weekly-digest.
    • Update the Release Milestones (Details) section to add Status (On Track, Delayed), Ship Date.
    • Review the Release Milestones section dates and update the status.
  3. Verify that QA have signed off manual testing for Desktop
    • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
  4. Verify that QA have signed off update tests on Beta.
    • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
  5. Verify that QA have signed off update tests on release-localtest
    • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
  6. Monitor for QA sign-off on Fenix/Focus build validation before proceeding with Steps 7 to 10.
    • Please Note: Build Validation sign-off is usually provided the day after Fenix/Focus builds are produced.
    • Focus and Fenix QA sign-off are sent via #qa-coordination Slack Channel.
  7. Turn off managed publishing in the Play Console for Fenix to allow it to roll out to 5% once the app is reviewed
  8. Turn off managed publishing in the Play Console to allow it to roll out to 5% once the app is reviewed and push it to the Open Testing track at 100%
  9. Upload Fenix to Samsung Store with 5% rollout when approved
    • Upload ARM32 and ARM64 APKs (armeabi-v7a/arm64-v8a)
      • The APKs are available as an artifact in the signing-apk job from the Promote graph.
    • Set the rollout rate to 5%
    • Set to publish automatically
    • Please note: It can take up to 2 days for review.
  10. Upload Focus to Samsung Store with 5% rollout when approved
    • Perform the same steps as Step 9

As available, the following should be monitored/performed during RC week for iOS

Focus iOS

  1. Monitor for QA sign-off on Focus iOS build validation before proceeding with Steps 2 and 3.
    • Please Note: Build Validation sign-off is usually provided the day after the builds are produced.
    • Focus iOS QA sign-off is sent via #qa-coordination Slack Channel.
  2. Submit iOS Focus and iOS Klar in the AppStore for review.
    • Select a phased rollout when submitting.
    • Select to automatically release after App Store Review on Sunday night Eastern prior to go-live (i.e. go-live date -2)
      • Note: If the Monday of go-live week is a Canadian public holiday, then set the automatic release date to Monday night Eastern instead.
  3. Tag the release in GitHub in the firefox-ios repo using the format `focus/klar-vXXX.X`

Firefox iOS

  1. Monitor for QA sign-off on Firefox iOS build validation before proceeding with Steps 2 to 6.
    • Please Note: Build Validation sign-off is usually provided the day after the builds are produced.
    • Firefox iOS QA sign-off is sent via #qa-coordination Slack Channel.
  2. Add the External Beta Testers group to the Firefox Beta TestFlight Build
  3. Gather Firefox iOS release notes.
    • This can be done in parallel to the Desktop/Android release note gathering.
    • The store release notes are needed before submitting.
  4. Submit Firefox iOS in the AppStore for review.
    • Select a phased rollout when submitting.
    • Select to automatically release after App Store Review on Sunday night Eastern 10PM (Monday 2AM GMT) prior to go-live (i.e. go-live date -2)
      • Note: If the Monday of go-live week is a Canadian public holiday, then set the automatic release date to Monday night Eastern instead.
  5. Tag the release in GitHub in the firefox-ios repo using the format `firefox-vXXX.X`

General iOS tasks

  1. Bump the Firefox iOS version in the release branch to XXX.1
    • Wait until the Friday of RC week to avoid bumping the version if another RC is needed

RC Uplifts

This tab is for tracking bugs that are being tracked for possible uplift to the mozilla-release repository for RC builds. The primary objectives are:

  • Track whether there are any drivers for a respin of the RC builds during RC week.
  • Assess whether Desktop, Mobile, or both are affected by the issues noted.
  • Verify that all drivers have had an explicit decision made.

The following tasks need to be performed during RC week (the week before go-live)

  1. Monitor Uplift Requests for Beta/Release
    • For Example
    • Track if there are any drivers for an RC respin during RC week.
    • Assess whether Desktop, Mobile, or both are affected by the issues noted.
    • Verify that all drivers have had an explicit decision made.
  2. If a new RC is required, for example, a release blocker was identified, then create a new RC checklist tab and perform the same steps as for the initial RC but with the following exceptions:
    1. Cancel the previous RC build in ship-it
    2. Delete the Microsoft Store submission before creating a new RC build
      • Please Note: The RC build will fail if trying to create an MSIX submission when there is already a pending submission
      • To cancel and delete the pending submission you may first need to change it from publishing automatically to publishing manually
  3. Determine if a new RC build impacts Android.
    • If a new Android build is not required, then there is no impact on the Fenix/Focus builds and no action is required.
    • If a new Android build is required, then repeat the RC steps with the following exceptions:
      • Cancel the previous Firefox Android (AC, Focus) build in Ship-It.
      • Cancel the previous Fenix build in Ship-It.
      • If Fenix/Focus/Klar already rolled out - Replace the 5% rollout of Fenix/Focus/Klar in Google Play and the Samsung Store.


Go-Live Checklist

Once RC week is over and the release candidate is ready to go-live, preparations for go-live begin. After go-live, monitoring continues and if needed preparing for potential dot releases.

The following tasks need to be performed prior to go-live

  1. Ensure that feedback for release notes was gathered.
    • Monitor the #release-notes Slack channel
    • For more information on the release notes doc process see here
  2. Run the new-contributor.py script and add to the release notes
    • For more information on the release notes doc process see here
  3. Release notes RelMan peer review.
    • For more information on the release notes doc process see here
  4. Incorporate feedback into draft release notes.
    • Add a release in Nucleus for Firefox, Firefox for Android, Firefox for iOS
    • Duplicate the previous release, edit the version, dates, and remove the previous release notes. For example Firefox, Firefox for Android, Firefox for iOS
    • Create release notes entries as outlined in the document shared on the #release-notes Slack channel
    • Please Note: If images are included with the release note then create a PR for Bedrock, example PR
    • Share a link to the staged release notes on #release-notes
  5. Ensure that Legal signoff on release notes was requested by UX/Product
    • Please Note: This step is not always required, if the release notes are not atypical
  6. Review tracking-firefoxXX+ bugs and confirm there are no blocking issues
  7. Check for crash spikes with RC builds
  8. Schedule push to CDN from Ship-It
    • Click Push
    • Screenshot 2022-06-07 at 10-45-18 RelMan Activities Overview.png
    • Click Schedule
    • Screenshot 2022-06-07 at 10-47-14 RelMan Activities Overview.png
  9. Ensure the automated email was sent after the push to release-cdntest
    • Taskcluster email: “firefox xx.0 buildx/mozilla-release has been pushed to cdntest.
  10. Ensure QA have performed the Update test on release-cdntest
    • QA will post a message to the #qa-coordination channel in Slack when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
  11. Ensure Legal have signed off on release notes
    • Please Note: This is only required based on the outcome of Step 3.
  12. Ensure that the security advisories are ready
    • Check with the security team on the #security Slack channel

The following tasks need to be performed on go-live day

  1. Schedule push to release at 25% from Ship-It
  2. Bump Fenix rollout rate in the play store to 25%
  3. Bump Focus/Klar rollout rate in the play store to 25%
  4. Bump Fenix rollout rate in the Samsung Store to 25%
  5. Bump Focus rollout rate in the Samsung Store to 25%
  6. Ship Firefox Android (AC, Fenix, Focus) from Ship-It via Ship
  7. Verify that the Microsoft Store is published at 25% rollout
  8. Make release notes live in Nucleus
    • Please Note: There is a 15-30min lag between making the change in Nucleus and the live website picking up this change, so plan to do this 15-30min prior to go-live.
  9. Upload security advisories to GitHub
  10. Signoff on scheduled rule change in Balrog
    • Set the rollout percentage to 25%
    • Sync with release management for another team member to signoff on the rule change.
  11. Send Slack message to #release-coordination to notify of the release
    • For example “Firefox Desktop 115.0, ESR 115.0.0, Fenix/Focus/Klar Android 115.0, and Firefox/Focus/Klar iOS 115.0 are live”
    • Include links to the release notes.
  12. Verify that the new release is live on mozilla.org
  13. Verify that the release notes are live here
  14. Verify LATEST_FIREFOX_VERSION in firefox_versions.json
  15. Verify that security advisories are live here
  16. Ship new Desktop release in Ubuntu Snap Store
  17. Verify that the release is live on Flathub
    • The `release-flatpak-push-firefox` job runs as part of the ship graph and the publication is handled by Flathub.
    • https://flathub.org/api/v1/apps/org.mozilla.firefox contains the current release available on Flathub.
    • It can take up to 120 minutes for the release to publish on Flathub. If the release is not updated then reach out in the #flatpaks Matrix channel.
  18. Email release-drivers & release-signoff that updates are live at 25%
  19. Ensure QA have performed the Update tests on Release
    • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
  20. Email announce list
  21. Schedule Desktop update rate to 0% in Balrog after 24 hours.
  22. Signoff on scheduled rule change in Balrog.
    • Sync with release management for another team member to signoff on the rule change.

The following tasks need to be performed 24 hours after go-live

  1. Verify Desktop update rate at 0% in Balrog
  2. Email release-signoff & release-drivers to confirm 0% throttling.

The following tasks need to be performed 48 hours after go-live

  1. Review release crash rates and incoming bugs for new blockers.
  2. Bump Desktop update rate to 100% in Balrog.
  3. Signoff on scheduled rule change in Balrog.
    • Sync with release management for another team member to signoff on the rule change.
  4. Bump the Microsoft Store rollout to 100%.
    • Click Finalize package rollout
  5. Review fenix/focus release crash rates, incoming bugs for new blockers, and spike/trend in negative play store reviews.
    • Check to see if there is anything concerning that would prevent completing the rollout.
  6. Bump fenix/focus/klar rollout rate in the play store to 100%.
  7. Bump fenix/focus rollout rate in the Samsung Store to 100%.
  8. Email release-signoff & release-drivers to confirm full rollout.

The following tasks need to be performed daily after go-live

  1. Review release crash rates and incoming bugs for new blockers
  2. Review uplift requests for dot release drivers and ride alongs

The following tasks need to be weekly after go-live

  1. Update the Weekly Digest document, as requested on #fx-weekly-digest.
    • Update the Release Milestones (Details) section to add status on the release, dot release, etc.


Dot Release Uplifts

Similar to the RC Uplifts tab. The primary purpose of this tab is to track any bugs driving a dot release and bugs that are under consideration to ride-along, It is also used to assess which products are affected by any drivers. It is up to the release manager what should go into a dot release, and we try to keep these uplift guidelines in mind. Adding even what looks like a trivial fix can add risk, both to the process (delay, extra testing, work for other teams) and to causing new regressions. For security issues, we may not want to take them in a dot release unless there is particular pressure to do so.

The following information needs to be added for each bug:

  • Bug number
  • Component
  • Is the bug a dot release driver?
  • Does the patch affect Desktop, Mobile, or both?
  • Is the fix ready?
  • Is the patch on mozilla-central?
  • Is there an uplift request on the ticket?
  • Is the uplift approved?
  • Is the patch uplifted to release?
  • Does the bug need a release note?
  • Is the patch for a security bug?
  • Any additional notes
    • For example, outline why the bug is being taken in a dot release.

Please Note: For each dot release, create a section in the Dot Release Uplifts tab to track the bugs in consideration: For example, “Desktop 101.0.1& Fenix/Focus 101.2.0 - GTB 2022-06-08”


Dot Release Checklist

The following tasks need to be performed at the start of preparing a dot release

Desktop

  1. Duplicate the Desktop Dot Release Checklist tab and name it appropriately.
    • Example: Desktop 101.0.1
  2. Email release-drivers group to inform them a dot release is being created.
    • Include information on the driver, if there will be an Android release, and a link to the sheet used to track the uplifts.
    • Example Email
  3. Review any pending release uplift requests
    • Example Filter
    • If they are low risk consider including them in the dot release
  4. Uplift patches to release.
  5. If there are security bugs uplifted then sync with the security team member on rotation
    • The security team member on point is whoever is on rotation for the next release version to ship.
    • The security team member on will assign the CVEs and prepare the advisories. This also covers any Android advisories.
  6. Create a release in Nucleus to add release notes.
    • Add a release note with a link to the major release for reference.
    • For bugs that have been identified as needing a release note:
      • If a release note is required and the release note text is clear from the bug content then create it directly in Nucleus.
      • Otherwise, request wording before adding the release note.
  7. Follow the same process as creating an RC build.
    • If any uplifts impact Android then duplicate the Android Dot Release Checklist tab and name it appropriately.
  8. Follow the same process as monitoring when an RC build has QA sign-off

Android

  1. Duplicate the Android Dot Release Checklist tab and name it appropriately.
    • Example: Android 101.0.1
  2. Follow the same process as creating an RC build with the following exceptions:
    1. Turn on managed publishing in the Play Console.
      • This allows you to submit an app for review but control when it is published.
    2. Manually create a Fenix/Focus/Klar release on the Production track with a 25% rollout.
      • If an app is still pending review on a closed track then you cannot promote it to a production track.
      • See Prepare and roll out a release for more information.
    3. Submit the Fenix/Focus/Klar release for review.
      • Keep the same release notes as the major release. Add any new notes at the top prefixed by the dot release version.
      • With managed publishing turned on, the app will not automatically release after the store review.
  3. Follow the same process as monitoring when an RC build has QA sign-off with the following exceptions:
    1. Turn off managed publishing in the Play Console.
      • With managed publishing turned off, the app will be automatically released after store review.

The following tasks need to be performed during go-live of a dot release

  1. Follow the same steps as an RC go-live, with the following exceptions:
    • The dot release is rolled out at 25% but there is no schedule drop to 0%
      • Please Note: Under chemspill situations, a greater rollout percentage may be required.
    • There may not be any security advisories.

The following tasks need to be performed 24 hours after go-live of a dot release

  1. Follow the same steps as after an RC go-live, with the following exceptions:
    • If there are no blocking issues then bump the rollout to 100% after 24 hours

The following tasks need to be performed daily after go-live of a dot release

  • Follow the same monitoring steps as after an RC go-live.
Retrieved from "https://wiki.mozilla.org/index.php?title=Release_Management/Release_Process_Checklist_Documentation&oldid=1252175"