Sandbox:Release:Automation:Updates

From MozillaWiki
Jump to: navigation, search

Update/snippet Generation (Firefox only)

Code used:

Deliverables:

  • Updated patcher config file, checked into CVS
  • Partial mar file per-platform per-locale
  • Update snippets on test & live channels
  • Snippet comparison log

After checking out the necessary modules and tools this builder goes to work doing the following:

  • Bumping & checking in a patcher config file
  • Cloning sourceRepo & building update-packaging tools
  • Creating & uploading partial MARs
  • Creating & uploading update snippets
  • When applicable, creating snippets for previous builds of the release
  • Pushing test snippets live
  • When applicable, doing a comparison of live channel snippets vs. releasetest

This step is generally pretty failsafe but in the event that it needs to be run a second time or restarted at later step it's necessary to comment out the patcher config bump step (see below for details on how). Until bug 466999 FIXED is resolved the patcher config file can only be bumped once.

Note that both the sourceRepo and the tools repo will be updated to the patcherToolsTag. If the update-packaging tools or patcher config script has been updated since the last release it is a good idea (and sometimes necessary) to create a new UPDATE_PACKAGING_R# tag before starting the release. ReleaseEngineering/PatcherTags documents the tag history, please keep it up to date.

Snippet Comparison Log Analysis

Once updates is completed there will be a log from the snippet comparison step. This step compares the contents of the releasetest channel snippets to the contents of the live channel (beta or release) snippets. There's a few things to note about this log:

  • It will usually contain warnings about missing snippets for alphas and betas. These are expected and ignorable.
  • When there are warnings about other snippets missing, a 'Warnings' log will be created and the step will be orange. These must be investigated and explained/fixed before any live snippets are pushed.
  • If any snippets differ in contents between the test and live channels a 'Diffs' log will be created and the step will be orange. The Diffs log will list the files which differ; the full diffs can be viewed in the stdio log. Any differences must be investigated and explained/fixed before any live snippets are pushed.

Update Verify (Firefox only)

Code used:

Deliverables:

  • Update verify logs per platform. Details on them are below.

Update verify, in its current form, verifies a few different things:

  • That the "updater" binary works
  • That all MAR files can be applied, and result in a functionally equivalent build to the installer.
  • That all previous versions (across all locales), have snippets available

It does this in two different types of tests:

  • Full update verify unpacks a previous version's installer, pings AUS, downloads the MAR it finds, applies it, and diffs the result against the current version's installer. Differences between the two are reported in the log.
    • Run for all locales in the oldVersion
    • Run for select locales ("de", "en-US", "ru" at the time of writing) for all other previous versions
    • Completes and partials (if present) are both tested
  • Quick update verify does a subset of the above: it pings AUS, downloads the MAR it finds, and ensures that it is non-zero in size
    • Run against previous versions for all locales not tested as part of the full update verify.

Update verify is split into multiple chunks per platform (10, by default). The full and quick checks are spaced evenly throughout the chunks giving each one roughly equivalent run time.

Log Analysis

This step is pretty good at reporting errors. Most errors will turn the step red, and be pretty clearly visible in the log. Nonetheless it's a good idea to have a scan through the log for anything abnormal (searching for FAIL and WARN are helpful here).

Here's some common failures:

  • "Binary files source/bin/freebl3.chk and target/bin/freebl3.chk differ" (or softokn3.chk) on win32 in the complete MAR
    • This is normal in the win32 logs and is caused by bug 404340. Once this bug is fixed it should go away.
  • "CRC check failed" and/or "LoadSourceFile failed" on a partial MAR
    • This typically happens when the previous release's complete MAR differs from the complete package. Because partial MARs are generated by diffing the previous complete MAR with the current complete MAR the partial will fail the CRC check when this happens. There's nothing that can be done about this, because the problem is with an already-release-build. The correct thing to do here is inform release-drivers that there will not be partials available on this platform, trim them out of the candidates directory, and remove the partial snippets.
  • "FAIL: partial from xxxxx wrong size", "FAIL: actual size: 0"
    • This is usually caused by missing MARs. You should first check to see if the MAR in question exists in the candidates directory. If it doesn't, go back and look at the logs to see if there was a problem generating or uploading it. If it does exist in the candidates directory make sure the permissions are set properly (ffxbld:firefox, 664).
  • "FAIL: Could not download source xxx"
    • Usually indicates missing packages or bad permissions in the candidates directory. Take the same action as above.
  • "FAIL: no partial update found for xxx" or "FAIL: no complete update found for xxx"
    • Check the logs from the previous step to make sure 'pushsnip' was run successfully. This error usually indicates that snippets weren't pushed.

Major Update (Firefox only)

Code:

As part of any release that is on a less than current branch we generally create and push a Major Update as part of it. For example, when we did a 3.5.8 release we generated a major update for 3.5.8 -> 3.6 alongside it. Note that major updates happen as part of the "from" release, not the "to" release.

Major update generation is not kicked off automatically because it depends on both the "to" and "from" releases having had their builds and repacks signed. Once they are at that point "Force Build" can be used to initiate the Major Update generation.

If either of the "from" or "to" releases respin at any point you must regenerate the major update. If the "to" release respins, majorUpdateBuildNumber needs to be updated.

At the end of the factory snippets will be uploaded to the AUS server and the test snippets will be pushed live. After that, the major update verify builders will be triggered.

Major Update Verify (Firefox only)

Everything in the previous "update verify" section applies here.

Generally, major updates generate more harmless warnings than other releases. Analysis on these is done when testing a new major update and noted on release notes. When verifying a major update it's always helpful to look at the previous one's notes to find a list of the harmless warnings. If you find warnings that are NOT in the previous major update's notes they must be investigated.

It's rare to see major update verify builders go green because of all of the harmless warnings.

Pre Push to Mirrors (Firefox only)

Code: scripts/release/push-to-mirrors.py

Note: you can run check perms, AV, update verify and even push to mirrors in parallel with update_verify, at the same time QA may check the builds/updates. It's worth waiting for update verifies before you push the snippets live to the release channel, however.


Check Permissions

This builder:

  1. Checks if the FTP directories and files have proper permissions and modes
  2. Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

This builder should be run second time before pushing file to mirrors by hitting "Rebuild" button or forcing the builder with 2 properties set:

  • Property 1 name: script_repo_revision: use FIREFOX_*_RELEASE tag here
  • Property 2 name -> release_config: mozilla/release-firefox-mozilla-2.0.py

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

Antivirus check

This builder:

  1. Runs antivirus locally ← report these results in build notes
  2. Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

If a long time has passed since the first run, this builder should be run a second time before pushing files to mirrors by hitting "Rebuild" button or forcing the builder with 2 properties set:

  • Property 1 name: script_repo_revision use FIREFOX_*_RELEASE tag here
  • Property 2 name -> release_config: mozilla/release-firefox-mozilla-2.0.py

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

Prepare for Beta Release (Firefox only)

Code: /cvsroot/mozilla/tools/release/bin/backupsnip

This script preserves the current (prior to release) state of the snips, to prepare for rollback if there are serious problems with the new release. As of Q2 2012, it takes around 10-20 minutes to run.

Manual step (no builder):

  1. ssh as yourself to aus3-staging.mozilla.org
  2. sudo to user ffxbld
  3. find the existing snip directories for this release:
    1. cd /opt/aus2/snippets/staging/
    2. ls -d Firefox-VERSION*
      • if there are directories ending in *.time, it means the corresponding snippets have already been pushed. Do not run backupsnip for snippets that have been pushed.
      • TODO: confirm never backup *-test snippets
    3. Backup the current snippets under the new release's name.
    4. ~/bin/backupsnip Firefox-VERSION-buildN
      • TODO: confirm backupsnip uses new release dir name
      • TODO: find out what procedure s/b for build numbers > 1

Beta Release (Firefox only)

Code: /cvsroot/mozilla/tools/release/bin/pushsnip

This script publishes the update snippets on the internal mirrors. As of Q1 2012, it takes around 45 minutes to run.

Manual step (no builder):

  1. ssh as yourself to aus3-staging.mozilla.org
  2. sudo to user ffxbld
  3. find the existing snip directories for this release:
    1. cd /opt/aus2/snippets/staging/
    2. ls -d Firefox-VERSION*
      • if there are directories ending in *.time, it means the corresponding snippets have already been pushed. Do not run pushsnip for snippets that have been pushed.
      • TODO: confirm when to push *-test snippets
    3. Publish the current snippets:
    4. time ~/bin/pushsnip Firefox-VERSION-buildN
      • record runtime & completion in build notes

Push to Mirrors (Firefox only)

Code: scripts/release/push-to-mirrors.py

This builder:

  1. Runs rsync and pushes files to the release directory
  2. Triggers mirror uptake monitoring poller.

To trigger this builder you need to specify some build properties:

  • Property 1 name: script_repo_revision use FIREFOX_*_RELEASE tag here
  • Property 2 name -> release_config: mozilla/release-firefox-mozilla-2.0.py

Press 'Force Build' in the Buildbot waterfall.

No reconfig needed for this builder.

Final Verification (Firefox only)

Code:

Once mirror uptake has reached 50 or 60% this builder can be started with 'Force Build' from the Buildbot waterfall.

This is a simple test which is run after mirrors are propagated to ensure there's no (major) problems with them. It does the same thing as the second part of Update Verify: for every locale/platform/version combination it checks for an update snippet, and then performs an HTTP HEAD request on the MAR the snippet points to.

The step will turn red if there are http codes other than 200 or 302, or if FAIL is found in the log. If it does this, you should have a look in the log, and find out which mirror is failing. It's a good idea to keep an eye on that mirror for awhile to make sure it finishes getting all of the files.

Searching the log for errors is made easier by downloading it and running the following locally: 

grep HTTP/1 text | grep -v 302 | grep -v 200
grep FAIL text

Bouncer Submitter (Firefox only)

Code:

Configuration:

  • firefox-tuxedo.ini for stable versions, and firefox-devpreview-tuxedo.ini for Alpha releases.
  • BuildSlaves.py file should contain the following variables:
    • tuxedoUsername: Tuxedo username with appropriate permissions for adding new entries via API
    • tuxedoPassword: password for this user

This builder can be started any time with 'Force Build' from the Buildbot waterfall.

This step adds adds bouncer entries for installers, complete and partial updates for all platforms listed in enUSPlatforms and l10nPlatforms and locales specified in shipped-locales (en-US is being added in any case).

The step will turn red if tuxedo-add.py exits non zero. There are 2 major error sources:

  • tuxedoUsername has not enough privileges to add entries
  • Tuxedo is not aware of new locales added in the current version. Usually the log contains "HTTPError: HTTP Error 400: BAD REQUEST". See this link for an example workflow on how to resolve this problem.

Unlock Build Slaves

Once you are done with the build machines that you locked to your buildbot master, unlock them in slavealloc and clear the comments field.

Retrieved from "https://wiki.mozilla.org/index.php?title=Sandbox:Release:Automation:Updates&oldid=439964"