User:NThomas:AUS throttling

Introduction

AUS2 supports two types of throttling, which are applied in this order of precedence

• app-version-channel specific
• global

In both cases, an update query which carries the parameter force=1, eg

https://aus3.mozilla.org/Firefox/.../update.xml?force=1

will not be throttled. Checks which are initiated by the user have this parameter, while background checks (initiated by the app itself) do not.

In both cases the throttle is number from 0 to 100, used in the sense of a percentage a gas pedal is pushed down. So

• 0 means stopped, no requests will be served an update
• 100 means full speed, all requests will be served an update
• 25 means quarter speed, 1 in 4 requests will be served an update (on average!)

These are configured in config-dist.php. The machines that serve AUS symlink/copy config-dist.php to config.php.

Specific Throttling

This mode allows you to configure a throttle level for an application and version, and to set exceptions for channels that it should not apply to. The version comes from the request, rather than the update served for that request.

eg, Updates from Firefox 3.6.17 are turned off on the release channel, but not beta or either test channel

$productThrottling = array(
    'Firefox' => array(
        '3.6.17' => 0
    )
);
$throttleExceptions = array(
    '3.6.17' => array (
        'betatest',
        'releasetest',
        'beta'
    )
);

We always add exceptions for the two test channels, so that QA and RelEng can test updates without any randomness.

Global Throttling

RelEng almost never uses this because an emergency stop is much quicker by chmod'ing snippets directories than writing a patch, getting review, landing, taging, and asking IT to deploy. It also doesn't block queries with ?force=1.

To use it, eg to stop all (background) updates, set the following in config-dist.php

define('THROTTLE_GLOBAL',true);
define('THROTTLE_LEVEL',0);

Common Use Cases

Prior to the rapid release cycles we mostly used throttling in conjunction with major updates, that is updates that connected one branch to another. It's not yet clear what will happen in the rapid release world. There are two common cases, and we'll work through the complete release cycle for both.

To set the scene, lets suppose we are releasing MyApp 1.0.x and 2.0.y today. There are minor updates for all prior 1.0 versions to 1.0.x, and the same setup for 2.0.y. We have also prepared and tested a major update from 1.0.x to 2.0.y, which will prompt the user to upgrade to the newer branch. We want to be confident that 2.0.y contains no regressions before we offer it to our large base of users on 1.0.x, so at first we will throttle the major update. Only users which manually go Help > Check For Updates (or Help > About > Check for Updates) will be offered the major update. Later we will advertise (or unthrottle) the major update to all requests.

Key

The following conventions for cell formatting are used below

Formatting	Meaning
1.0.x	Throttled
1.0.x	Unthrottled
1.0.x	Changed since last snapshot

Each table shows the update destination for a request of Version and Channel.

Unthrottled major update present at release start

Channel	Version						AUS Config
	Old 1.0	1.0.x-1	1.0.x	Old 2.0	2.0.y-1	2.0.y
We start off with 1.0.x-1 and 2.0.y-1 released, and an unthrottled major update between them:
betatest, releasetest	1.0.x-1	2.0.y-1	-	2.0.y-1	-	-	$productThrottling = array( 'MyApp' => array( ) ); $throttleExceptions = array( );
beta	1.0.x-1	2.0.y-1	-	2.0.y-1	-	-
release	1.0.x-1	2.0.y-1	-	2.0.y-1	-	-
After the automation finishes build1 for 1.0.x and 2.0.y (but not the major update) we have:
betatest, releasetest	1.0.x	1.0.x	-	2.0.y	2.0.y	-	$productThrottling = array( 'MyApp' => array( ) ); $throttleExceptions = array( );
beta	1.0.x-1	2.0.y-1	-	2.0.y-1	-	-
release	1.0.x-1	2.0.y-1	-	2.0.y-1	-	-
Channel	Version						AUS Config
	Old 1.0	1.0.x-1	1.0.x	Old 2.0	2.0.y-1	2.0.y
And after generating the major update and pushing to test channels we have:
betatest, releasetest	1.0.x	1.0.x	2.0.y	2.0.y	2.0.y	-	$productThrottling = array( 'MyApp' => array( ) ); $throttleExceptions = array( );
beta	1.0.x-1	2.0.y-1	-	2.0.y-1	-	-
release	1.0.x-1	2.0.y-1	-	2.0.y-1	-	-
When we push the minor updates to beta throttling is already off on 1.0.x:
betatest, releasetest	1.0.x	1.0.x	2.0.y	2.0.y	2.0.y	-	$productThrottling = array( 'MyApp' => array( ) ); $throttleExceptions = array( );
beta	1.0.x	1.0.x	-	2.0.y	2.0.y	-
release	1.0.x-1	2.0.y-1	-	2.0.y-1	-	-
Channel	Version						AUS Config
	Old 1.0	1.0.x-1	1.0.x	Old 2.0	2.0.y-1	2.0.y
On release day we first push the minor updates:
betatest, releasetest	1.0.x	1.0.x	2.0.y	2.0.y	2.0.y	-	$productThrottling = array( 'MyApp' => array( ) ); $throttleExceptions = array( );
beta	1.0.x	1.0.x	-	2.0.y	2.0.y	-
release	1.0.x	1.0.x	-	2.0.y	2.0.y	-
Then we ensure that 1.0.x is throttled on beta & release:
betatest, releasetest	1.0.x	1.0.x	2.0.y	2.0.y	2.0.y	-	$productThrottling = array( 'MyApp' => array( '1.0.x' => 0 ) ); $throttleExceptions = array( '1.0.x' => array ( 'betatest', 'releasetest' ) );
beta	1.0.x	1.0.x	-	2.0.y	2.0.y	-
release	1.0.x	1.0.x	-	2.0.y	2.0.y	-
Channel	Version						AUS Config
	Old 1.0	1.0.x-1	1.0.x	Old 2.0	2.0.y-1	2.0.y
Now we can push the 1.0.x -> 2.0.y major update on beta & release
betatest, releasetest	1.0.x	1.0.x	2.0.y	2.0.y	2.0.y	-	$productThrottling = array( 'MyApp' => array( '1.0.x' => 0 ) ); $throttleExceptions = array( '1.0.x' => array ( 'betatest', 'releasetest' ) );
beta	1.0.x	1.0.x	2.0.y	2.0.y	2.0.y	-
release	1.0.x	1.0.x	2.0.y	2.0.y	2.0.y	-
Some days/weeks later we can advertise the major update by removing the throttling (and setting it up for the next release):
betatest, releasetest	1.0.x	1.0.x	2.0.y	2.0.y	2.0.y	-	$productThrottling = array( 'MyApp' => array( '1.0.x+1' => 0 ) ); $throttleExceptions = array( '1.0.x+1' => array ( 'betatest', 'releasetest' ) );
beta	1.0.x	1.0.x	-	2.0.y	2.0.y	-
release	1.0.x	1.0.x	-	2.0.y	2.0.y	-
Then the cycle begins again.