Deprecated

The project described by this proposal was never implemented. The Socorro team did implement a script that scrapes the FTP site to discover newly released versions on various channels. The information is not available via an API.

Goal

To provide a service that contains canonical data about product releases / builds. This data will be generated by the release enginering build automation tools, and it will initially serve the Metrics and Socorro teams, but the goal is to make it available to anyone via a simple REST API.

Background

When builds are created, they are pushed to an FTP site and there is a long checklist of things that must be coordinated at that point. Some of these coordination items involve notifying other teams of the release details. Several teams in Mozilla need fresh data about releases to be able to update tools, services, or sites to accommodate them. Sometimes a team does not get notified or the notification isn't properly handled and that can cause systemic problems. A canonical service to manage the data should alleviate many of these problems and provide a clean audit trail when actions are not taken by a team for a particular release.

Deliverables

Stage 1

An authorized REST API for pushing new build data into the system

used solely by the build automation system.

A REST API for pulling/querying data

A simple tool that can be easily integrated with other teams/tools or even just interacted with manually is preferable. It needs to be able to return a specific document based on key, or a set of documents using filtering for a list of builds matching particular characteristics is needed.

Stage 2

A REST API for subscribing for push events

In many cases, it would be better for an external system to be notified whenever a new build is started/completed. The subscription should allow filtering for only desired builds and provide some end point to which the build information can be posted.

Trending and aggregation metrics

It might be useful to provide a simple way to show aggregate statistics on the generation of builds. This might not be needed because there are several other tools that already look at various statistics about builds.

Schema

The data will be stored in a simple JSON format. The initial set of attributes is dependent on feedback from Release Engineering.

Version 1

Core required attributes

{
  product_name: <string>,
  version: <string in version format>,
  major version: <string in version format>,
  build_id: <string typically in YYYYMMDDHHMM format>,
  build_target: <string e.g. WINNT_x86-msvc, Darwin_x86-gcc3-u-i386-x86_64>,
  build_locale: <string in locale format>,
  channel: <string e.g. release, beta, aurora, nightly, default, release-cck-XXX>,
  beta_number: <string>
}

Question: why is the beta number a string, rather than an INT?

Metrics required attributes

{
  application_guid: <string>,
  distribution_name: <string>,
  distribution_version: <string>,
}

Optional attributes

Some of these would need definitions and/or validation that they make sense or would be useful.

{ // core attributes...
  ftp_uploads: [{ path: <string>, time: <timestamp>, label: <string> }, ...],
  aus_pushes: [{ path: <string>, time: <timestamp>, label: <string> }, ...],
  source_control_repo: <string e.g. moz-central>,
  source_control_branch: <string ???>,
  source_control_label: <string such as commit id?>,
  descriptive_name: <string e.g. Firefox 8.0b1>,
  urls: [{url: <url>, label: <string>}, ...],
  foreign_keys: [ {system: <string>, reference: {table: <string>, column: <string>, value: <int>}}, ...],
  <Anything else?>...
}

Implementation notes

The metrics team intended to build stage one of this service by 2011-07-20 using Bagheera as the REST API and storing the data to both a simple disk file for backup and to ElasticSearch to provide access to the data.