BMO/ChangeNotificationSystem

Bugzfeed has been decommissioned due to lack of usage. This page exists for historical reasons only.

For a more conversational introduction to the notification system, see this blog post: http://mrcote.info/blog/2014/04/04/bugzfeed-bugzilla-push-notifications/

Status

Bugzfeed is available via WebSockets at ws://bugzfeed.mozilla.org/. There is also an example app at http://people.mozilla.org/~mcote/bugzfeed_example/.

Bugzfeed, being a relatively simple project, is currently serving its function well; there are no plans for further development at this time.

Team

BMO team (dkl, glob, mcote), peterbe (advisor from web engineering)

Problem

With a focus on Bugzilla as a platform, facilitating responsive, JavaScript-based front ends, detecting changes to bugs in a timely fashion is increasingly important. At the moment, the only way to determine if a bug has been recently updated is to poll; there is no push mechanism of any kind. This model has inherent problems, including but not limited to scalability (opening and closing connections is costly) and performance (polling must go through Bugzilla's permission system and other logic layers).

Goals & Considerations

Provide a push-based notification system to inform clients of changes to bugs. Plan for scalability by minimizing server load and time from change to notification.

Providing details of what has changed is not necessary in the notification itself, although preferably this information would be available somehow, perhaps in a separate call.

Non-Goals

Re-implementing Bugzilla's permissions system is not an option. It is complex enough that changing the current model would be major surgery (and would further diverge BMO from upstream), and maintaining two parallel implementations would incur maintenance costs and be error prone.

Notifications should not include changes to dependent/related bugs. This is harder to track based on the current Bugzilla database schema, and a properly designed system should be able to track them independently at the client's discretion. The exception here would be adding or removing dependencies, e.g. a notification will be sent for both bug X and Y if X is set to depend on Y.

Design and Approach

We will implement a separate server that polls the database directly for changes to bugs and passes on *only the ID and change time* of the changed bug to its clients. The clients can then use the main Bugzilla REST API to determine the exact changes, which will enforce permissions as usual.

Clients will be able to subscribe to one or more bugs, and will be able to get a list of their subscriptions and unsubscribe from them. Subscriptions will only be valid for the current session; if a client disconnects, they will have to resubscribe to their bugs.

Important note: this whole design relies upon the idea that knowing the ID of a changed bug is not a security risk. This should be reasonable, given that the only information that is conveyed is that some bug has changed. One could use this information to determine the frequency of changes to a particular bug over some time frame, and hence perhaps an increased interest in a particular bug, but the changes could be to anything--main bug fields, comments, tracking flags, dependencies, etc.

Conceptually, there are three parts:

• Database poller. There would be exactly one process that frequently polls the database for changes (period TBD but on the order of seconds, not minutes). This would keep the time of the last poll in memory and would ask for only the ID of all bugs changed since the time of the last poll.
  • We will use a separate table to hold the notifications, which will be written by a Bugzilla extension called PushNotify. The poller will read entries from this table and publish messages on pulse.

• TCP servers. There would be one or more processes acting as servers that accept client connections and maintain them indefinitely. WebSockets is the preferred protocol for easy integration with browsers. These servers would listen for notifications from the database poller and fan out notification messages to all clients according to subscriptions. For scalability, multiple server processes could be launched with a load-balancer (such as Zeus) spreading out connections amongst them. Note that the server will receive notifications for *all* bugs, since we don't expect there to be many servers. It will be up to the server to implement a subscription protocol and keep track of them.

• Messaging middleware. We are using Pulse for a couple reasons: it's already set up and easily extended, and we can eventually support a variety of transports/servers.

System diagram (courtesy of contributor musingmario):

Implementation

Work on the prototype is tracked in bug 923849.

If you want to hack on the system, see the HACKING.md file in bugzfeed (https://github.com/mozilla/bugzfeed/blob/master/HACKING.md).

Bugzilla and Pulse

The changes are written by Bugzilla using the ZPushNotify extension (http://bzr.mozilla.org/bmo/4.2/files/head:/extensions/ZPushNotify/).

The changes are picked up and delivered to Pulse by the Bugzilla Simple Shim (https://hg.mozilla.org/automation/pulseshims/ - relevant files are bugzilla_simple_shim.py, requirements_bugzilla_simple_shim.txt, and the bottom of config.ini.example).

bugzfeed

The Pulse messages are consumed by and served via WebSockets through bugzfeed (https://github.com/mozilla/bugzfeed).

Bugzfeed takes a number of commands, sent in JSON format. Each command message is an object containing at least a command attribute. Most commands take other, command-specific attributes as well. The server responds to each command with a JSON message containing at least the attributes result and command. result is either the string ok or error; if the latter, the response object also has an error attribute containing a descriptive string. command is the name of the command this message is in response to. Most commands will include other attributes in the response object as well.

Commands

The supported commands are

subscribe

Subscribe to one or more bugs. Attributes for this command are

• bugs: Required. An array of one or more bug IDs.
• since: Optional. A timestamp in UTC in the form YYYY-MM-DDTHH:MM:SS. If provided, any cached notifications (see below) matching those in the given bugs array are immediately sent after the command response.

The response object also includes a bugs attribute, which is an array of all currently subscribed bugs after the command is executed.

unsubscribe

Unsubscribe to one or more bugs. Attributes are

• bugs: Required. An array of one or more bug IDs.

The response object also includes a bugs attribute, which is an array of all currently subscribed bugs after the command is executed.

subscriptions

List subscribed bugs. There are no extra attributes for this command. The response object also includes a bugs attribute, which is an array of all currently subscribed bugs.

version

Get the version of the bugzfeed server. There are no extra attributes for this command. The response object also includes a version attribute, which is a string representing the current version, e.g. 0.3.

Examples

To subscribe to bugs 1, 2, and 3, you would send the message {"command": "subscribe", "bugs": [1, 2, 3]}. If you were currently not subscribed to any others, the response would be {"command": "subscribe", "result": "ok", "bugs": [1, 2, 3]}.

To later unsubscribe from bug 1, send {"command": "unsubscribe", "bugs": [1]}, to which the response would be (assuming no subscribe/unsubscribe commands were sent in between) {"command": "unsubscribe", "result": "ok", "bugs": [2, 3]}.

Caching

The bugzfeed server caches the last 10 000 notifications it has received. This mechanism was put into place to account for brief client connections without maintaining per-connection state on the server. If the connection drops, after reconnecting a client can resubscribe to all its bugs and provide the time of the last known update to see if any updates have occurred while it was disconnected. It's suggested to use the exact time of the last update, not one second later, since technically multiple bugs can be updated in the same second. Just be aware that the last known update will be resent in this case, as all cached notifications with timestamps equal to or greater than the the value of the since are sent.

Note that caching is implicitly performed on all other pieces of the notification system. If the pulse shim goes down, Bugzilla will continue to write updates to the database, which will be read when it is restarted. The bugzfeed production server is configured to use a durable pulse queue, so if it goes down, it will catch up on all queued messages when it restarts.