Labs/Jetpack/JEP/18

JEP 18 - Audio

• Champion: Anant Narayanan <anant at mozilla dot com>
• Status: Implementing
• Type: API Track
• Created: 6 July 2009
• Reference Implementation: jetpack.future.import("audio")
• JEP Index

Introduction and Rationale

The Audio module allows the developer to record audio to an Ogg/Vorbis file in the browser. In the future, we also plan on adding capability for processing that audio and encoding it into files.

Proposal

The Audio module will be available under jetpack.audio.

To begin recording to a file, call: jetpack.audio.recordToFile.

To stop the recording, call: jetpack.audio.stopRecording. This function will return the path of the Ogg/Vorbis file that the audio was recorded to.

To check if a recording is in progress (simultaneous recording of audio is not allowed), access the jetpack.audio.isRecording property.

Example

This Jetpack shows a 'Record' button on the status bar, which when clicked will start recording audio to a file. On clicking it again, the recording is stopped, a notification with the file path is displayed and the audio is played back.

var path = '';
jetpack.future.import('audio');

jetpack.statusBar.append({
  html: 'Record<i>!</i>',
  width: 55,
  onReady: function(w) {
    $(w).click(function() {
      if (jetpack.audio.isRecording) {
        path = jetpack.audio.stopRecording();
        jetpack.notifications.show("Saved to " + path + ", now playing!");
        $(jetpack.tabs.focused.contentDocument).find('body').
          append('<audio src="file://'+path+'" autoplay="true"></audio>');
      } else {
        jetpack.notifications.show("Recording...");
        jetpack.audio.recordToFile();
      }
    });
  }
});

Streaming

The ability to manipulate the incoming audio stream would be useful. The Jetpack audio module allows access to the raw audio stream. Here's an example:

jetpack.future.import("audio");
var start = 0;
var samples = 0;

function myCallback(data, num) {
  samples += num;
}

jetpack.statusBar.append({
  html: 'Stream<i>!</i>',
  width: 45,
  onReady: function(doc) {
    $(doc).click(function() {
      if (jetpack.audio.isRecording) {
        jetpack.audio.stopRecording();
        jetpack.notifications.show("Received " + samples +
          " in " + (Date.now() - start) + " seconds.");
      } else {
        start = Date.now();
        jetpack.audio.recordToPipe(myCallback);
        jetpack.notifications.show("Started recording to pipe");
      }
    });
  }
});

The callback receives a byte array and the number of samples contained in that array. The array consists of PCM float stereo samples at a frequency of 44000Hz (pretty high-quality audio!).

We're working on adding support for re-encoding this byte stream (before or after manipulation) back into an OGG/Vorbis file. Check out http://hg.mozilla.org/labs/jetpack/file/tip/components/experimental/audio/AudioEncoder.cpp if you want to help!