Overview

Audio Data API

Background links

• API Spec and Wiki Page
• Bug 490705

Security

• Is this feature a security feature?
  • No.

• Is system or subsystem security compromised in any way if your project's configuration files / prefs are corrupt or missing?
  • No.

• If any content or UI is displayed to the user, in what context is that content presented? Does it have chrome privileges, for example?
  • No UI.

• Does the feature include any new cryptographic functions or other security-critical code?
  • No.

Data

• What data is read or parsed by this feature?
  • This API exposes two new data pipes to content scripts. The first is a read-only framebuffer (via a DOM event) of raw audio data (floats) representing the decoded audio information in a media element (video or audio). The decoded audio information is the same information that is sent to the audio hardware created by the internal audio decoders from the network stream. The second is a set of functions allowing JavaScript developers to create and write to an audio stream.

• What types of validation are done on data inputs (e.g., type checking, string decoding/encoding, etc.)?
  • When writing data to an audio stream from JavaScript, users must pass a regular JavaScript Array or a Typed Float32 Array. Passing other JavaScript objects or primitives (including null, undefined, etc.) will cause an error to be thrown (we have tested sending every possible type, which is how we found a few crash bugs before landing). Next, we check to see how much data the underlying audio stream can write without buffering, and only write this amount. We do this in order to avoid overflowing the various audio buffers with overly large arrays. Also the amount of the data array items must be divisible by the channels count.
  • When initializing the audio stream for writing (the mozSetup method), the feature validates "channels" parameter to be non-equal to 0 to avoid the division by zero error during validation of the amount of the written data that has to be divisible by the channels count (see previous paragraph). The "channels" and "rate" parameters further passed to the nsAudioStream implementation -- the feature expects nsAudioStream's Init method return an error if these parameters are incorrect or audio hardware is not capable to support the audio writing.
  • The check is placed on users' ability to alter framebuffer sizes. We allow users to set a new framebuffer size for audio events (default is channels * 1024), but force them to pick a value between 512 and 16384. Any other value causes an error, and the default value to still be used. We did this in order to make sure that users can't set very large sizes and request too much memory, or very small sizes and flood the event system.

• What is the output of this feature?
  • As mentioned above, raw audio data in the form of a framebuffer (Typed Float32 Array) is exposed to the DOM. The feature outputs raw audio data if the media (the source for the data) and the HTML element pass the "same-origin" check.

• What storage formats are used?
  • No data is ever stored.

Configuration

• Can the end user configure settings (via UI, about:config, or environment variables)?
  • Preferences affecting same-origin checks will influence the availability of the framebuffers. We perform an origin check before releasing audio data into the DOM. In order to use audio from file:/// URIs, for example, the security.fileuri.strict_origin_policy pref has to be turned off (it is on by default).

• Are there build options for developers (e.g. #ifdefs, ac_add_options)
  • This feature integrates with the main media features, and would be turned on/off as they were.

• What are its on-going maintenance requirements (e.g. Web links, perishable data files)?
  • The Audio Data API wiki page currently serves as the most reliable and complete set of documentation, and needs to be kept up-to-date.

Review comments

Notes and bug numbers will be recorded here. Let's try not to spend too much time on any one topic during the meeting.

• As far as we know there are no limits on # channels or sampleRate -- just passed to the audio code and if it errors we throw otherwise we use it
  • need to fuzz bogus numbers of channels and sampleRates
  • on various hardware/drivers
• There's a same-origin check on reading audio data
  • don't know if it's on the src URI or after following redirects (but should mimic <canvas>)
  • https://mxr.mozilla.org/mozilla-central/source/content/html/content/src/nsHTMLMediaElement.cpp#707
• Can't read data from non-audio files (decoder would fail, the API hooks in after the decoder)