Accelerated SVG And CSS Filters

Requirements

Spec: https://dvcs.w3.org/hg/FXTF/raw-file/default/filters/index.html

There are three kinds of filters we need to support:

• CSS filter shorthands (e.g. opacity(), grayscale())
• SVG filters
• Custom filters (vertex and fragment shaders defined in GLSL)

CSS filter shorthands are defined by mapping into SVG filters and custom filters in a fairly straightforward way, so we can limit our attention to the latter two if we want to.

There are a few places we will want to use filters (sooner or later):

• CSS 'filter' property applied to an element, which makes the element into a stacking context and filters the element and its descendants as a group. In general this requires that we support filters in the OMTC compositor *and* in non-compositor content-drawing contexts such as printing, drawWindow and -moz-element().
• Some yet-to-be-proposed 'globalFilter' 2D canvas attribute, which applies the filter when rendering canvas primitives.
• The 'filter()' CSS image value, which applies a filter to an image wherever a CSS image value can be used (e.g. background-image).

We currently support SVG filters on all platforms, and we shouldn't regress that. It should be OK to limit support for GLSL custom filters to platforms where we support WebGL.

Obviously we want to use the GPU for filter processing wherever possible, especially in situations where we don't need the results of filter processing to be read by the CPU.

One non-obvious requirement is that we want to minimize the dependence of filter processing time on the pixel values being filtered, to protect against timing-based data extraction attacks.

Implications

We will want some kind of Layers API for attaching filters to layers. This is essential for OMTC to work with filters.

We will want some kind of Moz2D API for 2D drawing with filters. This is needed for canvas 'globalFilter' to be efficient, and will be very helpful for supporting filters in BasicCompositor (which implements OMTC using Moz2D).

These APIs should both consume the same kind of internal representation of filters, so we can share code for mapping CSS filters, SVG filters and custom filters into our internal representation. This representation needs to be IPDL-compatible so we can easily ship it as part of a layer transaction.

Filter Objects

We could try to map all filters down to GLSL and support only GLSL filters (using ANGLE on Windows). This would make it difficult to have a performant CPU-only implementation of standard filters --- we could pattern-match particular GLSL filters in a CPU implementation (and reject the rest entirely, probably), but that could be more fragile than having a real API that captures which filters are supported by CPU implementations. Also, because a single filter can be a sequence of (vertex shader, fragment shader) custom filter primitives, and an SVG filter can be a DAG of filter primitives, we can't boil a filter down to a single (vertex shader, fragment shader) combination; we'd need something at least as complex as a DAG of GLSL filters.

Proposal: make a filter object a DAG of filter primitives, where each filter primitive is one of the SVG filter primitives (which could be a GLSL custom filter)

Many filter primitives take parameters. There are a few things to watch out for:

• The common animation infrastructure shared by the compositor and style system needs to be extended to support interpolation between filter objects, with their parameters.
• When a CSS transition or animation applies to parameters of filter primitives in an SVG <filter> element (not yet supported in Gecko, but will have to be eventually since the SVG and CSS WGs have resolved to promote some of those parameters from attributes to CSS properties), it can affect the rendering of the filtered content but there is no easy way to express this to the OMTC compositor so it can do OMTA of those parameters.
  • Proposal: ignore this issue for now and not support OMTA of those parameters
• Some primitives (<feImage>, custom filters) take parameters that are images. These will have to be external to the filter object and supplied separately when the filter object is used.
  • Proposal: When using a filter object with Moz2D, pass in a list of Moz2D SourceSurfaces to use as the image parameters
  • Proposal: When attaching a filter object to a Layer, attach a list of special child Layers to use as the image parameters (similar to the way we attach a special child Layer to mask with)

Implementation

There are lots of ways we could organize this. At our filters meeting in Taipei, we agreed on an approach like the following (with some interpolation by roc on issues we skirted around). API names are just placeholders.

• Start by adding a Moz2D API for filters.
  • Define an object which describes a graph of filter primitives. Let's call it the FilterDescription.
  • Moz2D DrawTargets provide an API to construct a backend-specific Filter object from the FilterDescription.
  • The Filter object contains methods for setting parameters for each filter primitive. Parameters can be images (SourceSurfaces), scalars, lists of scalars.
    • It might make sense to bake "enum" parameters, and the length of each list of scalars, into the FilterDescription since it doesn't make much sense to animate those (though they are technically animatable).
  • The Filter object does not allow changes to the graph structure.
  • Moz2D DrawTargets provide a DrawSurfaceWithFilter API similar to DrawSurfaceWithShadow (and will eventually replace DrawSurfaceWithShadow), which takes a Filter.
• Create a support library which works with DataSourceSurfaces that can be used by any Moz2D backend to implement Filter via readback. Skia code might be useful.
  • This library should use integer arithmetic as much as possible to avoid timing attacks.
• Write code to convert an SVG filter element into a FilterDescription and set of parameter values.
• Write code to convert a CSS filter value into a FilterDescription and set of parameter values. This would depend on the SVG filter converter for CSS filter values that reference an SVG filter.
• Add some kind of canvas-2d 'globalFilter' attribute that takes a CSS filter value and uses this new filter code. This is good for writing tests and won't break any existing filter support.

It probably makes sense to slice the work horizontally by just supporting a couple of filter primitives and doing all the above work for them before adding the rest.

Additional work required:

• D2D/D3D Filter backend
• GL Filter backend
• Custom GLSL filters (GL/D3D only)
• Define a LayerFilter object that a LayerManager constructs from a FilterDescription, that supports setting of filter parameters, and that can be attached to a Layer
• Implement LayerFilter backends for our LayerManager implementations. Would mostly reuse Moz2D implementation code.
• OMTA for LayerFilter parameters

As soon as all Moz2D DrawTargets support the complete set of (non-custom) SVG filters, we can remove our existing SVG filter implementation.

Unaddressed Issues

Filters can reference the page contents behind a filtered element using BackgroundImage/BackgroundAlpha. Supporting this is somewhat orthogonal to the above concerns.