Platform/GFX/Moz2D

The Moz2D graphics API, part of the Azure project, is a cross-platform interface onto the various graphics backends that Gecko uses for rendering such as Direct2D (1.0 and 1.1), Skia, Cairo, Quartz, and NV Path. Adding a new graphics platform to Gecko is accomplished by adding a backend to Moz2D (see #Implementing a new backend).

Some of the notable features of the API are:

• Mostly stateless—better suited to CSS rendering and eliminates overhead
• Floating-point—better suits platform APIs
• API methods line up with HTML canvas

Current status

The Gecko graphics API preceding Moz2D was Thebes which mostly consists of a C++ wrapper around Cairo, along with some Gecko-specific utility code, and a text API that uses platform text handling. Currently both the Thebes API and Moz2D API are in use. A Thebes wrapper around Moz2D allows code that still uses the Thebes API to run on top of Moz2D. Over time this code will be migrated to using the Moz2D API directly (see Azure Conversion).

Once the Azure project is completed—that is, all the backends have been implemented and all the calling code converted to use the new Moz2D API—the old Thebes API will be removed.

Side note: For historical reasons, there are some classes with “Thebes” in the name which will be surviving this process as they're not actually part of the graphics API. They may get renamed eventually, to reduce the number of magical words you need to know about, but be warned that just because some code has “Thebes” in the name, it does not mean that the code is on its way out!

See also:

• Progress on Moz2D - July 2013 update
• Looking for a good first place to contribute to Gecko gfx? - August 2013 summary and how you can get involved in moving to Moz2D
• Azure Conversion - Tracking page for the work to use Moz2D directly

Stateful vs stateless

Many of the performance gains anticipated from Moz2D come about due to its mostly-stateless API.

Stateful APIs

The Thebes API wrapped the Cairo graphics library which uses a stateful context model much like Postscript or Quartz 2D. To draw content, you make individual API calls to set up various bits of state in a context object, followed by another API call to actually perform drawing. For example, to stroke a shape with a dashed line, you would typically set the color, set the line width, set the dashing style, start a new path, emit path segments, and finally draw the stroke—all as separate API calls. In Cairo, even drawing an image requires the caller to set the source surface, emit a rectangle path, and fill—at least 6 API calls.

The overhead of setting up state can be justified when that state is re-used, but for CSS rendering this is typically not the case. Rather, for each item that is drawn fresh CSS values are fetched and re-set. Because the browser does not know what it is drawing, it is unable to group operations and re-use state effectively.

On the other hand, the HTML5 canvas element has a stateful API but because its state-tracking differs from that of Cairo drawing state must be tracked twice: once for canvas and once in Cairo.

A further complication specific to Cairo arises because internally Cairo uses a stateless surface API. On OS X, Cairo uses its Quartz backend for all drawing. However, when using Cairo, despite the fact that both Firefox and Safari ultimately use Quartz as their backend for drawing, Safari is faster than Firefox on some demos. We believe one reason this is the case is because Quartz is stateful. As a result, Cairo needs to convert from its stateful API to its internal, stateless surface API, then back to Quartz’s stateful API.

The Moz2D API, despite being mostly-stateless, was designed to map onto stateful APIs like Quartz in a more efficient manner.

Moz2D: Mostly-stateless

Almost all the operations on an Moz2D DrawTarget (see #Draw targets below) take relevant state as parameters and do actual drawing. The only state carried by a DrawTarget is the destination surface itself plus a current transform and a current clip stack. We let the transform and clip state remain in the DrawTarget because those are the only pieces of state not constantly reset by CSS rendering. Our CSS rendering needs to always render under some given transform and clip and we don't want all our rendering code to have to pass those around all the time. Because of this, the Moz2D API is called mostly-stateless.

Learning Moz2D: Introducing the API

Following is an overview of the main pieces of the API. For the details including the most recent changes, please refer to http://mxr.mozilla.org/mozilla-central/source/gfx/2d/2D.h.

The two main components of the Moz2D API are:

• Source surfaces, and
• Draw targets.

Source surfaces

Source surfaces represent a read-only graphics buffer. There are actually two categories of source surfaces as follows:

Both SourceSurface and DataSourceSurface are abstract interfaces that are implemented by the various backends.

A SourceSurface is an opaque representation of a buffer, a handle. That means you can’t access its pixel data directly.

So what can you do?

• Use it in various drawing operations on a DrawTarget. For example,
  • Draw it, or just part of it (DrawSurface)
  • Use it as a mask when drawing a pattern (MaskSurface)
  • Use it as a pattern when filling a rectangle (FillRect after wrapping the SourceSurface in a SourcePattern)
• You can also get a DataSourceSurface from it which does let you access its data.

A DataSourceSurface is a subclass of a SourceSurface that provides direct access to pixel data.

It is possible to get the DataSourceSurface corresponding to a SourceSurface using the SourceSurface::GetDataSurface method.

Where do SourceSurfaces come from?

• DrawTarget::Snapshot() – SourceSurface corresponding to the current contents of a draw target.
• DrawTarget::CreateSourceSurfaceFromData() – SourceSurface for an existing memory buffer (suitable for using with the draw target it is called on).
• DrawTarget::CreateSourceSurfaceFromNativeSurface() – SourceSurface for an existing buffer of some… ?
• DrawTarget::OptimizeSourceSurface() – Converts an existing SourceSurface into one that can be readily used with the given DrawTarget.
• SourceSurface::GetDataSurface() – Gets a DataSourceSurface corresponding to an existing SourceSurface.
• Factory::CreateDataSourceSurface() – Creates an new memory buffer to be used as a DataSourceSurface.
• Factory::CreateWrappingDataSourceSurface() – Creates a DataSourceSurface to wrap an existing buffer.

Each backend provides its own implementation of these interfaces however client code always using the backend-independent interfaces.

The implementation of these interfaces typically takes the following form:

Draw targets

A draw target performs drawing operations on some backing store (e.g. a D3D texture or other buffer).

Includes methods such as the following:

• FillRect
• Fill
• StrokeRect
• StrokeLine
• Stroke
• …

The following methods are stateful:

• PushClip
• PushClipRect
• PopClip
• SetTransform

Typically the arguments to draw commands take the following order:

1. <shape> (Rect / Path) — the exceptions are FillGlyph and DrawSurface
2. A Pattern object that describes the source for filling/stroking
3. A DrawOptions object (alpha, comp-op, anti-aliasing, snapping etc.)
4. Call-specific draw options

Paths

To draw paths, we create a PathBuilder for a DrawTarget using CreatePathBuilder. The PathBuilder has methods such as:

• MoveTo
• LineTo
• …

Finally, calling Finish returns a Path that can be used for drawing (e.g. with Fill or Stroke). After calling Finish the PathBuilder cannot be used again!

Patterns

Patterns describe the pixels used as the source for a masked composition operation that is done by the different drawing commands. The different types of patterns are:

• ColorPattern — a solid color fill
• SurfacePattern — a SourceSurface plus a repeat mode, transformation matrix, and resampling filter
• LinearGradientPattern
• RadialGradientPattern

For linear and radial gradient patterns, the list of gradient stops is backend specific.

These objects are not ref-counted and may be allocated on the stack.

Developing Moz2D

Building Moz2D

The Moz2D project can be developed independently of the rest of Gecko/Firefox by checking out the dedicated repository. The steps for each platform follow.

Linux/Mac

Depending on which backend you are interested in building, the steps are as follows:

Cairo

Dependencies:

• libcairo2-dev (http://cairographics.org/download/)
• If you plan on using Player2D, qt4-sdk (http://qt-project.org/downloads)

Steps:

1. Clone the repository: hg clone http://hg.mozilla.org/users/bschouten_mozilla.com/moz2d
2. On Mac, add the directory containing the cairo.pc file to your PKG_CONFIG_PATH (if it's not there already).
   • One location seen was /usr/X11/lib/pkgconfig
   • Another location seen was /opt/local/lib/pkgconfig
3. On OSX clang++ is required: CXX=clang++
4. Build using one of the following targets:
   • release: make release
   • debug: make debug
   • check: make check
   • player2d: make player2d

When building Player2D beware of the following:

• Currently you need to be building with clang in order for Player2D to build due to a Qt dependency.
• If you don't get qmake producing a Makefile, but instead get a xcodeproj file instead, you may need to create the Makefile yourself, using qmake options (details forthcoming) and run make manually.

Skia

Dependencies:

• Skia:
  • Use a non-debug build. For download and build instructions see https://skia.org/user/download
• If you plan on using Player2D, qt4-sdk (http://qt-project.org/downloads)

Steps:

1. Clone the repository: hg clone http://hg.mozilla.org/users/bschouten_mozilla.com/moz2d
2. make clean release MOZ2D_SKIA=true
3. If you didn't checkout skia under the moz2d src directory then set MOZ2D_SKIA to it's relative path

Windows

Dependencies:

• VS2012, Express 2012 for Windows Desktop is fine
• A default Cairo and Skia build in the cairo/skia directories in the parent directory of Moz2D if you plan to use skia/cairo

Steps:

1. Check out the mercurial repository hg clone http://hg.mozilla.org/users/bschouten_mozilla.com/moz2d
2. Open the gfx2d.sln visual studio solution
3. Build the solution
   • Build the Debug configuration first since it generates the shader code
   • Don't build Release (Skia) without installing Skia / Cairo first

Building Player2D

After building Moz2D, you can build the Player2D project (see below) as follows.

Dependencies:

• QtSDK
  • Use the VS2012 32-bit version
  • You will need at least version 5.1.0 since earlier versions don't have a 32-bit binary for VS2012 and have a bug (QTBUG-30822) that cause Visual Studio to pick up the wrong Windows SDK headers when building in an environment with both VS2010 and VS2012 installed.
  • Older versions can be used if needed. Use one of the installers and select "Source Components" when installing. Then, from a VS2012 command prompt run:
    • C:\Qt\Qt5.0.2\5.0.2\Src\qtbase> configure -opensource -confirm-license -static -debug-and-release -no-angle -no-opengl -nomake examples -nomake tests -platform win32-msvc2012
    • nmake
    • (Requires Perl)

From a VS2012 command prompt:

1. Go to the moz2d repository clone, cd c:\users\me\moz2d
2. mkdir player2d-build
3. cd player2d-build
4. \qt\Qt5.1.0\5.1.0\msvc2012\bin\qmake.exe ..\player2d\player2d.pro -tp vc -spec win32-msvc2012
   (The final -spec win32-msvc2012 might not be necessary, particularly if you only have one version of Visual Studio installed)
5. Open the generated player2d.vcxproj file from the player2d-build directory in VS2012.
6. Press F7 to build.
7. If you get an error about surfaceview.h not being found when referenced from ui_sourcesurfaceview.h, change the line that reads:
   #include <surfaceview.h>
   to
   #include "surfaceview.h"
8. Add the directory containing the QT DLLs to your path (e.g. C:\qt\Qt5.1.0\5.1.0\msvc2012\bin)
9. Run player2d

Recording and playback

In order to test different backends and conduct performance analysis it is possible to make a recording of calls to the Moz2D API—for example, those used when rendering a certain Web page—and then replay them step by step on different backends.

Making a recording

On a platform where Moz2D content is enabled in Firefox (currently only Windows):

• Create a profile for recording if you don't already have a suitable one,
  

 c:\Program Files (x86)\Nightly\firefox.exe -no-remote -ProfileManager

• Assuming we want a recording for http://www.website.com and our profile is called “recording” we can save a recording to mytest.aer with the following command line:
  

 c:\Program Files (x86)\Nightly\firefox.exe -no-remote -P recording -recording http://www.website.com/ -recording-output mytest.aer

Sometimes recordings are prematurely ended and come up empty. This is a known bug.

Playing a recording

Having made a recording it is possible to replay the steps on different backends using the Player2D tool (see #Building Moz2D above). The steps can be played back one by one and it is also possible to step backwards and forwards to inspect the result of each step.

The result is as follows:

(Note that in order to see anything in the View window you must double-click an appropriate object from the Objects window.)

Tools available

Some of the other tools available in the Moz2D repository include:

• recordbench — reports timings for a recording across a range of backends. Using this application we should be able to do automated performance comparisons between different drawing backends.
• micro-benchmark suite — allows us to make detailed performance comparisons on a per-backend basis. Hopefully this will give us the ability to make well-informed decisions as to what drawing backends to prefer on specific platforms/configurations.

Implementing a new backend

So you want to add your own backend… great! Here’s how to get started:

1. Create a recording using Player 2D
2. Implement DrawTarget
3. Implement Factory::CreateDrawTarget
4. Add BackendType to static array
5. Add new backend to TestDrawTarget

Further reading

The following resources describe the motivation for and goals of this API:

• Introducing the Azure Project
• Mozilla 2D graphics (PDF)
• Graphics API Design