Platform/AreWeFunYet

From MozillaWiki
Jump to: navigation, search

This page tracks the status of Gecko as a games platform.

Also see HTML5 Games.

Tracking Bug for Game Related Issues: https://bugzilla.mozilla.org/show_bug.cgi?id=gecko-games

Fast code

Feedback from Games Work Week

Here is a benchmark comparing ActionScript to several JavaScript engines. It shows that as of June 2011, JavaScript is faster than Flash 10.3 in all major browsers. Here's a runnable copy of the v8 flash benchmark.

Box2D Benchmark - This is a great blog post about some experiments done in Dec 2011 comparing different ways of getting Box2D to run in a browser and seeing it performs relative to the C version.

TODO: Find/generate more benchmarks? At least update that one to include Flash 11 and the latest JS engines (Firefox 9 with TI) ?

Known Issues

Garbage collection pauses

An important question is what's an acceptable length for a GC pause. There is no magic value there that will ensure that we maintain 60 FPS: an arbitrarily short pause might make us skip a frame. But there is an important value that will ensure that we don't skip more than one frame: 1s/60 = 16 ms, so as long as GC pauses are no longer than 16 ms, they don't cause us to skip more than one frame. Thus 16 ms seems like the first goal to aim for, allowing us to maintain an effective 30 FPS when GC occurs.

Incremental GC (bug 641025) is on track to achieve this goal. The current version keeps mark phase pause times down to 10ms.

The code for IGC has landed to Nightly, but is disabled by default because of nonreproducible memory leaks on our automated tests (which we are working on). You can turn it on by going to about:config and setting javascript.options.mem.gc_incremental to true.

Potential 10ms+ pauses left over after bug 641025:

  • The sweep phase (mostly because of objects with finalizers, e.g., DOM objects) still runs all at once, and can be 20ms+. Incrementalizing this is much easier that incrementalizing the mark phase and is planned follow-on work.
  • We throw away compiled code on GC, requiring recompilation after the GC is done, which can be a pause. We're aware of the issue and have talked a bit about solutions.
  • The JS program can run more slowly during an incremental GC. It depends on what the JS program does: writing to heap locations (e.g., object properties, array elements) is slowed down, so programs that do a lot of that can be slowed down substantially, but other programs are hardly affected at all.

Generational GC (bug 619558), now underway and expected to finish around mid-year, should further alleviate GC issues. With generational GC, short-lived objects are collected with negligible cost, so a regular incremental GC needs to be started only when there is garbage from long-lived objects that are no longer used.

A further improvement would be to provide an API that means "don't GC now". This would combine well with generational GC: with generational GC, major collections don't need to be done very often so delaying them is unlikely to cause a problem.

Impact of other tabs

See bug 710359. In the web-gaming era, it will not be tolerable that just having a GMail tab on the side severely harms FPS in a game in another tab. This should have been solved by electrolysis ( = process separation), but it's on hold.

For now, bug 712731 and bug supersnappy aim to mitigate the problem.

Performance issues with large JS files

See bug 644244. Large JavaScript files (which are not uncommon in games) can in some cases run slowly. Example large files would help us work on this.

Typed Arrays for Managing Assets

It seems that game developers would like to store assets in binary form in Typed Arrays. How possible/feasible is that? For example, can I say 'take this typed array and interpret it as a JPEG image'?

No you can't. You can convert the data into a Blob and get a URL for that, but that involves a memory copy.

Another orthogonal question: can one cache on disk (IndexedDB?) the contents of a Typed Array?

Yes, you can store Typed Arrays and ArrayBuffers in IndexedDB just like many other data types.

Currently if you want to take data in a Typed Array and display in an <img> or otherwise end up interacting with the DOM, you'll have to copy the data at least once. One solution might be the ability to create a blob-url representing the contents of a Typed Array.

Yes, we should add the ability to get a URL which represents the contents of a TypedArray/ArrayBuffer. Basically that simply means supporting a TypedArray or ArrayBuffer to URL.createObjectURL()

Compression of JavaScript code

Game developers want to minimize the size of code to deliver it faster to the client. Current mechanisms for that are JS minification, and gzip compression at the level of the HTTP server. Some game developers expressed concerns with that: they wanted better compression algorithms (LZMA) and they wanted something that can be used without knowing how to configure a HTTP server.

-- Idea: can we standardize a flag on XMLHTTPRequests which says ... "and decompress it before giving it to me"? So they could have two versions on the server - a minified and a .gz. They check for support for this feature and, if it's present, set the flag and stick ".gz" on the end of the requested filename. XMLHTTPRequest fetches the data, decompresses it internally and just presents it as if it had not even been compressed. -- Gerv

One topic brought up was the effect of gzip and/or minified JS on file size. There is a great breakdown of jQuery file sizes available here.

Speed of first-run

Game developers are concerned about the time taken to compile large JavaScript codebases. Note that there are two separate compilation stages: bytecode compilation (JS->bytecode), and native compilation (bytecode->native). Currently SpiderMonkey compiles all bytecode immediately when it is loaded by a page, but functions are native-compiled only after they have run a few times (or for a few loop iterations).

We need data to show which compilation(s) are a problem.

If it's bytecode, one solution is to compile the code as it is downloaded (bug on file?). Another is to parse only enough to get early errors, but not actually compile, making the front-end faster.

Native compilation is difficult to make faster: there aren't necessarily any quick fixes.

Ahead-of-time compilation or caching compiled code would address both issues.

Management of large JavaScript codebases

Does JavaScript need to evolve to make large codebases more manageable? Some things that were mentioned:

  • modules (better includes) - ES6 modules should be available around mid-year in Firefox and Chrome. It is planned for Q2 for JS team.
  • namespaces - this is probably taken care of by ES6 modules.
  • add more OO features (classes, inheritance) - not coming soon to the JS language. There are libraries, though, e.g. prototype.js.

HTML5 Gaming Benchmarks

Spaceport.io

Results from spaceport.io performance tests on a Lenovo W520 (hot laptop with NVidia GFX acceleration). We could use runs on desktop machines with typical on-board GFX, and ones with mid-end 3D cards. Also Mac and Linux. Summary: "The short answer is that Firefox appears to be significantly faster at 2D scaling and rotating transforms, but slower at everything else" (than Chrome)

Original report (focused on mobile): Spaceport.io report

Low-level or 3D Graphics

Feedback from Games Work Week

WebGL is a quite thin layer on top of GL or GLES, with an extensive conformance suite and solid lines of communication between implementers.

We currently have hardware acceleration support everywhere, but that is subject to driver blacklisting and as a result, does not serve all of our users. There is also a bug to add support for the Mesa llvmpipe software renderer. Chrome is integrating the proprietary SwiftShader software renderer.

Bug tracking

We use the following Status Whiteboard keywords to track various WebGL-related bugs:

Status Whiteboard Description Bugzilla queries
webgl-conformance Conformance bugs Open bugs only All bugs
webgl-test-needed Missing conformance tests All bugs
webgl-extension Extension implementation bugs Open bugs only All bugs
webgl-next Important items that don't fall in above categories Open bugs only All bugs


Compositing

Though we have similar GL support everywhere, one bottleneck is compositing between our accelerated WebGL contexts and our layer managers. Sharing GLTextures between WebGL and Layers is possible with OGL Layers, available on Mac and Linux. (though heavily blacklisted on Linux) Sharing EGLSurfaces provides the same functionality between ANGLE EGL+GLES and D3D10 Layers. A more complete description of the backend combinations is at Platform/GFX/WebGL/Backends.

Currently, we lack composition acceleration on WinXP, Linux, and Mobile. Mobile and Linux are being investigated as part of OMTC. WinXP can't run D3D10 to take advantage of ANGLE's EGLSurface sharing.

On Windows, we prefer ANGLE's implementation of EGL+GLES instead of using native WGL+GL. Part of the reason for this is blocklisting, but also because we don't yet have a way to interop between native GL and D3D Layers. There is a bug for implementing WGL_NV_DX_interop, which provides this interop. In the future, it might be preferable to default to using WGL+GL+interop, falling back to ANGLE based on blacklisting.

Extensions

WebGL extensions are tracked with webgl-extension in the Status Whiteboard, see 'Bug tracking' above.

Compressed textures (S3TC) has been added as of FF15. Crunch is a good option for transporting and transcoding to S3TC.

We should gather telemetry for extension support to improve our knowledge of extensions' penetration. Alternatively, WebGLStats has some very useful data on this subject.

There should also be a way to expose fast-paths, especially to eliminate costly pixel format swizzling. (For example between BGRA and RGBA)

Tools

There is demand for better tools for understanding what WebGL is doing, so that it's easier to debug for developers.

High-level 2D Graphics

Feedback from Games Work Week

Web technology: Canvas 2D. Well, there is also SVG, but somehow it's less popular for game development. Someone should add an explanation of why.

SVG is a retained mode API which means the developer has less control over whats painted when. SVG also requires using the clumsy DOM APIs which means it's harder to get started.

Canvas 2D is a traditional 2D API, modeled after Apple's CoreGraphics, itself modeled after PostScript. Its main advantages over WebGL are ease of use, availability of important 2D features (including text) and ubiquity: it's supported in all modern browsers and is always available, regardless of possible GPU acceleration. Its downsides are that it's more specific to certain kinds of graphics (2D and some casual 3D), offers less low-level control to the programmer, and is relatively hard for the browser run fast on GPUs.

This is key to understanding the discrepancies in Canvas 2D performance across browsers, and within a single browser, across platforms.

Gecko's implementation of Canvas 2D is currently transitioning from an old model to a new one, with the new model already in use on some platforms.

The old model used Cairo as the internal 2D graphics API on all platforms, then dispatched to platform-specific Cairo back-ends. Some were GPU-accelerated, like the Cairo Direct2D back-end on Windows.

The new model uses a new lightweight API called Azure, that carries minimal overhead and maps perfectly well to Direct2D, which was its first backe-nd. More Azure back-ends are being developed as we transition to using Azure everywhere.

The current and near-future status of Gecko for Canvas 2D is summarized in this table:

Platform Implementation Performance comments Version comments
Windows 7 and Vista Azure / Direct2D Very fast GPU-accelerated Since Firefox 7
Windows XP cairo gdi/software rectangular unscaled blitting should be quite fast, transformed blitting is slower than it could be Since Firefox 3
Mac Azure / (CoreGraphics or Skia ???) much improved blitting performance Since Firefox 12
Linux/X11 cairo X11 performance depends the users drivers. This can range from good to bad. Since Firefox 3
Android cairo software rectangular unscaled blitting should be quite fast, transformed blitting is slower than it could be

As always, notice that GPU acceleration is subject to driver blacklisting.

Audio

Audio is an area that doesn't currently have a universally accepted standard.

Miscellaneous:

FullScreen

This feature allows you to build a web application that runs full screen. This includes any HTML element so you can build full screen games, full interactive video experiences, presentation software or anything else that should dominate the experience.

Pointer Lock (Mouse Lock)

Web technology: Mouse Lock API

The Pointer Lock (Mouse Lock) API is currently under development by David Humphrey and his students at Seneca College. It is also under development in Chrome. This API is currently in W3C Draft Status.

  • This is targeted to land in Firefox 14

Relevant links:

Gamepad

Web technology: GamepadAPI.

The Gamepad API is currently under development in Firefox and Chrome. Firefox builds are available for download here. Chromium builds need to have --enable-gamepad (?) passed on start-up. This API is currently in W3C draft status

  • This is targeted to land in the first quarter 2012

Relevant links:

Keyboard input that ignores keyboard layouts

Owner: Masayuki Nakano

Many games use the WASD keys for direction control. However, depending on the active keyboard layout, their usual (US-layout) key places can be occupied by different letters. In the AZERTY keyboard layout, for example, these keys have the letters ZQSD, and in Dvorak {comma}AOE.

Users of those keyboard layouts shouldn't need to reconfigure their key mapping or change the system's keyboard layout. Instead, the game should be able to get a pressed key's absolute position on the keyboard, or its "scan code". Key events don't provide that information yet.

Standardization:

Competition:

  • unknown

Implementation:

  • This was implemented as keyevent.code in bug 865649 and targets Firefox 32.

Relevant links:

Asset Management

Feedback from Games Work Week

From a game developer: "A robust resource/asset loading/unloading/streaming system - The browser is capable of doing a whole lot on this end, particularly streaming and fetching from remote sources. However, a game developer's problem isn't just obtaining and loading the assets, it's managing them from a memory and "readyness" point of view. This is especially key in mobile spaces where you are severely constrained by memory and the traditional mindset of "load all of a level's assets at start and play" begins to break down."

Gladius is actually working on code for this, and our current theory is that we expect to split it out into a standalone library before too long.

IndexedDB

IndexedDB is a database specification with a bad name. It allows storing large amounts of data client side. It's currently a W3C working draft and in various levels of support in Firefox, Chrome and IE 10. The Firefox 11 implementation is basically feature complete and we expect the IE10 version to be very close to complete once IE10 launches. Chrome is lagging behind currently, but still supports a large part of the specification.

IndexedDB supports storing a large set of data types directly in the database. This includes JSON objects, binary ArrayBuffers, ArrayBufferViews and Blobs/Files.

IndexedDB will write separate Blobs as separate files in the OS file system which means that database operations won't be slowed down even if you store very large Blobs directly in IndexedDB. However we don't know if the OS file system too slow? Especially when scaling up to lots of files in a single OS directory.

How big are these files usually. We've been talking about making the IndexedDB back-end collapse small files into a big OS-file. This would reduce reliance of OS large-directory handling.

DeviceStorage API

We're planning on implementing the the DeviceStorage API which will give pages access to the system's "Pictures" and "Documents" folders. This can be useful for things like saving screenshots etc.

Current state

We support IndexedDB almost completely as of Firefox 11. The main missing part is error handling isn't up to spec yet.

IndexedDB supports storing File and Blob objects directly in the database. When that happens we store the File/Blob as a separate OS-file which means that performance is very good and doesn't affect the performance of the database.

You can also use IndexedDB as just normal data storage rather than read it out yourself from custom file formats. This likely won't have significantly different performance, but can be more convenient.

When you store things in IndexedDB Firefox will prompt the user when the first database for your origin is created. If the user answers yes, the site is granted 50MB of storage. When you hit that limit, the user is prompted again and if he/she answers yes again, the site is granted unlimited storage.

Hopefully we can remove the prefix for IndexedDB in the Firefox 14 or Firefox 15 timeframe.

What we're currently working on

We're currently working on an API to let you do more direct File-IO. I.e. the ability to open a 10MB file, seek to the middle of it, and write 100 bytes of data there. The storage for these files will still be IndexedDB.

We're also going to remove the prompting for any website that gets installed as an OWA App. All OWA apps will likely get unlimited storage space without any prompting. You will be able to tell when you're installed as a OWA app and thus have this storage space available.

What we're planning on doing in fairly short term

We want to adding the ability for websites to write IndexedDB, and other, data "optimistically" without prompting. I.e. the website will be able to indicate that a given chunk of data doesn't need to be permanent and can be deleted if the user runs low on space. That way we won't need to ask the user before using the user's resources, but in turn we will delete the data if there is that need.

We'll likely also add APIs for doing synchronous IO from inside workers. That way emscripten should be able to compile programs that read and write directly to disk.

Longer term we're also planning on

Adding APIs for dealing with .zip files.

Adding APIs for interacting with the HTTP cache. I.e. to load URLs into it, enumerate what's there, and possibly extract cache tokens that ensure that data remains in the cache as long as the token is held alive.

Adding APIs for converting between Typed Arrays and strings, in both directions. We should find someone that can work on this in a more short order

WebSockets

WebSockets is a bi-directional connection-oriented reliable-transport protocol designed to give an option to developers for low-latency communications between browsers and servers. It isn't a raw socket protocol for security reasons, and is initiated on HTTP ports via an HTTP upgrade.

WebSockets is not intended to be a Peer2Peer protocol. It's intended to be the next step after HTTP as far as client-to-server communications goes and allows for long running connections, bi-directional communication with delivery guarantees. If what you are looking for is UDP based communication, or Peer2Peer connections, then WebRTC is the answer there.

You can actually do most of what WebSockets does over normal HTTP with long-hanging gets. However, the overhead of doing that over HTTP means that for small transactions - think key strokes, or a few bytes at a time - can have quite a bit of overhead because of HTTP headers. So WebSockets is really built for applications that require low-latency communications. Think live games or keystroke-based interactive applications.

The low-level WebSockets protocol is RFC 6455 and is fully implemented in Firefox 11. The protocol is TCP-based and can use SSL or operate in the clear. Messages are framed with length and can be either binary or UTF-8. Messages sent by the browser are obfuscated on the wire with a key known to the browser and the server, but is not available to JavaScript. This feature exists to prevent some attacks against transparent proxy servers which may intercept the initial WebSocket connection.

There is a separate WebSockets API maintained at the W3C. (The editor's draft is the best source of information and contains the most recent API. The API is message-based. That is, individual WebSocket frames are sent as messages via an event handler in JavaScript. Messages can be handled as strings, ArrayBuffers or Blobs.

As of Firefox 11 we've unprefixed the WebSockets API. This means that we don't expect the protocol or the API to change. In other words, we're basically "done" with our WebSockets support.

The two outstanding item that I know about are:

  • If you download a large binary message as a Blob, we don't stream the data to disk, but instead back the Blob using an in-memory buffer. This is transparent to the page, but causes us to use more memory. Tracking bug is bug 704447.
  • WebSockets aren't supported in Workers bug 504553

Multithreaded programming

We currently have WebWorkers as solution for multithreaded programming. WebWorkers use a shared-nothing message passing model. I.e. there are no mutexes, monitors etc.

Is this going to work for game developers?

We also haven't done a lot of performance testing, should get feedback from developers if there are bottlenecks. For example is the postMessage code fast enough?

We're also missing a large number of APIs. The following APIs are known to be missing:

  • IndexedDB bug 701634
  • WebSockets bug 504553
  • Asynchronous FileReader (is this important given that we have FileReaderSync?)
  • Transferrable ArrayBuffers between threads bug 735474
  • Shared Workers bug 643325

We also know that our XMLHttpRequest performance is pretty bad, especially for large resources.

Device orientation control

Owner: Mounir Lamouri

Ability to block, define and get informed when the device orientation changes. Basic use case is for a game that has to be played in landscape mode. The game should lock the orientation to landscape mode even if the device is currently in portrait mode.

Game-oriented documentation

TODO: fill this section

  • Tutorials
  • Example games
  • Articles in Gamedev.net, Gamasutra, etc.

Converting C/C++ games and libraries to Javascript

  • Emscripten is an open source compiler from C/C++ to JavaScript, through LLVM.
    • Emscripten functions as a drop-in replacement for gcc
    • Can compile individual libraries and provide friendly JavaScript wrappers for their C or C++ APIs, see ammo.js, a port of Bullet, and box2d.js, a port of Box2D.
    • Supports the main parts of APIs like libc, libc++, SDL, POSIX
    • Some support for OpenGL ES 2.0 exists (see demo), ehsan is working on general OpenGL support.
    • Mozilla is working towards getting the relevant parts of Emscripten into upstream LLVM.
  • Mandreel is a commercial product for Windows that similarly compiles C/C++ to JavaScript through LLVM. It also supports Objective-C and OpenGL and is more focused on compiling entire games as opposed to libraries.

High Resolution Timers

Feedback from Games Work Week

[Update] window.performance.now() is now available in Firefox 15!

See [1] for the spec.

Mobile

Feedback from Games Work Week

Animation timing/syncing

VSync

We need to synchronize compositing with VSync, and do what we can to ensure that games get their requestAnimationFrame callbacks at the right time so they can queue draw operations ahead of the next VSync.

Reliable animation timing

In bug 731974 we discuss that requestAnimationFrame sometimes calls back at random times, sometimes every millisecond! Seems to be a general problem with our timers.

bug 640796 covers part of the problem on Windows. Once that's fixed, we can revisit bug 590422 and hopefully fix it. Alternately, we need a fix for bug 625258 and hope that works.

Anyway, this test case allows one to check if requestAnimationFrame is behaving vaguely correctly (you shouldn't see red "too short" warnings).

Sandboxing

Games often want a restricted execution environment for some of their code. Modding, customizations, and player-implemented behaviors are example use cases. Sandboxing may be to prevent cheating, to control execution overhead, or to limit the modifiable parts of the game.

Current facilities include sandboxes, same-origin restrictions, workers, and compartments and proxies. Not all of these are accessible to content, and it is unclear how much of this requirement is handled.

Miscellaneous

Feedback from Games Work Week about missing APIs.

Regarding developer tools, see these etherpads from the Games work week:

Retrieved from "https://wiki.mozilla.org/index.php?title=Platform/AreWeFunYet&oldid=984699"