This is currently a WIP page detailing everything about driver blocklisting testing in December and perhaps onward.

Background

GPUs are imperfect. Like any other software, their firmware ships with bugs, and perhaps the design of the GPUs themselves are flawed in some way which cannot be corrected by their vendor except for creating a new design. These flaws, unfortunately, impact our software. Some of our code uses features in these GPUs which cannot work, even if on other GPUs they work fine. We can't fix these problems, so what is our natural reaction? We deal with them by avoiding using them. This is where driver blocklisting kicks in.

What is Driver Blocklisting?

Driver blocklisting allows us to blocklist specific devices, drivers, etc. from using some GPU features. When a configuration is blocklisted and a user is using the configuration blocked, it is never put into use. A configuration refers to the set of unique settings a user has, such as what video card they have installed on their computer, their OS, and their video card driver version.

The driver blocklist system is comprised of 3 components:

Downloaded Blocklist

Whenever we run into problems where we find out after a release that something is broken on a specific set of hardware, we can push updates to a downloadable blocklist which is downloaded every time Firefox starts or every day (whichever happens sooner). Structurally, this is stored in an xml file. This xml file gets parsed into the exact same structure as the Static Blocklist item below.

Static Blocklist

This is a list of items compiled directly into Firefox and which cannot be modified except for releasing a new build. It contains things we are "sure of" and that we generally don't want to reverse, or that are acceptable to reverse sometime later.

Special Cases

Special cases are similar to the static blocklist except that they don't follow any structure and are basically just thrown into the code piecemeal because they handle special situations which are difficult to generalize. For example, blocking a range of drivers is significantly more difficult right now than blocking only a single version and all drivers earlier, so we special case whenever we need to do this. Special cases are more difficult to understand and to maintain, but are sometimes necessary.

What features can be blocked?

There are several features which can operate incorrectly. For example, some configurations may allow OpenGL layers to work perfectly fine, but for WebGL to be completely mangled. For this reason, we need to be able to block specific features, several of them, or all of them. The new blocklisting implementation allows us to do this. A full list of all features can be found at Blocklisting/Testing#Quickly_Checking_Features.

What can we blocklist?

We can blocklist on a series of criteria, including all of them or only one of them. Here are some examples:

Operating system
Video card vendor
Video card (the card itself), with the ability to add several devices
Video driver version (specific version, earlier than a version, newer than a version, etc.)
Feature (i.e. if we want to we can completely disable an entire feature on all configurations)
Special cases (cannot be added into a blocklist xml file, must be compiled in)

Old Implementation

As was alluded to previously, we have had a Windows blocklist for some time now. This old implementation was tested using: https://docs.google.com/a/sherk.me/spreadsheet/ccc?key=0Aj-5ZZLSQ6mrdHZOSTl4ZE1tNkFfeThHUlNPckRrRHc&hl=en_US#gid=1

This document is a list of a bunch of configurations that we installed Fx on with this old blocklist implementation and tested features with. We used IBeta to provide these machines and do most of the manual testing. We are going to re-use some of these configurations.

Compatibility Testing

We often refer to compatibility testing in meetings, which is a completely different issue altogether. Compatibility testing is a new effort which is set of tests we run on Firefox as a whole going forward with every release. While we are engaging IBeta for blocklisting, we're also going to do compatibility testing. We talk about these issues together because of this.

What has changed in December?

Previously, nothing other than special cases worked on platforms other than Windows. As a result, we worked on adding support for a generalized blocklist to other platforms. The focus of this effort is to get the downloaded and static blocklists working on Android, but as a side effect of these changes, other platforms may have their blocklists broken due to refactoring. In addition, Mac blocklist support has recently been added, as well as Linux blocklist support. These implementations should be tested for the small set of entries that they block. Here's the tracking bug which contains a list of all changes: https://bugzilla.mozilla.org/show_bug.cgi?id=706702

Getting Started

To get started with blocklisting testing, see the Blocklisting/Testing. This page details all the resources needed for testing, but not specifically what we're testing.

What is currently blocked?

Blocklisting/Blocked_Graphics_Drivers contains a list of everything currently blocked.

What we're testing

Note that we're not testing all of the configurations from the above article to make sure that they're still blocked. The reason for this is that the refactoring is likely to either break everything or nothing. There may be some edge cases that we handle incorrectly, but it's fairly likely that we'll catch most or all of them by just doing a small subset of the machines and configurations we tested on during the last run.

What has changed the most

Note that the below is not a complete list of what we're testing; it's just a list of things that were impacted disproportionately by the refactor.

(Mac) For MSAA, we block all ATI cards except for AMD Radeon HD 6490M (device id 0x6760) and ATI Radeon HD 4670 (device id 0x9488). Any Mac with a Radeon HD 6490M or Radeon HD 4670.
(Mac) For layers acceleration, we also block all old graphics adapters that do not fully support OpenGL 2.1 in hardware (use slow software fallbacks), or that can't render to non-power-of-two texture-backed framebuffers. That includes the following generations of GPUs: ATI Radeon X1000 and older, NVIDIA Geforce FX and older, and Intel GMA 950 and older. Any Mac with a Radeon X1000.
The Android downloadable blocklist. I have written up instructions on it here: https://wiki.mozilla.org/Blocklisting/Testing#Testing_on_Android This also includes a link to a sample XML file which can be used to block a Nexus S device. I can write up more for other devices, but the only one I have available at the moment to test with is a Nexus S. While we want the OS X and Linux downloadable blocklists to work too, Android is the priority.

All Cases

What's in the configuration matrix mostly looks good. There are also some Mac and Linux configurations which should cover most cases that we want except for the above. It's more important that we get a wide diversity of video cards than anything else. So to the extent that we want to limit the amount of configurations we're using compared to last time, we should choose combinations such as (referring to the "Merge Plus Configurations" document) #1, #2, #4 and skip near-duplicates such as #4 and #5. There are 47 entries here in total and I think we can get away with doing ~10-15 of them.

Doug: "Here are my picks, although I'm certainly not the expert on this:

1. 1, #4, #8, #9, #12, #19, #28, #34, #35, #42, #45, #46, #47, +Mac with Radeon X1000, +Mac with Radeon HD 6490M or 4670, +Sandy Bridge as per Joe Drew's request.

I avoided high-power GPUs which, from my experience, usually do everything or almost everything properly. My picks prefer more exotic or low-end GPUs that our common non-power-users are less likely to have, or OS's other than Windows.

I am also concerned that we're only testing the latest drivers and would prefer it if we could install old ones as well. I don't expect this to be an issue, but to the extent that we're going to put together these machines, I don't expect it to take much more in the way of resources to install an additional driver."

Note that this list has yet to be finalized.

What has yet to be added

There still are yet to be any reftests which account for any of the new functionality. Doug will be working on adding several. Aside from that, everything else is ready.

Test Plan

(QA people please fill this in with more detail as I don't know what you're planning on right now)

In general, IBeta will be doing manual testing on different real hardware configurations, while Mozilla will be implementing automated tests and perhaps doing some manual testing.

Testing can be fairly generalized for each platform. However, there are some specifics depending on the platform that we're working with.

General Testing

There are several features that we can blocklist, as well as several configurations. While the configurations have been discussed above, what is actually blocked has not been discussed. The Blocklisting/Testing#Quickly_Checking_Features page has a list of all features that can be blocked, as well as how to determine whether they're enabled or disabled.

What we will generally be doing is building a configuration that we know should have, for example, WebGL's OpenGL variant disabled. We will then run Firefox with the changes discussed on this page and check that it is indeed disabled. If it is enabled, there is a problem with blocklisting and thus a bug.

To find your current device vendor and device id, look at about:support. This should be a first step if you think that you have found a bug.

Writing XML Files

You can create an XML file to be parsed and point the browser to it fairly easily. Instructions on how to point it at a different URL than the default Mozilla servers can be found at Blocklisting/Testing#Testing_Staged_Blocklist.

If you clone http://hg.mozilla.org/mozilla-central and look in browser/app/blocklist.xml, there are several examples of blocklist entries there already. Here are a list of constants that you can use to create new entries:

<os>

"WINNT 5.0", Windows 2000
"WINNT 5.1", Windows XP
"WINNT 5.2", Windows 2003
"WINNT 6.0", Windows Vista
"WINNT 6.1", Windows 7
"Linux", all Linux except Maemo(?) and Android
"Darwin 9", Mac OS 10.5
"Darwin 10", Mac OS 10.6
"Darwin 11", Mac OS 10.7
"Android", all Android
"All", all OS's, may not work and this is not a priority

<vendor>

"0x8086", Intel
"0x10de", NVIDIA
"0x1022", AMD
"0x1002", ATI
"herring", Nexus S's GPU I think (sorry, I don't know anything else about it)
Do not fill this in if you want all vendors.
You can add other entries, but don't expect them to work if you aren't using any of the above vendors.
Can copy this out of your about:support.

<devices><device></device></devices>

"Nexus S", Nexus S device
"0x6741", ATI Radeon HD 6750M
You can add other entries, but don't expect them to work if you're using either of the above.
Can copy this out of your about:support.
This can be a list of devices, as evidenced by the XML structure for it.

<feature>

"DIRECT2D"
"DIRECT3D_9_LAYERS"
"DIRECT3D_10_LAYERS"
"DIRECT3D_10_1_LAYERS"
"OPENGL_LAYERS"
"WEBGL_OPENGL"
"WEBGL_ANGLE"
"WEBGL_MSAA"
Can ONLY be one of these. See Blocklisting/Testing#Quickly_Checking_Features for a somewhat complete description of each.
Not entering this into the XML will block all features.

<featureStatus>

"NO_INFO", can be used to whitelist a device, although this won't work if it's already in the static blocklist or a special case
"BLOCKED_DRIVER_VERSION", only use this if you set a driver version and comparison operator (explained below)
"BLOCKED_DEVICE", blocked because of a device, always safe to use (i.e. you don't have to set anything else)
"DISCOURAGED", not sure, I think this is specific to each feature
"BLOCKED_OS_VERSION", blocked because of the OS version, always safe to use
Does not have to be one of these, but will default to "NO_INFO" if it isn't.

<driverVersion>

See browser/app/blocklist.xml for examples for Windows
"9" for Android 2.3
"8" for probably Android 2.2 (didn't try this, just assuming)
This only applies to Windows and Android
You must also set the driverVersionComparator field if you use this field.

<driverVersionComparator>

"LESS_THAN"
"LESS_THAN_OR_EQUAL"
"GREATER_THAN"
"GREATER_THAN_OR_EQUAL"
"EQUAL"
"NOT_EQUAL"
"BETWEEN_INCLUSIVE", I don't think this works on Android
"BETWEEN_EXCLUSIVE", I don't think this works on Android
"BETWEEN_INCLUSIVE_START", I don't think this works on Android

Testing on Windows

Windows has the most complex and configurable setup. We can blocklist by the following criteria:

OS, including 2000, XP, Vista, and 7.
Primary video card vendor
Primary video card device
Secondary video card vendor (for SLI, XFire or Optimus)
Secondary video card device
Driver version, including specific version numbers (as well as modifier such as "if your driver version is earlier than this" or "if your driver version is this specific version")
Feature

Aside from that, Windows blocklisting is basically the baseline implementation. There are several special cases listed in the Blocklisting/Blocked_Graphics_Drivers#On_Windows document.

Testing on Linux

Linux has no added entries to the static or downloaded blocklist. Thus, all we're doing in this case is creating our own xml files and pointing the browser to them to see if artificially created tests work as expected. In addition, there are some special cases listed in the Blocklisting/Blocked_Graphics_Drivers#On_X11 document.

Note: Linux is missing driver version functionality, as well as proper OS detection. Effectively, all we can blocklist by on Linux are:

OS, only "Linux"
Video card vendor
Video card device
Feature

Testing on Mac

Mac has several entries in the static blocklist, and none in the downloaded blocklist. These are detailed in the Blocklisting/Blocked_Graphics_Drivers#On_Mac document. Macs can blocklist by the following criteria:

OS, including 10.5, 10.6, 10.7
Video card vendor
Video card device
Feature

Note that we can't blocklist by driver version here either. The reason for this being that Apple rarely releases major GPU driver updates, except for between full major OS versions like 10.5->10.6. In this way, the OS version is almost acting like the driver version.

Testing on Android

Currently, Android only has 1 special case, which is detailed in the Blocklisting/Blocked_Graphics_Drivers#On_Android document. Android can blocklist on the following criteria:

OS, just "Android"
Video card vendor ("herring" for Nexus S, unlike other implementations which use PCI-style id's which are hex values like 0x1234)
Video card device ("Nexus S" for Nexus S, pretty much the device itself)
Driver version (this is peculiar because it's actually the SDK version. for example, if you set this to "must be 9 or later", then this feature will be blocked on OS's earlier than Android 2.2)
Feature

Blocklisting/QA-December

Contents