Accessibility/Virtual buffer smash

This project aims to support NVDA and other assistive technologies on Windows to get rid of virtual buffers for their support of Firefox and other Mozilla products.

Background

The concept of virtual buffers was originally invented in the late 1990s to overcome shortcomings in accessibility APIs and slow cross-process COM calls. After a page loads, it is sucked into the screen reader in whole, analyzed, and then presented to the user in a linear fashion, much like a text document in Microsoft Word and other word processing environments. The text is augmented with semantic information such as links, graphics, headings, tables, lists, form fields and others. Whenever changes happen dynamically on a web page, such as is common since the mid 2000s, browsers fire certain events the screen reader listens to, such as EVENT_REORDER, and updates its "copy" of the page with the changes that happened. It is the browser's responsibility to fire the appropriate events for certain happenings, and it is the screen reader's responsibility to keep its view of the world in sync.

Because this screen reader specific copy of the document does not really exist outside the screen reader itself, it is often referred to as a virtual document, and the cursor to browse it, is subsequently called a virtual cursor. Other screen readers call it browse mode. The user navigates this document via arrow keys, the tab key, or so-called screen reader specific quick navigation keys to jump quickly to certain types of elements within the document. Such elements can include headings, even by specific levels 1 through 6, form fields, lists and list items, graphics, annotations and objects (such as Flash objects). The keyboard is completely taken by the screen reader, and the browser does not see any of the goings-on besides occasional instructions to set focus to elements.

Whenever a user has to interact with the page, they have to tell the screen reader to switch out of this linear document into another mode which is called focus mode, or forms mode, depending on the screen reader in use. This is usually done by focusing an element where direct interaction is needed, such as text inputs or dropdown comboboxes, and pressing Enter. Other screen readers allow an automatic switch to this focus mode via tabbing, or even switch to the mode automatically whenever the virtual cursor lands on such an element. Simpler widgets such as links, buttons, or checkboxes, can usually be interacted with from within the virtual mode directly using Enter or Space to activate or toggle.

The problem

While these modes of operation are very efficient when surfing the web, they can also pose problems, especially in more modern web applications such as Gmail. It is not always clear which mode is appropriate for what, and users have to manually switch modes to get the best interaction. Sometimes, the WAI-ARIA role Application can help, but if used wrongly, this can also lead to a lot of problems.

Besides, other operating systems such as Linux or OS X never made use of the virtual buffer concept. While on Linux with the Orca screen reader, navigation is done in a similar way, the interaction actually takes place within the browser. On OS X, VoiceOver treats web pages just like any other application, not giving the user a document-like experience at all. Consequently, quick navigation commands there can be used even in applications that have nothing to do with the web at all, to jump to headings or tables.

The goal

Inspired by the fact that Microsoft Edge, the successor to Internet Explorer, does no longer allow in-process access to its accessibility representation of a web page, going fully UI Automation, screen readers are forced to come up with ways to interact with the browser and give the user a good experience without the use of a virtual buffer. This can lead to a lot of good things:

• Getting rid of two copies that need to be kept in sync constantly.
• Easier transition between browsing and interacting with web applications.
• A more consistent user experience.
• An easier time for inexperienced developers who test their web pages for accessibility.

To that end, Mozilla should use its experience it has gained over the years with the Linux accessibility API and put that to good use for the Windows screen readers to best support a transition to a virtual-buffer-less interaction with Firefox and other Mozilla products.

Bugs that need fixing

Text

One of the most important aspects is getting our text interfaces correct for all elements, be it for browsing or editing. Our meta bug 368895 collects all open and closed bugs related to our text interfaces (IAccessibleText, IAccessibleHypertext).

TBD

TBD