Classification of user actions/flows that will be perceived as task with discrete performance expectations and settings/configurations that affect the raw performance of those tasks.

Pageload

Metrics

Term	Metadata	Definition	Relevant Links
		One or two sentences	Full definition Related tests/coverage Results User research
SpeedIndex	Visual	(SI) is a page load performance metric that shows you how quickly the contents of a page are visibly populated. It is the average time at which visible parts of the page are displayed. Expressed in milliseconds, and dependent on the size of the viewport, the lower the score, the better.	MDN Documentation
Contentful SpeedIndex	Visual	This metric uses edge detection to calculate the amount of “content” that is visible on each frame. It was designed to have a good metric to measure the amount of text that is visible and ignore the impact of popup or login screens at the end of a page load.	sitespeed.io documentation
Perceptual SpeedIndex	Visual	A metric that measures the above-the-fold visual performance of a webpage, while accounting for the layout stability.	* Perceptual Speed Index (PSI) for measuring above-fold visual performance of webpages Visual metrics github repo
Visual Completeness	Visual	The point in time when a web browser has finished rendering viewable content in the viewport, but not the entire page.
First Paint event	Standard	The time between navigation and when the browser renders the first pixels to the screen, rendering anything that is visually different from what was on the screen prior to navigation.	MDN documentation
First Contentful Paint	Standard	How long it takes the browser to render the first bit of content from the DOM providing the first feedback to the user the page is actually loading.	MDN documentation
DOMContentLoaded event	Standard	A DOM event that is fired once the underlying document has finished being parsed and loaded, but before any of its additional resources (stylesheets, images, subframes) have finished loading.	MDN documentation
Onload event	Standard	A DOM event that is fired once the underlying document has finished loading, including all of its stylesheets, images and subframes. This does not include additional resources that start to load due to script.	MDN Documentation
Time to Interactive	Proposed standard	(TTI) is a non-standardized web performance 'progress' metric defined as the point in time when the last Long Task finished and was followed by 5 seconds of network and main thread inactivity. The page has displayed useful content Event handlers are registered for most visible page elements When a user interacts with the page, the page consistently responds within 50ms - the user does not experience jank.	MDN Documentation Proposed web standard
Cold page load		browser started and stabilized/idle, same profile dir as previous, site not loaded or cache cleared
Warm page load		browser started and stabilized/idle, same profile dir as previous, site already loaded once, any click-throughs clicked
Reload		To re-request the rendering of a document. This may mean re-retrieving the entire document and any of its resources off of the network, or might mean reloading the resources out of a local cache.
Backwards/Forwards navigation
Level 1 cookie blocking	Tracking Protection
Level 2 cookie blocking	Tracking Protection
Level 1 content blocking	Tracking Protection
Level 2 content blocking	Tracking Protection

Responsiveness (mobile vs Desktop)

Scroll

Term	Metadata	Definition
Checkerboarding	Metrics	Checkerboarding occurs when Gecko is unable to paint the viewable portion of a webpage while we're scrolling. Instead, we just paint a color instead of the content.
Velocity	Metrics
Touch	Variations
Keyboard	Variations
Mouse/UI	Variations

Browser

Term	Metadata	Desktop Definition	Mobile Definition
Pageload Jank
Time to first blank	Open	The time from process start to display the very first painted frame. On Windows, this is currently a white rectangle, which we consider "first blank"	N/A
Time to first chrome paint	Open	The time from process start to displaying the first frame of the browser UI (not including the content area)	N/A
Time to settled about:home	Open	The time from process start to displaying a fully loaded about:home	N/A
Fresh profile	Open	new profile dir, user.js, first time network requests	Application process starts for the very first time. This includes onboarding, but no migration.
Cold start	Open	The browser is started when it hasn't already been started in this operating system session, so the disk cache is empty. For example, the first time that Firefox starts up after the operating system boots is a cold start.	Application process is not running. Not a first run, and no onboarding. The application is started either via the homescreen or applink. In GPlay terminology, there's no stored state (i.e. no `onSaveInstanceState` bundles) When the Fenix team measures cold start times, the generally measure from process start to visual completeness of the homescreen.
Warm start	Open	The browser starts when a previous instance of the browser was already started and shutdown within the same operating system session.	Application process is running, but the HomeActivity is not. The application is started either via the homescreen or applink. In GPlay terminology, "warm start" includes many situations and is not well defined. For example, it also includes if the app process is killed but there are `onSaveInstance` state bundles (which seems closer to Cold start) This is not currently measured by the Fenix team.
Hot start	Open	Depending on user settings this will either open new window or new tab	Application process is running, HomeActivity is also running. The application is started either via the homescreen or applink, which brings the HomeActivity to the foreground. In GPlay terminology, `onTrimMemory` may or may not have been called) This is not currently measured by the Fenix team.
Delete history on close	Close	A non-default configuration that has Firefox forget all user browsing history on shut down.	N/A
From crash	Session Restore		N/A
User initiated	Session Restore		N/A
App link	Variations	N/A
With WebExtension->Theme	Variations	At least one or more WebExtension themes is installed, and one is enabled.
With WebExtension->Addon	Variations	At least one or more WebExtension addon is installed and enabled.
Secure Proxy	Variations	The "Firefox Private Network" WebExtension.
Time to interface change complete	UI	Click on a "share" button and the delay to when the display is shown.
Open	Window	To open a new browser window.	N/A
Close	Window	To close a new browser window.	N/A
Switch	Window	To switch focus from one browser window to another.	N/A
Resize	Window	To resize a browser window.	N/A
Open	Tab	To open a new browser tab.
Close	Tab	To close a new browser tab.
Switch	Tab	To switch from one browser tab to another within the same window.
Open	Menu	To open one of Firefox's built in toolbar menus or context menu.
Close	Menu	To close one of Firefox's built in toolbar menus or context menu.
Touch	Input	Input from the user via a touchscreen.
Keyboard	Input	Input from the user via the keyboard.
Mouse	Input	Input from the user via a mouse.

Resource

Term	Metadata	Definition	Relevant Links
Memory
Power
CPU
GPU
Disk
Network
App idle in foreground	Network variations
App idle in background	Network variations
Audio	Media
Dropped Frame	Video
Resolution Change	Media
Stall length	Media
AV sync	Media
Number of stalls	Media
AV sync	Media
Time to payback state change	Media
Real Time Communication (RTC)	Media
Frame per second (FPS)	Virtual Reality (VR)
Visual Distortion	Virtual Reality (VR)
Z-fighting	Virtual Reality (VR)
Judder	Virtual Reality (VR)

Components

Term	Metadata	Relevant Links
Graphics
Layout
Parsing	Javascript	JavaScript
Compilation	Javascript	JavaScript
Network		Networking
DOM		DOM
Storage		Storage
Media		Media

Mozilla specific terms

Backfill

Trigger skipped jobs (Treeherder terminology). Also refers to filling in missing data points from Perfherder's Graphs view.

Data point

Performance metric value visible in Perfherder's Graph view.

DOM Content Flushed (dcf)

This is a Mozilla proprietary extension and not part of the performance/navigation timing specification. It marks the completion of the first presentation flush after DOMContentLoaded. Internal Firefox metric activated by setting the preference dom.performance.time_to_dom_content_flushed.enabled to true.

Downstream

Alert triggered by a commit that originally happened on another repo and was merged into the current one after.

First Non-Blank Paint (fnbpaint)

The timestamp of a shipped frame that contains the first paint to draw more than the document background. Has low value to a user, as this paint may be completely meaningless or useless. The first non-blank paint is often represented as "Start Render" in performance products.

Gecko profile

The Firefox Profiler is a profiler that is built into Firefox and is available at profiler.firefox.com. It has tighter integration with Firefox than external profilers, and has more of a platform focus that the DevTools performance panel. It can be used a variety of situations where external profilers are not available, and can provide more information and insight into what the browser is doing.

mitmproxy

Mitmproxy is a third-party open-source tool (found here in github) that allows the recording of a live web page and saving it as a playback archive. Using Mitmproxy's 'mitmdump' tool, the recorded page can be played back through a local proxy browser proxy; then browsing to the URL loads the page from the mitmproxy recording instead of the live site.

The Raptor performance testing framework uses Mitmproxy for Raptor page-load performance tests. Mitmproxy is available as a python mozbase/mozproxy module for import in any python project.

Noise (regarding performance tests)

Generally a test reports values that are in a range instead of a consistent value. The larger the range of 'normal' results, the more noise we have. Some tests will post results in a small range, and when we get a data point significantly outside the range, it is easy to identify. The problem is that many tests have a large range of expected results. It makes it hard to determine what a regression is when we might have a range += 4% from the median and we have a 3% regression. It is obvious in the graph over time, but hard to tell until you have many future data points.

Perf Sheriff

Using perfherder, a perf-sheriff will perform these tasks:

Look at the graph and determine the original branch, date, revision where the alert occurred
Look at Treeherder and determine if we have all the data.
Retrigger jobs if needed (more noise, more retriggers)
Once you have more data, look at the data in compare view to see if other tests/platforms have changed
Add all related alerts you see to the summary with the reassign button

Perfherder

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).

Raptor

Raptor is a performance testing framework used for running browser page-load and browser benchmark tests. Raptor can run performance tests on Firefox desktop, Firefox android (Fennec, Geckoview example app, and the android components Reference Browser), and Google Chrome/Chromium. For more information please refer to the Raptor wiki page.

RelMan glossary

These terms are of use, especially when doing Performance sheriffing.

Retrigger

Any run of the same job on the same commit for the same repo triggered after the first one. Retriggers usually happen when the available datapoint in the graph are not enough for the sheriff to determine the culprit.

SETA

SETA finds the minimum set of jobs to run in order to catch all failures that our automation has found in the recent past on Firefox development repositories. SETA creates job priorities for all jobs found in the runnable-jobs API for that repository.

Talos

Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.

Treeherder

Treeherder is a reporting dashboard for Mozilla checkins. It allows users to see the results of automatic builds and their respective tests. The Treeherder service manages the etl layer for data ingestion, web services, and the data model behind Treeherder.

Time To First Interactive (ttfi)

This is a newer metric and reports when the page is first expected to be usable and will respond to input quickly (with the possibility of slow responses as more content loads).

Start looking for First Interactive at first contentful paint
Look for the first interactive window (with no regard to in-flight requests)
First Interactive is the start of the interactive window from step 2, first contentful paint or DOM Content Loaded, whichever is later

R/b

Short name for retrigger/backfill.

WebPageTest

Accurately measure, track, and report on key Firefox-performance metrics, over time.

Fenix

The Fenix team, for better or worse, hasn't been coordinating with desktop strongly to make all terms consistent. Some issues with trying to do so:

The fenix team is unfamiliar with desktop terminology and we don't have working patterns across desktop and mobile performance
We're actively learning more and more about measuring performance on fenix so definitions may adapt as we learn more

See Performance/Fenix/Glossary for Fenix-specific terminology.

Performance/Glossary

Contents