Firefox/Content Performance Program/Observations

Windows Desktop

Reference system

• HP Pavilion 14t laptop (model # K9E07AV)
• 15" touchscreen, native resolution 1366x768
• CPU: i3-5010u
• GPU: (integrated): HD5500

Experiments

Scroll perf - e10s configs and Windows 8 / 10 (Firefox)

• Performed by: aklotz, avih
• Compare Firefox scrolling performance between Windows 8.1 / 10 and e10s modes, using different power modes, test pages, and tabs
• Tested using the scroll-test bookmarklet from bug 894128 (this)
• On each config, run the bookmarklet few times and register the result (average and stddev) of a single run which seems typical
• Base configuration:
  • System: HP reference laptop, Windows 8.1 and Windows 10 - always connected to power.
  • Firefox Nightly 42 2015-08-02 with APZ disabled
  • Bookmarklet config: 7000 (ms scroll), 10 (pixels/step), 0 (don't ignore first frames)
• Tested configurations: all the combinations of the following:
  • Page: facebook / twitter / Yahoo search results
  • OS: Windows 8.1 / Windows 10
  • Power mode: balanced / power saver
  • Active background tab (gmail while it reloads): Yes / No
• Observations:
  • It's really hard to take/get consistent measurements while gmail loads in the bg, due to network latency, gmail load timing, etc.
  • Facebook scrolls worse than yahoo/twitter on most configs (except single tab + balanced where the system is fast enough and all pages reach ~60fps)
  • Bug 1213425 Power saver shows consistently worse results than balanced
  • No major differences between win8 and 10 across the board
  • No major differences between non-e10s and e10s
  • Single tab - no major differences between non-e10s/e10s/e10s-multi(3)
  • Bug 1213413 Gmail loading in bg - affects most* configurations badly
    • Except e10s-multi(3) which improves performance on this case dramatically

Scroll perf - GFX and e10s configurations (Firefox)

• Performed by: aklotz, avih
• Reproduced partially: vladan
• Compare scrolling performance between different Firefox gfx configurations
• System: HP reference laptop, Windows 8.1
• Firefox nightly 42 2015-07-27, with APZ disabled
• Tested using the scroll-test bookmarklet from bug 894128 (this)
  • Bookmarklet config: 3000 (ms scroll), 10 (pixels/step), 1 (ignore the first frame)
  • On each config, run the bookmarklet few times and register the result (average and stddev) of a single run which seems typical
• Tested configurations: all the combinations of the following:
  • Page: facebook / Yahoo search results / google image search results
  • e10s: Yes / No
  • Power mode: balanced / power saver
  • Firefox gfx configs:
    • D3D11, D2D1_1 (Default)
    • D3D11, D2D1_1, OS scaling 125%
    • D3D11, D2D1_1, Firefox scaling 125%
    • D3D11, D2D1_1, on an external 1080p display
    • D3D11, D2D1_0
    • D3D11, no-D2D
    • HW accel off, no-D2D
    • D3D9, no-D2D
    • D3D11-warp, no-D2D
• Observations:
  • The google image search results scrolls best across configurations (and typically perfectly), yahoo results scroll somewhat worse, and facebook scrolls the worst of these pages.
    • For this reason, it's probably better to use facebook for scroll benchmarks.
  • Bug 1213440 D2D1_1 scrolls worse than other D2D configs (including no-D2D) on different pages, different GFX configs, and e10s yes/no.
  • facebook scrolls quite badly with D3D11-warp (bug 1213432) and D3D9 (bug 1213429), especially with e10s.
  • Except for D2D1_1, D3D11-warp and D3D9, most other configs perform similarly.
  • With e10s, the performance differences between gfx configs seem more extreme than without e10s.

Navigation speed from search results (between browsers)

• performed by: vladan, avih
• Test how quickly different browsers navigate links from a search results page.
• System: HP reference laptop, Firefox Nightly 44.
• Procedure:
  • Navigate to google.com, type in the search query, press enter, wait for results.
  • Click the first non-ad link
  • Observe navigation speed
• Observations:
  • Searching "facebook":
    • Bug 1213434 With a clean browser cache, Chrome navigates almost instantly - quite faster than Firefox and IE.
    • With warm cache, all browsers navigate at similar speeds.
  • Searching "cnn"
    • Clean cache: Chrome faster than Firefox/IE, but it's not as instant as with "facebook".
    • Warm cache: All browsers navigate at similar speeds.
  • Searching "mozilla" or "edge":
    • Both Clean/warm cache: all browsers are similar.
  • Note that if a user routinely uses google search to access facebook (or cnn etc), it's highly likely that the main page will already be cached by the browser (or at least many components which this page uses), therefore the "facebook instant effect" which chrome has but Firefox doesn't is not likely to be a meaningful differentiator in practice.

Refresh rates correctness - two monitors configurations (between browsers)

• performed by: avih
• Test how well browsers use the correct refresh rate in a two monitors configuration.
• tested using: http://www.vsynctester.com/index.html
• Procedure - with a single browser window, check all combinations of the following variables:
  • Browser: Firefox release (40) / nightly (44) / Chrome / IE / Edge (win10 only)
  • OS: Windows 8.1 / Windows 10
  • Windows "main" display: laptop's built-in / external monitor
  • The browser window is at: laptop's built-in / external monitor
  • Laptop's built in display refresh rate: lower/identical/higher than that of the external monitor
• Observe both the reported animation rate and the smoothness on screen - on all cases the reported value matched the visual observation.
• Observations:
  • When both monitors have the same rate - all browsers behave optimally.
  • Firefox is the best, with only a single suboptimal case (and not worse than other browsers in this case):
    • When the Windows "main" display has lower rate than the extended-to display (regardless if the main is the built-in/external, or windows 8.1/10), then the animation rate on the extended-to display is the refresh rate of the "main" display (therefore lower than it should be). I suspect this to be an inherent Windows issue (need quote).
  • Chrome comes second:
    • Same issue as Firefox, plus few other cases where it incorrectly uses the rate of the "other" monitor.
  • IE/Edge are last:
    • Same issues as Firefox and Chrome, plus when the main display is the laptop, and it has a lower rate than the external display, on both win8/10 -> it uses weird rAF rate (67hz, 64hz) which doesn’t match any monitor.

Memory usage comparison (Firefox in few e10s configs and others)

• Performed by: vladan, avih
• Compare browsers memory usage with different number of tabs
• System: HP reference laptop, Windows 10
• Browsers: Firefox Nightly 44 (with different e10s modes), Chrome, IE11, Edge
  • Firefox configs: no-e10s, e10s (normal two-processes e10s), e10s-multi (with processes limited to 20), e10s-8 (limited to 8 processes)
• Procedure:
  • Cases:
    • "No URL tabs" (Open the browser with only the "new tab" tab)
    • With 4 tabs (yahoo, cnn, youtube, facebook)
    • With 15 tabs (above plus: twitter, wikipedia, amazon, microsoft, mozilla, ebay, yandex, reddit, imgur, imdb, apple)
  • For each case:
    • Wait for a minute or so until the memory usage settles
    • Observe the number of processes the browser uses
    • Observe "private bytes" usage across all the browser processes.
• Observations:
  • Firefox (all configs) is more responsive than other browsers. Chrome is not far behind, IE/Edge much behind with terrible responsiveness (when opening a new tab + enter a URL + navigate to it).
  • Chrome always has at least one process per tab.
  • IE/Edge seem to limit the number of processes to ~8.
  • Firefox seem to scale reasonably linearly per tab with e10s-multi.
  • Chrome (with process per tab) would be the reference:
    • Firefox no-e10s is similar to chrome (more than chrome for the "no URL tabs" case)
    • IE is second with ~60-70% more memory than chrome (less than chrome for the "no URL tabs" case)
    • Bug 1213435 Firefox e10s-multi uses roughly twice the memory which Chrome uses
    • Edge slightly more memory than Firefox e10s-multi (though less than Firefox for the "no URL tabs" case)
  • E.g. for 15 tabs:
    • Chrome: 720M, 18 processes
    • Firefox nightly 44 no-e10s: 720M, 1 process
    • Firefox nightly 44 e10s: 810M, 2 processes
    • Firefox nightly 44 e10s-8: 1146M, 9 processes
    • IE11: 1190M, 9 processes
    • Firefox nightly 44 e10s-multi: 1502M, 16 processes
    • Edge: 1550M, 8 processes

Google maps zoom animation smoothness (between browsers)

• Performed by: vladan, avih
• Test how well the google maps zoom in/out animation is (using mouse wheel scroll over the map).
• We tested this zoom animation since vladan thought Firefox might perform worse than chrome.
• System:
  • vladan: Lenovo W520 laptop + NVidia GPU + 3 monitors (1 built in, two external), Windows 7.
  • avih: Reference HP laptop, Windows 10.
• Procedure:
  • Tested browsers: Firefox release (40) / Nightly (44) / Chrome / IE / Edge
  • Load maps.google.com, cue it to a city (e.g. Toronto), then use mouse-wheel to zoom in and out (stayed zoomed in within city limits at all times).
• Observations (avih):
  • TL;DR: Firefox is not worse than others and all browsers are quite similar.
  • All browsers initially zoom in/out in discrete steps, but once enough data has loaded, the zoom change switches to smooth.
  • While in discrete steps: Edge and IE update slightly slower than nightly/release/chrome.
  • _maybe_ Firefox release and Edge/IE take longer to load the data than nightly/chrome.
  • While in smooth zoom mode, all browsers behave pretty much the same, and very nicely at that.
• TODO:
  • Settle the disagreement: vladan: Firefox is worse, avih: all browsers are similar enough.

Fennec

Reference Device

• Samsung Galaxy S4
• Android Lollipop 5.0.1 (official update)
• Not rooted

Experiments

Navigation and scroll using top 20 sites (between browsers)

• Performed by: avih
• Reproduced partially: vladan
• Tested using:
  • Fennec Aurora 43 API11 2015-09-30 (chosen together with :kats)
  • Chrome
  • The device default "Internet" browser
• Procedure:
  • Visit the top 20 sites (if the main page is too plain, use a meaningful page instead, e.g. facebook.com/barackobama):
    • google, facebook, youtube, baidu, yahoo, amazon, wikipedia, qq, twitter, taobao, sina, linkedin, ebay, yandex, bing, msn, instagram, aliexpress, apple, ask.
  • On each site scroll manually and observe the behavior
  • Visit some internal links and observe the navigation behavior
  • Take notes of the relative performance diffs between browsers
• Observations:
  • "Internet" and Chrome are similar, but between them Chrome is slightly better. We'll focus on Fennec vs Chrome.
  • Page load is similar between browsers
  • Bug 1217415 Page navigation is faster in Chrome on some sites (e.g. Wikipedia, ebay) but similar on other sites.
  • Bug 1217372 Fennec has text input lag in autocomplete boxes (google, bing) which Chrome doesn't.
  • In general, scroll is better in chrome, but on "light" pages Fennec gets pretty close.
  • Scroll _while_ the page is loading is more responsive in Fennec (but still not very good)
  • Bug 1217370 On fast scroll swipes, sometimes the momentum is less than expected.
  • Scroll on some pages (e.g. yahoo.com) is considerably worse in Fennec:
    • Bug 1217364 Inconsistent progression (momentum) without user inputs.
    • Bug 1217366 Visible low resolution rendering, especially for fast swipes.