YouTube playback performance test

The benchmark is measuring the playback performance of video streaming by assembling dropped frames during video playback. The test suite is a copy of the Playback Performance Tests from Google, and run in our own infrastructure.

Running locally

For information in how to setup your system to run the tests locally consult the general Raptor documentation first.

The benchmark test can be run on Firefox desktop via the following command:

   $ ./mach raptor-test --test=raptor-youtube-playback 

To run the benchmark on the android geckoview example app, use this command:

   $ ./mach raptor-test --test raptor-youtube-playback-geckoview --app=geckoview --binary="org.mozilla.geckoview_example"

Customizing the test execution

There are some variables you can tweak in the raptor-youtube-playback manifest file to customize the benchmark test.

Subset of tests

To only run a subset of tests the benchmark test page supports the tests argument, which allows to add any number of tests based on their index. There are two possible methods to specify the list of tests:

• Run arbitrary videos, eg. of index 6, 47, and 90:

   test_url = http://...main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&tests=6,47,90

• Run a range of videos eg. from index 23 to 42

   test_url = http://...main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&tests=23-42

This is helpful if specific video codecs, resolutions, or playback speeds are causing performance issues.

Running with audio muted

Especially on Android the tests are running with audio muted to workaround the auto-play blocking issue as covered on Bug 1547717.

If the same should be done for the desktop applications, the muted argument of the test page:

   test_url = http://...main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true

Profiling

For information in how to enable and customize the profiler consult the general Raptor documentation first.

By default the profiler runs with a lower sampling rate to allow the recording of all the videos as best without loosing data due to log rotation. It means that not all the stack frames will be recorded, and as such information might be lost. But given the fact that the markers are more important, but those are all still present, it can be neglected.

If there is a performance issue with one of the videos, it would be wise to only test that specific video with an increased sampling rate for the profiler. Therefore modify the gecko profiler settings in the manifest file to something like:

   gecko_profile_interval = 1
   gecko_profile_entries = 14000000

Updating the test suite

In irregular intervals Google updates its test suite. As we also want to have those changes mirrored for our instance the perf-youtube-playback repository has been created. Changing its content on the raptor branch will automatically be picked-up by yttest.prod.mozaws.net where the tests are hosted on.

Steps in how to update the code can be found at https://github.com/mozilla/perf-youtube-playback/wiki.