Project Eideticker/Profiling with Eideticker

Scenario 1: Measuring with video capture

Advantages:

• Gives the most comprehensive and reliable results.

Disadvantages:

• Slow. Each capture/analysis combination takes about 5-10 minutes. Multiplied by the number of tests that you would want to run and the number of runs you'd need to get a representative result and you're easily talking several hours.
• Requires a dedicated machine with an HDMI-capable phone connected to it.

ssh into an Eideticker machine (as of this writing, we have several in Mountain View and a few on the way in Toronto, ask William Lachance for details) and run the "get-metric-for-build" script:

cd src/eideticker
source bin/activate
wget <apk of build>
./bin/get-metric-for-build.py --num-runs 5 <apk of build> <path to test>

Tests that are good to run for a representative sample of performance include CNN.com ('src/tests/cnn/cnn.com/index.html'), nytimes-zoom ('src/tests/ep1/nytimes/nytimes.com/nytimes-zoom.html') and nytimes-scroll ('src/tests/ep1/nytimes/nytimes.com/nytimes-scroll.html').

You will get a bunch of output regarding amount of checkerboarding and number of unique frames. This may be compared to the existing Eideticker dashboard.

Scenario 2: Measuring with telemetry

Advantages:

• Fast. Can do a test in under a minute.
• Work on any suitably configured linux machine.

Disadvantages:

• Not technically a black box test, relies on internal measurements that may be inaccurate
• Only measures checkerboarding, not the smoothness of animations, etc.
• Results are only relevant to Fennec, and are not comparable with other browsers.

The Eideticker README contains instructions on how to set up / run this type of test.