TestEngineering/Performance/Raptor/Mitmproxy
Contents
Mitmproxy
Instead of using live web pages for performance testing, Raptor uses a tool called Mitmproxy. Mitmproxy allows us to record a live web page and save it as a playback archive. Then during the Raptor pageload test (i.e. raptor-tp6) we use Mitmproxy's 'mitmdump' tool to playback the archive through a local proxy. Raptor automatically configures Firefox to use the proxy, and when the test browses to the test page URL, it loads the page from the Mitmproxy playback archive.
For more information about Mitmproxy installation, etc. see the documentation. Mitmproxy is an open source tool and the source is found here on GitHub.
Test Page Recordings
Test pages used for Raptor pageload tests (i.e. raptor-tp6, raptor-gdocs) are mitmproxy recordings that are played back during the test (and ultimately loaded in Firefox via the local proxy). Each test page is a separate mitmproxy recording (*.mp) file, and all the page recordings for each suite are contained in a single zip for that suite (i.e. mitmproxy-recordings-raptor-tp6.zip) on tooltool.
When the Raptor pageload test is run, the mitmproxy recording archive for use during the test is automatically downloaded from tooltool.
Custom Playback Script
When the mitmproxy recording is played back in production, we use a custom playback script. The script will return 404s for unknown URLs instead of dropping the entire connection.
This is an example of the command line used in production (Linux x64) to start mitmproxy and playback one of the recording archives, using the custom playback script:
/home/cltbld/tasks/task_1541153570/testing/raptor/mitmdump -k -q -s /home/cltbld/tasks/task_1541153570/build/tests/raptor/raptor/playback/alternate-server-replay.py /home/cltbld/tasks/task_1541153570/testing/raptor/facebook.mp
How to Record a Mitmproxy Test Page on Firefox Desktop
Test pages used for Raptor pageload tests (i.e. raptor-tp6, raptor-gdocs) are mitmproxy recordings that are played back during the test (and ultimately loaded in Firefox via the local proxy). Each test page is a separate mitmproxy recording (*.mp) file, and all the page recordings for each suite are contained in a single zip for that suite (i.e. mitmproxy-recordings-raptor-tp6.zip) on tooltool.
When the Raptor pageload test is run, the mitmproxy recording archive for use during the test is automatically downloaded from tooltool.
The following process was used to record the mitmproxy page archives (on macOS):
1. Install Mitmproxy 2.X following the mitmproxy installation instructions. We use version 2.0.2 in production (and that was the version used to record the current pagesets). Note that we are unable to upgrade to a newer Mitmproxy because of some non-backwards compatible changes they made, see Bug 1457274).
2. Set up a local proxy in Firefox:
- Start Firefox
- Preferences => General
- Network Proxy => Settings
- On the "Connection Settings" screen, select "Manual proxy configuration"
- For "HTTP Proxy" type in "127.0.0.1" with port "8080"
- For "SSL Proxy" use the same "127.0.0.1" with port "8080"
- Click the "OK" button to save the proxy settings
3. Install the Mitmproxy CA certificate:
- Open a terminal window
- Startup Mitmproxy in host mode:
mitmproxy --host
- In Firefox, browse to "mitm.it" and follow the directions on how to accept the CA certificate
- Shutdown the Mitmproxy tool (in terminal hit "Q", then "Y" to quit)
4. Record a new page:
- Start Firefox with the proxy still enabled
- Clear the browser history/cache
- In a terminal window start the mitmdump recording tool:
mitmdump -w /path/to/recording.mp
- Inside Firefox browse to the URL that you want to record (i.e. www.spacex.com)
- Wait for the page to be fully loaded and displayed
- In the mitmdump terminal window press "ctrl + c" to stop the recording
5. To test playing back your recorded page:
- Be sure you have the custom playback script available
- Start Firefox with the proxy still enabled
- With Mitmproxy NOT running, browse to your recorded URL (i.e. www.spacex.com); you'll just get an error saying that the proxy server is refusing connections
- In a terminal window, start Mitmproxy playback, using the custom playback script:
mitmdump -k -s /path/to/alternate-server-replay.py /path/to/recording.mp
For example:
mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor $ ./mitmdump -k -s "/Users/rwood/mozilla-unified/testing/raptor/raptor/playback/alternate-server-replay.py /Users/rwood/Desktop/new_recordings/no_hero/google-search-no-hero.mp"
NOTE: On some platforms you will need quotes around the args as used above.
- In Firefox browse to the URL that you recorded already (i.e. www.spacex.com). This time the page will load successfully; it is actually loading the page from the local mitmdump archive file (*.mp) and not the external site
- You can actually turn off your local WiFi connection if you want and verify the page still loads
- In the terminal window press "ctrl + c" to stop the playback
6. When you're finished remember to turn off your Firefox proxy:
- Preferences => General
- Network Proxy => Settings
- Select "No proxy" and click the "OK" button
Adding Hero Elements
Hero elements are special html attributes that can be inserted into existing html elements in pages, so that we can measure pageload up to the time that specific element is displayed. You basically just add an 'element_timing' attribute to an existing html element i.e. <element_timing='hero1'>.
Raptor supports multiple measurements per single pageload including hero elements. To have a Raptor test measure an existing hero element, you simply add 'hero' to the 'measure = ' line in the Raptor test INI file, and below that add an 'hero = hero1' line that specifies the hero element attribute text (i.e. 'hero1') to look for.
Since Raptor uses Mitmproxy to playback web pages, in order to use hero elements they must be manually added to the web page archive. Tarek created a script ('mitmflow') to add hero elements to existing Mitmproxy page recordings. See the mitmflow repo for more information, but the basic steps to add a hero element to an existing Mitmproxy page archive are:
1. Copy Tarek's mitmflow replace script into the same folder where you have the mitmdump binary.
2. Startup Firefox, turn on the proxy (see settings above).
3. Use Mitmproxy (mitmdump) to playback the web page recording of the page you wish to add the hero element to (see above for mitmdump playback command line syntax).
4. Use the Firefox dev tools page inspector and find an element in the test page where you wish to add the hero element. It should be a unique element like a picture, something with a unique id for example.
5. Update the Mitmproxy replacement script accordingly to indicate which element you want to add the hero element to.
6. Use Mitmproxy (mitmdump) to read your page recording, run it through the Mitmflow replace script, and write out a new Mitmproxy page recording with the element having been added. i.e. with Mitmproxy 2.x:
./mitmdump -dd -s "./replace.py" -r /Users/rwood/Desktop/recordings/google.mp -w /Users/rwood/Desktop/recordings/google-hero.mp
7. Be sure to use Mitmproxy (mitmdump) to playback your new page recording and verify with inspector that the hero element was added successfully (and only once - if there are other elements with the same id then the hero element could be added to multiple elements by mistake).
How to Record a Mitmproxy Test Page on Android
For Raptor page-load tests that run on android (i.e. tp6m-1) the mitmproxy recordings were actually made on an android device (Google Pixel 2) with the geckoview example app.
Recording a mitmproxy page on android is very similar to desktop except it's easier to run an existing android page-load test (i.e. tp6m-1) first to get the device setup before recording. Also you need to `adb reverse` a port so the device can access mitmdump running on the host machine. Here's how to create a mitmproxy recording using the android geckoview example app:
1. Ensure your android device (i.e. GP2) is already setup to run Raptor on the geckoview example app, see [Raptor on the Geckoview Example App]
2. In order to get mitmdump installed on your host machine, and the android device ready to record (i.e. the mitmdump CA certificate installed in the geckoview example app, proxy turned on, etc) run the Raptor tp6m-1 test. With your android device attached to USB, run:
mozilla-central$ ./mach raptor-test --test raptor-tp6m-1 --app=geckoview --binary="org.mozilla.geckoview_example" --page-cycles 1
Wait for that to finish. The geckoview example app will remain open on the device.
3. Clear geckoview_example app data and cache: adb shell pm clear org.mozilla.geckoview_example
4. In your terminal, change into the obj../testing/raptor folder, that is where mitmdump is located, i.e.:
Roberts-MacBook-Pro-1927:mozilla-unified rwood$ cd obj.../testing/raptor
5. ADB reverse the port so that the android device can talk to the mitmproxy server on the host, by running this command in a terminal:
adb reverse tcp:8080 tcp:8080
6. On the android device in the geckoview example app, browse to "about:blank".
7. From within the obj../testing/raptor folder, startup mitmdump recording and specify the path and name for the new recording file, i.e.:
Roberts-MacBook-Pro-1927:raptor rwood$ ./mitmdump -w "/Users/rwood/mozilla-unified/obj-ff-dbg/testing/raptor/new-mobile-recording.mp"
8. With recording running, on the geckoview example app browser to the <new recording's https mobile url> that you wish to record. Wait for the page to load and display fully in the geckoview example app, then in the terminal where mitmdump is running press `ctrl + c` to stop the recording.
To test the new mobile recording:
1. Leave the new recording in the obj../testing/raptor dir, and add a section in the tp6m-1 test INI:
[raptor-tp6-new-mobile-recording-geckoview] page_cycles = 15 apps = geckoview test_url = <new recording's https mobile url> playback_recordings = new-mobile-recording.mp measure = fnbpaint, fcp, dcf, ttfi, loadtime
2. In your terminal change back into the root of your repo i.e. \mozilla-central\ and run the modified tp6m-1 on geckoview with the command:
mozilla-central$ ./mach raptor-test --test raptor-tp6-new-mobile-recording-geckoview --app=geckoview --binary="org.mozilla.geckoview_example"
Watch the test run on the android device and verify that the test page is loaded correctly in the geckoview example app. Wait for Raptor to finish and report the results - verify that all of the measurements were successfully retrieved.
Raptor-Studio
A simple command line tool for recording and replaying web traffic for Raptor.
Note: Raptor-Studio is still under development. Currently limited to record/replay on the GeckoView example app for Android.
Source
Source code can be found in github at https://github.com/davehunt/raptor-studio
Installation
Use pipenv to install dependencies
$ pipenv install
Usage
See command line options on how to configure and run the app
$ pipenv run python studio.py --help
