Raptor WebExtension¶

Note: WebExtension is being deprecated, new tests should be written using Browsertime

WebExtension Page-Load Tests ¶

Page-load tests involve loading a specific web page and measuring the load performance (i.e. time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed).

For page-load tests by default, instead of using live web pages for performance testing, Raptor uses a tool called Mitmproxy. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on tooltool and are automatically downloaded by Raptor when they are required for a test. Raptor uses mitmproxy via the mozproxy package.

There are two different types of Raptor page-load tests: warm page-load and cold page-load.

Warm Page-Load ¶

For warm page-load tests, the browser is just started up once; so the browser is warm on each page-load.

Raptor warm page-load test process when running on Firefox/Chrome desktop:

A new browser profile is created
The desktop browser is started up
Post-startup browser settle pause of 30 seconds
A new tab is opened
The test URL is loaded; measurements taken
The tab is reloaded 24 more times; measurements taken each time
The measurements from the first page-load are not included in overall results metrics b/c of first load noise; however they are listed in the JSON artifacts

Raptor warm page-load test process when running on Firefox android browser apps:

The android app data is cleared (via adb shell pm clear firefox.app.binary.name)
The new browser profile is copied onto the android device SD card
The Firefox android app is started up
Post-startup browser settle pause of 30 seconds
The test URL is loaded; measurements taken
The tab is reloaded 14 more times; measurements taken each time
The measurements from the first page-load are not included in overall results metrics b/c of first load noise; however they are listed in the JSON artifacts

Cold Page-Load ¶

For cold page-load tests, the browser is shut down and restarted between page load cycles, so the browser is cold on each page-load. This is what happens for Raptor cold page-load tests:

Raptor cold page-load test process when running on Firefox/Chrome desktop:

A new browser profile is created
The desktop browser is started up
Post-startup browser settle pause of 30 seconds
A new tab is opened
The test URL is loaded; measurements taken
The tab is closed
The desktop browser is shut down
Entire process is repeated for the remaining browser cycles (25 cycles total)
The measurements from all browser cycles are used to calculate overall results

Raptor cold page-load test process when running on Firefox Android browser apps:

The Android app data is cleared (via adb shell pm clear firefox.app.binary.name)
A new browser profile is created
The new browser profile is copied onto the Android device SD card
The Firefox Android app is started up
Post-startup browser settle pause of 30 seconds
The test URL is loaded; measurements taken
The Android app is shut down
Entire process is repeated for the remaining browser cycles (15 cycles total)
Note that the SSL cert DB is only created once (browser cycle 1) and copied into the profile for each additional browser cycle, thus not having to use the ‘certutil’ tool and re-created the db on each cycle
The measurements from all browser cycles are used to calculate overall results

Using Live Sites ¶

It is possible to use live web pages for the page-load tests instead of using the mitmproxy recordings. To do this, add --live to your command line or select one of the ‘live’ variants when running ./mach try fuzzy --full.

Disabling Alerts ¶

It is possible to disable alerting for all our performance tests. Open the target test manifest such as the raptor-tp6*.ini file (Raptor tests folder), and make sure there are no alert_on specifications.

When it’s removed there will no longer be a shouldAlert field in the output Perfherder data (you can find the schema here). As long as shouldAlert is not in the data, no alerts will be generated. If you need to also disable code sheriffing for the test, then you need to change the tier of the task to 3.

High value tests ¶

We have a notion of high-value tests in performance testing. These are chosen based on how many alerts they can catch using this script along with a redash query. The lists below are the minimum set of test pages we should run to catch as many alerts as we can.

Desktop

Last updated: June 2021

amazon

bing

cnn

fandom

gslides

instagram

twitter

wikipedia

yahoo-mail

Mobile

Last updated: November 2020

allrecipes

amazon-search

espn

facebook

google-search

microsoft-support

youtube-watch

WebExtension Benchmark Tests ¶

Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI.

Scenario Tests ¶

Currently, there are three subtypes of Raptor-run “scenario” tests, all on (and only on) Android:

power-usage tests
memory-usage tests
CPU-usage tests

For a combined-measurement run with distinct Perfherder output for each measurement type, you can do:

./mach raptor --test raptor-scn-power-idle-bg-fenix --app fenix --binary org.mozilla.fenix.performancetest --host 10.0.0.16 --power-test --memory-test --cpu-test

Each measurement subtype (power-, memory-, and cpu-usage) will have a corresponding PERFHERDER_DATA blob:

22:31:05     INFO -  raptor-output Info: PERFHERDER_DATA: {"framework": {"name": "raptor"}, "suites": [{"name": "raptor-scn-power-idle-bg-fenix-cpu", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 0, "subtests": [{"lowerIsBetter": true, "unit": "%", "name": "cpu-browser_cpu_usage", "value": 0, "alertThreshold": 2.0}], "type": "cpu", "unit": "%"}]}
22:31:05     INFO -  raptor-output Info: cpu results can also be found locally at: /Users/sdonner/moz_src/mozilla-unified/testing/mozharness/build/raptor-cpu.json

(repeat for power, memory snippets)

Power-Use Tests (Android)¶

Prerequisites¶

rooted (i.e. superuser-capable), bootloader-unlocked Google Pixel 6 or Samsung A51: internal (for now) test-device setup doc.
set up to run Raptor from a Firefox source tree (see Running Locally)
GeckoView-bootstrapped environment

Raptor power-use measurement test process when running on Firefox Android browser apps:

The Android app data is cleared, via:

adb shell pm clear firefox.app.binary.name

The new browser profile is copied onto the Android device’s SD card
We set scenario_time to 20 minutes (1200000 milliseconds), and page_timeout to ‘22 minutes’ (1320000 milliseconds)

It’s crucial that ``page_timeout`` exceed ``scenario_time``; if not, measurement tests will fail/bail early

We launch the {Fenix, GeckoView, Reference Browser} on-Android app
Post-startup browser settle pause of 30 seconds
On Fennec only, a new browser tab is created (other Firefox apps use the single/existing tab)
Power-use/battery-level measurements (app-specific measurements) are taken, via:

adb shell dumpsys batterystats

Raw power-use measurement data is listed in the perfherder-data.json/raptor.json artifacts

In the Perfherder dashboards for these power usage tests, all data points have milli-Ampere-hour units, with a lower value being better. Proportional power usage is the total power usage of hidden battery sippers that is proportionally “smeared”/distributed across all open applications.

Running Scenario Tests Locally ¶

To run on a tethered phone via USB from a macOS host, on:

Fenix¶

./mach raptor --test raptor-scn-power-idle-fenix --app fenix --binary org.mozilla.fenix.performancetest --power-test --host 10.252.27.96

GeckoView¶

./mach raptor --test raptor-scn-power-idle-geckoview --app geckoview --binary org.mozilla.geckoview_example --power-test --host 10.252.27.96

Reference Browser¶

./mach raptor --test raptor-scn-power-idle-refbrow --app refbrow --binary org.mozilla.reference.browser.raptor --power-test --host 10.252.27.96

NOTE: it is important that you include --power-test, when running power-usage measurement tests, as that will help ensure that local test-measurement data doesn’t accidentally get submitted to Perfherder

Writing New Tests ¶

Pushing to Try server ¶

As an example, a relatively good cross-sampling of builds can be seen in https://hg.mozilla.org/try/rev/6c07631a0c2bf56b51bb82fd5543d1b34d7f6c69.

Include both G5 Android 7 (hw-g5-7-0-arm7-api-16/) and Pixel 2 Android 8 (p2-8-0-android-aarch64/) target platforms
pgo builds tend to be – from my limited empirical evidence – about 10 - 15 minutes longer to complete than their opt counterparts

Dashboards ¶

See performance results for our various dashboards.

Running WebExtension Locally ¶

WebExtension Prerequisites ¶

In order to run Raptor on a local machine, you need:

A local mozilla repository clone with a successful Firefox build completed
Git needs to be in the path in the terminal/window in which you build Firefox / run Raptor, as Raptor uses Git to check-out a local copy for some of the performance benchmarks’ sources.
If you plan on running Raptor tests on Google Chrome, you need a local install of Google Chrome and know the path to the chrome binary
If you plan on running Raptor on Android, your Android device must already be set up (see more below in the Android section)

Getting a List of Raptor Tests ¶

To see which Raptor performance tests are currently available on all platforms, use the ‘print-tests’ option, e.g.:

$ ./mach raptor --print-tests

That will output all available tests on each supported app, as well as each subtest available in each suite (i.e. all the pages in a specific page-load tp6* suite).

Running on Firefox ¶

To run Raptor locally, just build Firefox and then run:

$ ./mach raptor --test <raptor-test-name>

For example, to run the raptor-tp6 pageload test locally, just use:

$ ./mach raptor --test raptor-tp6-1

You can run individual subtests too (i.e. a single page in one of the tp6* suites). For example, to run the amazon page-load test on Firefox:

$ ./mach raptor --test raptor-tp6-amazon-firefox

Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json.

Running on the Android GeckoView Example App ¶

When running Raptor tests on a local Android device, Raptor is expecting the device to already be set up and ready to go.

First, ensure your local host machine has the Android SDK/Tools (i.e. ADB) installed. Check if it is already installed by attaching your Android device to USB and running:

$ adb devices

If your device serial number is listed, then you’re all set. If ADB is not found, you can install it by running (in your local mozilla-development repo):

$ ./mach bootstrap

Then, in bootstrap, select the option for “Firefox for Android Artifact Mode,” which will install the required tools (no need to do an actual build).

Next, make sure your Android device is ready to go. Local Android-device prerequisites are:

Device is rooted

Note: If you are using Magisk to root your device, use version 17.3

Device is in ‘superuser’ mode
The geckoview example app is already installed on the device (from ./mach bootstrap, above). Download the geckoview_example.apk from the appropriate android build on treeherder, then install it on your device, i.e.:

$ adb install -g ../Downloads/geckoview_example.apk

The ‘-g’ flag will automatically set all application permissions ON, which is required.

Note, when the Gecko profiler should be run, or a build with build symbols is needed, then use a Nightly build of geckoview_example.apk.

When updating the geckoview example app, you MUST uninstall the existing one first, i.e.:

$ adb uninstall org.mozilla.geckoview_example

Once your Android device is ready, and attached to local USB, from within your local mozilla repo use the following command line to run speedometer:

$ ./mach raptor --test raptor-speedometer --app=geckoview --binary="org.mozilla.geckoview_example"

Note: Speedometer on Android GeckoView is currently running on two devices in production - the Google Pixel 6 and the Samsung A51 - therefore it is not guaranteed that it will run successfully on all/other untested android devices.

To run a Raptor page-load test (i.e. tp6m-1) on the GeckoView Example app, use this command line:

$ ./mach raptor --test raptor-tp6m-1 --app=geckoview --binary="org.mozilla.geckoview_example"

A couple notes about debugging:

Raptor browser-extension console messages do appear in adb logcat via the GeckoConsole - so this is handy:

$ adb logcat | grep GeckoConsole

You can also debug Raptor on Android using the Firefox WebIDE; click on the Android device listed under “USB Devices” and then “Main Process” or the ‘localhost: Speedometer..” tab process

Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json.

Running on Google Chrome ¶

To run Raptor locally on Google Chrome, make sure you already have a local version of Google Chrome installed, and then from within your mozilla-repo run:

$ ./mach raptor --test <raptor-test-name> --app=chrome --binary="<path to google chrome binary>"

For example, to run the raptor-speedometer benchmark on Google Chrome use:

$ ./mach raptor --test raptor-speedometer --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome

Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json.

Page-Timeouts ¶

On different machines the Raptor tests will run at different speeds. The default page-timeout is defined in each Raptor test INI file. On some machines you may see a test failure with a ‘raptor page-timeout’ which means the page-load timed out, or the benchmark test iteration didn’t complete, within the page-timeout limit.

You can override the default page-timeout by using the –page-timeout command-line arg. In this example, each test page in tp6-1 will be given two minutes to load during each page-cycle:

./mach raptor --test raptor-tp6-1 --page-timeout 120000

If an iteration of a benchmark test is not finishing within the allocated time, increase it by:

./mach raptor --test raptor-speedometer --page-timeout 600000

Page-Cycles ¶

Page-cycles is the number of times a test page is loaded (for page-load tests); for benchmark tests, this is the total number of iterations that the entire benchmark test will be run. The default page-cycles is defined in each Raptor test INI file.

You can override the default page-cycles by using the –page-cycles command-line arg. In this example, the test page will only be loaded twice:

./mach raptor --test raptor-tp6-google-firefox --page-cycles 2

Running Page-Load Tests on Live Sites ¶

To use live pages instead of page recordings, just edit the Raptor tp6* test INI file and add the following attribute either at the top (for all pages in the suite) or under an individual page/subtest heading:

use_live_pages = true

With that setting, Raptor will not start the playback tool (i.e. Mitmproxy) and will not turn on the corresponding browser proxy, therefore forcing the test page to load live.

Running Raptor on Try ¶

Raptor tests can be run on try on both Firefox and Google Chrome. (Raptor pageload-type tests are not supported on Google Chrome yet, as mentioned above).

Note: Raptor is currently ‘tier 2’ on Treeherder, which means to see the Raptor test jobs you need to ensure ‘tier 2’ is selected / turned on in the Treeherder ‘Tiers’ menu.

The easiest way to run Raptor tests on try is to use mach try fuzzy:

$ ./mach try fuzzy --full

Then type ‘raptor’ and select which Raptor tests (and on what platforms) you wish to run.

To see the Raptor test results on your try run:

In treeherder select one of the Raptor test jobs (i.e. ‘sp’ in ‘Rap-e10s’, or ‘Rap-C-e10s’)
Below the jobs, click on the “Performance” tab; you’ll see the aggregated results listed
If you wish to see the raw replicates, click on the “Job Details” tab, and select the “perfherder-data.json” artifact

Raptor Hardware in Production ¶

The Raptor performance tests run on dedicated hardware (the same hardware that the Talos performance tests use). See the performance platforms for more details.

Profiling Raptor Jobs ¶

Raptor tests are able to create Gecko profiles which can be viewed in profiler.firefox.com. This is currently only supported when running Raptor on Firefox desktop.

Nightly Profiling Jobs in Production ¶

We have Firefox desktop Raptor jobs with Gecko-profiling enabled running Nightly in production on Mozilla Central (on Linux64, Win10, and OSX). This provides a steady cache of Gecko profiles for the Raptor tests. Search for the “Rap-Prof” treeherder group on Mozilla Central.

Profiling Locally ¶

To tell Raptor to create Gecko profiles during a performance test, just add the ‘–gecko-profile’ flag to the command line, i.e.:

$ ./mach raptor --test raptor-sunspider --gecko-profile

When the Raptor test is finished, you will be able to find the resulting gecko profiles (ZIP) located locally in:

mozilla-central/testing/mozharness/build/blobber_upload_dir/

Note: While profiling is turned on, Raptor will automatically reduce the number of pagecycles to 3. If you wish to override this, add the –page-cycles argument to the raptor command line.

Raptor will automatically launch Firefox and load the latest Gecko profile in profiler.firefox.com. To turn this feature off, just set the DISABLE_PROFILE_LAUNCH=1 env var.

If auto-launch doesn’t work for some reason, just start Firefox manually and browse to profiler.firefox.com, click on “Browse” and select the Raptor profile ZIP file noted above.

If you’re on Windows and want to profile a Firefox build that you compiled yourself, make sure it contains profiling information and you have a symbols zip for it, by following the directions on MDN.

Profiling on Try Server ¶

To turn on Gecko profiling for Raptor test jobs on try pushes, just add the ‘–gecko-profile’ flag to your try push i.e.:

$ ./mach try fuzzy --gecko-profile

Then select the Raptor test jobs that you wish to run. The Raptor jobs will be run on try with profiling included. While profiling is turned on, Raptor will automatically reduce the number of pagecycles to 2.

See below for how to view the gecko profiles from within treeherder.

Customizing the profiler ¶

If the default profiling options are not enough, and further information is needed the gecko profiler can be customized.

Enable profiling of additional threads¶

In some cases it will be helpful to also measure threads which are not part of the default set. Like the MediaPlayback thread. This can be accomplished by using:

the gecko_profile_threads manifest entry, and specifying the thread names as comma separated list
the –gecko-profile-thread argument for mach for each extra thread to profile

Add Profiling to Previously Completed Jobs ¶

Note: You might need treeherder ‘admin’ access for the following.

Gecko profiles can now be created for Raptor performance test jobs that have already completed in production (i.e. mozilla-central) and on try. To repeat a completed Raptor performance test job on production or try, but add gecko profiling, do the following:

In treeherder, select the symbol for the completed Raptor test job (i.e. ‘ss’ in ‘Rap-e10s’)
Below, and to the left of the ‘Job Details’ tab, select the ‘…’ to show the menu
On the pop-up menu, select ‘Create Gecko Profile’

The same Raptor test job will be repeated but this time with gecko profiling turned on. A new Raptor test job symbol will be added beside the completed one, with a ‘-p’ added to the symbol name. Wait for that new Raptor profiling job to finish. See below for how to view the gecko profiles from within treeherder.

Viewing Profiles on Treeherder ¶

When the Raptor jobs are finished, to view the gecko profiles:

In treeherder, select the symbol for the completed Raptor test job (i.e. ‘ss’ in ‘Rap-e10s’)
Click on the ‘Job Details’ tab below
The Raptor profile ZIP files will be listed as job artifacts;
Select a Raptor profile ZIP artifact, and click the ‘view in Firefox Profiler’ link to the right

Recording Pages for Raptor Pageload Tests ¶

Raptor pageload tests (‘tp6’ and ‘tp6m’ suites) use the Mitmproxy tool to record and play back page archives. For more information on creating new page playback archives, please see Raptor and Mitmproxy.

Performance Tuning for Android devices ¶

When the test is run against Android, Raptor executes a series of performance tuning commands over the ADB connection.

Device agnostic:

memory bus
device remain on when on USB power
virtual memory (swappiness)
services (thermal throttling, cpu throttling)
i/o scheduler

Device specific:

cpu governor
cpu minimum frequency
gpu governor
gpu minimum frequency

For a detailed list of current tweaks, please refer to this Searchfox page.