Talos

Talos is a cross-platform Python performance testing framework that is specifically for Firefox on desktop. New performance tests should be added to the newer framework mozperftest unless there are limitations there (highly unlikely) that make it absolutely necessary to add them to Talos. Talos is named after the bronze automaton from Greek myth.

Talos tests are run in a similar manner to xpcshell and mochitests. They are started via the command mach talos-test. A python script then launches Firefox, which runs the tests via JavaScript special powers. The test timing information is recorded in a text log file, e.g. browser_output.txt, and then processed into the JSON format supported by Perfherder.

Talos bugs can be filed in Testing::Talos.

Talos infrastructure is still mostly documented on the Mozilla Wiki. In addition, there are plans to surface all of the individual tests using PerfDocs. This work is tracked in Bug 1674220.

Examples of current Talos runs can be found in Treeherder by searching for “Talos”. If none are immediately available, then scroll to the bottom of the page and load more test runs. The tests all share a group symbol starting with a T, for example T(c d damp g1) or T-gli(webgl).

Running Talos Locally

Running tests locally is most likely only useful for debugging what is going on in a test, as the test output is only reported as raw JSON. The CLI is documented via:

./mach talos-test --help

To quickly try out the ./mach talos-test command, the following can be run to do a single run of the DevTools’ simple netmonitor test.

# Run the "simple.netmonitor" test very quickly with 1 cycle, and 1 page cycle.
./mach talos-test --activeTests damp --subtests simple.netmonitor --cycles 1 --tppagecycles 1

The --print-suites and --print-tests are two helpful command flags to figure out what suites and tests are available to run.

# Print out the suites:
./mach talos-test --print-suites

# Available suites:
#  bcv                          (basic_compositor_video)
#  chromez                      (about_preferences_basic:tresize:about_newtab_with_snippets)
#  dromaeojs                    (dromaeo_css:kraken)
#  flex                         (tart_flex:ts_paint_flex)
# ...

# Run all of the tests in the "bcv" test suite:
./mach talos-test --suite bcv

# Print out the tests:
./mach talos-test --print-tests

# Available tests:
# ================
#
# a11yr
# -----
# This test ensures basic a11y tables and permutations do not cause
# performance regressions.
#
# about_newtab_with_snippets
# --------------------------
# Load about ActivityStream (about:home and about:newtab) with snippets enabled
#
# ...

# Run the tests in "a11yr" listed above
./mach talos-test --activeTests a11yr

Running Talos on Try

Talos runs can be generated through the mach try fuzzy finder:

./mach try fuzzy

The following is an example output at the time of this writing. Refine the query for the platform and test suites of your choosing.

| test-windows10-64-qr/opt-talos-bcv-swr-e10s
| test-linux64-shippable/opt-talos-webgl-e10s
| test-linux64-shippable/opt-talos-other-e10s
| test-linux64-shippable-qr/opt-talos-g5-e10s
| test-linux64-shippable-qr/opt-talos-g4-e10s
| test-linux64-shippable-qr/opt-talos-g3-e10s
| test-linux64-shippable-qr/opt-talos-g1-e10s
| test-windows10-64/opt-talos-webgl-gli-e10s
| test-linux64-shippable/opt-talos-tp5o-e10s
| test-linux64-shippable/opt-talos-svgr-e10s
| test-linux64-shippable/opt-talos-flex-e10s
| test-linux64-shippable/opt-talos-damp-e10s
> test-windows7-32/opt-talos-webgl-gli-e10s
| test-linux64-shippable/opt-talos-bcv-e10s
| test-linux64-shippable/opt-talos-g5-e10s
| test-linux64-shippable/opt-talos-g4-e10s
| test-linux64-shippable/opt-talos-g3-e10s
| test-linux64-shippable/opt-talos-g1-e10s
| test-linux64-qr/opt-talos-bcv-swr-e10s

  For more shortcuts, see mach help try fuzzy and man fzf
  select: <tab>, accept: <enter>, cancel: <ctrl-c>, select-all: <ctrl-a>, cursor-up: <up>, cursor-down: <down>
  1379/2967
> talos