We verify and test Marionette in a couple of different ways. The server uses a combination of xpcshell unit tests and functional Python-based tests written with the Marionette harness and WPT; the client is tested with the same harness; and the Marionette test harness uses a collection of Python tests.
Additionally, for debugging hard-to-reproduce test failures in CI, a one-click loaner from Taskcluster can be used.
Marionette-based tests can be run through
mach with a local
checkout of central and a build of Firefox, as well as out-of-tree
marionette test harness.
Running in-tree tests,
mach will automatically manage the Python
virtual environment in which your tests are run. The Marionette
client that is picked up is the one that is in-tree at
If you want to run tests from a downloaded test archive, you will
need to download the
target.common.tests.zip artifact as attached
to Treeherder build jobs
B for your system. Extract that file
and set up the Python Marionette client and harness by executing
the following command in a virtual environment:
% pip install -r config/marionette_requirements.txt
The tests can then be found under
can be executed with the command
marionette. It supports the
same options as described below for
xpcshell unit tests¶
Marionette has a set of xpcshell unit tests located in _testing/marionette/test*.js_. These can be run this way:
% ./mach test testing/marionette/test/unit
Because tests are run in parallel and xpcshell itself is quite chatty, it can sometimes be useful to run the tests sequentially:
% ./mach test --sequential testing/marionette/test/unit/test_error.js
These unit tests run as part of the
X jobs on Treeherder.
Marionette functional tests¶
We also have a set of functional tests that make use of the Marionette Python client. These start a Firefox process and tests the Marionette protocol input and output. The following command will run all tests:
% ./mach marionette test
But you can also run individual tests:
% ./mach marionette test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
In case you want to run the tests with another Firefox binary:
% ./mach marionette test --binary /path/to/firefox TEST
When working on Marionette it is often useful to surface the stdout
from Firefox, which can be achived using the
See Debugging for usage instructions.
As these are functional integration tests and pop up Firefox windows sporadically, a helpful tip is to suppress the window whilst you are running them by using Firefox’ headless mode:
% ./mach marionette test -z TEST
-z is an alias for
--headless and equivalent to setting the
MOZ_HEADLESS output variable. In addition to
MOZ_HEADLESS_HEIGHT for controlling
the dimensions of the no-op virtual display. This is similar to
using xvfb(1) which you may know from the X windowing system, but
has the additional benefit of also working on macOS and Windows.
These functional tests will run as part of the
Mn job on Treeherder.
In addition to these two test types that specifically test the Marionette protocol, Marionette is used as the backend for the geckodriver WebDriver implementation. It is served by a WPT test suite which effectively tests conformance to the W3C specification.
WPT functional tests¶
Marionette is also indirectly tested through geckodriver with
WPT. WPT tests for WebDriver turn up with the
Wd try job and can
be run this way:
% ./mach wpt testing/web-platform/tests/webdriver/tests
This command supports a
--webdriver-arg '-vv' argument which
enables more detailed logging, as well as
--jsdebugger for opening
the Browser Toolbox.
The Marionette harness Python package has a set of unit tests, which are written by using the pytest framework. The following command will run all tests:
% ./mach python-test testing/marionette/
To run a specific test specify the full path to the module:
% ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py