We verify and test Marionette in a couple of different ways, using a combination of unit tests and functional tests. There are three distinct components that we test:
the Marionette server, using a combination of xpcshell unit tests and functional tests written in Python spread across Marionette- and WPT tests;
the Python client is tested with the same body of functional Marionette tests;
and the harness that backs the Marionette, or
Mnjob on try, tests is verified using separate mock-styled unit tests.
All these tests can be run by using mach.
xpcshell unit tests¶
Marionette has a set of xpcshell unit tests located in _remote/marionette/test/xpcshell. These can be run this way:
% ./mach test remote/marionette/test/unit
Because tests are run in parallel and xpcshell itself is quite chatty, it can sometimes be useful to run the tests sequentially:
% ./mach test --sequential remote/marionette/test/xpcshell/test_error.js
These unit tests run as part of the
X jobs on Treeherder.
Marionette functional tests¶
We also have a set of functional tests that make use of the Marionette
Python client. These start a Firefox process and tests the Marionette
protocol input and output, and will appear as
Mn on Treeherder.
The following command will run all tests locally:
% ./mach marionette-test
But you can also run individual tests:
% ./mach marionette-test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
In case you want to run the tests with another binary like Firefox Nightly:
% ./mach marionette-test --binary /path/to/nightly/firefox TEST
When working on Marionette it is often useful to surface the stdout
from Gecko, which can be achieved using the
See Debugging for usage instructions, but the gist is that
you can redirect all Gecko output to stdout:
% ./mach marionette-test --gecko-log - TEST
Our functional integration tests pop up Firefox windows sporadically, and a helpful tip is to suppress the window can be to use Firefox’ headless mode:
% ./mach marionette-test -z TEST
-z is an alias for the
--headless flag and equivalent to setting
MOZ_HEADLESS output variable. In addition to
there is also
controlling the dimensions of the no-op virtual display. This is
similar to using Xvfb(1) which you may know from the X windowing system,
but has the additional benefit of also working on macOS and Windows.
You have built Fennec.
You can run an Android emulator, which means you have the AVD you need.
When running tests on Fennec, you can have Marionette runner take care of starting Fennec and an emulator, as shown below.
% ./mach marionette-test --emulator --app fennec --avd-home /path/to/.mozbuild/android-device/avd --emulator-binary /path/to/.mozbuild/android-sdk/emulator/emulator --avd=mozemulator-x86
For Fennec tests, if the appropriate
emulator command is in your
PATH, you may omit the
--emulator-binary argument. See
./mach marionette-test -h
for additional options.
Alternately, you can start an emulator yourself and have the Marionette runner start Fennec for you:
% ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828
To connect to an already-running Fennec in an emulator or on a device,
you will need to have it started with the
-marionette command line argument,
or by setting the environment variable
MOZ_MARIONETTE=1 for the process.
Make sure port 2828 is forwarded:
% adb forward tcp:2828 tcp:2828
If Fennec is already started:
% ./mach marionette-test --app='fennec' --address=127.0.0.1:2828
If Fennec is not already started on the emulator/device, add the
option. Marionette Test Runner will take care of forwarding the port and
starting Fennec with the correct prefs. (You may need to run
adb forward --remove-all to allow the runner to start.)
% ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828 --startup-timeout=300
If you need to troubleshoot the Marionette connection, the most basic check is
to start Fennec with
-marionette or the environment variable
make sure that the port 2828 is forwarded, and then see if you get any response from
Marionette when you connect manually:
% telnet 127.0.0.1:2828
You should see output like
WPT functional tests¶
Marionette is also indirectly tested through geckodriver with WPT
Wd on Treeherder). To run them:
% ./mach wpt testing/web-platform/tests/webdriver
WPT tests conformance to the WebDriver standard and uses geckodriver. Together with the Marionette remote protocol in Gecko, they make up Mozilla’s WebDriver implementation.
This command supports a
--webdriver-arg='-vv' argument that
enables more detailed logging, as well as
--jsdebugger for opening
the Browser Toolbox.
A particularly useful trick is to combine this with the headless mode for Firefox:
% ./mach wpt --webdriver-arg='-vv' --headless testing/web-platform/tests/webdriver
The Marionette harness Python package has a set of mock-styled unit tests that uses the pytest framework. The following command will run all tests:
% ./mach python-test testing/marionette
To run a specific test specify the full path to the module:
% ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py
Additionally, for debugging hard-to-reproduce test failures in CI, one-click loaners from Taskcluster can be particularly useful.
All the above examples show tests running in-tree, with a local checkout of central and a local build of Firefox. It is also possibly to run the Marionette tests without a local build and with a downloaded test archive from Taskcluster
If you want to run tests from a downloaded test archive, you will
need to download the
target.common.tests.tar.gz artifact attached to
Treeherder build jobs
B for your system. Extract the archive
and set up the Python Marionette client and harness by executing
the following command in a virtual environment:
% pip install -r config/marionette_requirements.txt
The tests can then be found under
and can be executed with the command
marionette. It supports
the same options as described above for