Testing

We verify and test Marionette in a couple of different ways, using a combination of unit tests and functional tests. There are three distinct components that we test:

  • the Marionette server, using a combination of xpcshell unit tests and functional tests written in Python spread across Marionette- and WPT tests;

  • the Python client is tested with the same body of functional Marionette tests;

  • and the harness that backs the Marionette, or Mn job on try, tests is verified using separate mock-styled unit tests.

All these tests can be run by using mach.

xpcshell unit tests

Marionette has a set of xpcshell unit tests located in _testing/marionette/test/unit. These can be run this way:

% ./mach test testing/marionette/test/unit

Because tests are run in parallel and xpcshell itself is quite chatty, it can sometimes be useful to run the tests sequentially:

% ./mach test --sequential testing/marionette/test/unit/test_error.js

These unit tests run as part of the X jobs on Treeherder.

Marionette functional tests

We also have a set of functional tests that make use of the Marionette Python client. These start a Firefox process and tests the Marionette protocol input and output, and will appear as Mn on Treeherder. The following command will run all tests locally:

% ./mach marionette-test

But you can also run individual tests:

% ./mach marionette-test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py

In case you want to run the tests with another binary like Firefox Nightly:

% ./mach marionette-test --binary /path/to/nightly/firefox TEST

When working on Marionette it is often useful to surface the stdout from Gecko, which can be achieved using the --gecko-log option. See <Debugging.html> for usage instructions, but the gist is that you can redirect all Gecko output to stdout:

% ./mach marionette-test --gecko-log - TEST

Our functional integration tests pop up Firefox windows sporadically, and a helpful tip is to suppress the window can be to use Firefox’ headless mode:

% ./mach marionette-test -z TEST

-z is an alias for the --headless flag and equivalent to setting the MOZ_HEADLESS output variable. In addition to MOZ_HEADLESS there is also MOZ_HEADLESS_WIDTH and MOZ_HEADLESS_HEIGHT for controlling the dimensions of the no-op virtual display. This is similar to using Xvfb(1) which you may know from the X windowing system, but has the additional benefit of also working on macOS and Windows.

Android

Prerequisites:

When running tests on Fennec, you can have Marionette runner take care of starting Fennec and an emulator, as shown below.

% ./mach marionette-test --emulator --app fennec
--avd-home /path/to/.mozbuild/android-device/avd
--emulator-binary /path/to/.mozbuild/android-sdk/emulator/emulator
--avd=mozemulator-x86

For Fennec tests, if the appropriate emulator command is in your PATH, you may omit the --emulator-binary argument. See ./mach marionette-test -h for additional options.

Alternately, you can start an emulator yourself and have the Marionette runner start Fennec for you:

% ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828

To connect to an already-running Fennec in an emulator or on a device, you will need to have it started with the -marionette command line argument, or by setting the environment variable MOZ_MARIONETTE=1 for the process.

Make sure port 2828 is forwarded:

% adb forward tcp:2828 tcp:2828

If Fennec is already started:

% ./mach marionette-test --app='fennec' --address=127.0.0.1:2828

If Fennec is not already started on the emulator/device, add the --emulator option. Marionette Test Runner will take care of forwarding the port and starting Fennec with the correct prefs. (You may need to run adb forward --remove-all to allow the runner to start.)

% ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828 --startup-timeout=300

If you need to troubleshoot the Marionette connection, the most basic check is to start Fennec with -marionette or the environment variable MOZ_MARIONETTE=1, make sure that the port 2828 is forwarded, and then see if you get any response from Marionette when you connect manually:

% telnet 127.0.0.1:2828

You should see output like {"applicationType":"gecko","marionetteProtocol":3}

WPT functional tests

Marionette is also indirectly tested through geckodriver with WPT (Wd on Treeherder). To run them:

% ./mach wpt testing/web-platform/tests/webdriver

WPT tests conformance to the WebDriver standard and uses geckodriver. Together with the Marionette remote protocol in Gecko, they make up Mozilla’s WebDriver implementation.

This command supports a --webdriver-arg='-vv' argument that enables more detailed logging, as well as --jsdebugger for opening the Browser Toolbox.

A particularly useful trick is to combine this with the headless mode for Firefox we learned about earlier:

% MOZ_HEADLESS=1 ./mach wpt --webdriver-arg='-vv' testing/web-platform/tests/webdriver

Harness tests

The Marionette harness Python package has a set of mock-styled unit tests that uses the pytest framework. The following command will run all tests:

% ./mach python-test testing/marionette

To run a specific test specify the full path to the module:

% ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py

One-click loaners

Additionally, for debugging hard-to-reproduce test failures in CI, one-click loaners from <Taskcluster.html> can be particularly useful.

Out-of-tree testing

All the above examples show tests running in-tree, with a local checkout of central and a local build of Firefox. It is also possibly to run the Marionette tests without a local build and with a downloaded test archive from <Taskcluster.html>.

If you want to run tests from a downloaded test archive, you will need to download the target.common.tests.tar.gz artifact attached to Treeherder build jobs B for your system. Extract the archive and set up the Python Marionette client and harness by executing the following command in a virtual environment:

% pip install -r config/marionette_requirements.txt

The tests can then be found under marionette/tests/testing/marionette/harness/marionette_harness/tests and can be executed with the command marionette. It supports the same options as described above for mach.