Automated Testing

You’ve just written a feature and (hopefully!) want to test it. Or you’ve decided that an existing feature doesn’t have enough tests and want to contribute some. But where do you start? You’ve looked around and found references to things like “xpcshell” or “web-platform-tests” or “talos”. What code, features or platforms do they all test? Where do their feature sets overlap? In short, where should your new tests go? This document is a starting point for those who want to start to learn about Mozilla’s automated testing tools and procedures. Below you’ll find a short summary of each framework we use, and some questions to help you pick the framework(s) you need for your purposes.

If you still have questions, ask on Matrix or on the relevant bug.

Firefox Production

These tests are found within the mozilla-central tree, along with the product code.

They are run when a changeset is pushed to mozilla-central, autoland, or try, with the results showing up on Treeherder. Not all tests will be run on every changeset; alogrithms are put in place to run the most likely failures, with all tests being run on a regular basis.

They can also be run on local builds. Note: Most of the mobile tests run on emulators, but some of the tests (notably, performance tests) run on hardware devices. We try to avoid running mobile tests on hardware devices unnecessarily. In Treeherder, tests with names that start with “hw-” run on hardware.

Linting

Lint tests help to ensure better quality, less error-prone code by analysing the code with a linter.

Linters

Treeherder Symbol

Name

Platforms

What is Tested

ES

ESLint

All

JavaScript is analyzed for correctness.

ES-build

eslint-build

All

Extended javascript analysis that uses build artifacts.

mocha(EPM)

ESLint-plugin-mozilla

Desktop

The ESLint plugin rules.

f8

flake8

All

Python analyzed for style and correctness.

stylelint

Stylelint

All

CSS is analyzed for correctness.

W

wpt lint

Desktop

web-platform-tests analyzed for style and manifest correctness

WR(tidy)

WebRender servo-tidy

Desktop

Code in gfx/wr is run through servo-tidy.

A

Spotless

Android

Java is analyzed for style and correctness.

Functional testing

Automated Test Suites

Treeherder Symbol

Name

Platform

Process

Environment

Privilege

What is Tested

Shell

Browser Profile

R(J)

JS Reftest

Desktop

N/A

JSShell

N/A

N/A

The JavaScript engine’s implementation of the JavaScript language.

R(C)

Crashtest

All

Child

Content

Yes

Low

That pages load without crashing, asserting, or leaking.

R(R)

Reftest

All

Child

Content

Yes

Low

That pages are rendered (and thus also layed out) correctly.

GTest

GTest

All

N/A

Terminal

N/A

N/A

Code that is not exposed to JavaScript.

X

xpcshell

All

Parent, Allow

XPCShell

Allow

High

Low-level code exposed to JavaScript, such as XPCOM components.

M(a11y)

Accessibility (mochitest-a11y)

Desktop

Child

Content

Yes

?

Accessibility interfaces.

M(1), M(2), M(...)

Mochitest plain

All

Child

Content

Yes

Low, Allow

Features exposed to JavaScript in web content, like DOM and other Web APIs, where the APIs do not require elevated permissions to test.

M(c1/c2/c3)

Mochitest chrome

All

Child, Allow

Content

Yes

High

Code requiring UI or JavaScript interactions with privileged objects.

M(bc)

Mochitest browser-chrome

All

Parent, Allow

Browser

Yes

High

How the browser UI interacts with itself and with content.

M(remote)

Mochitest Remote Protocol

All

Parent, Allow

Browser

Yes

High

Firefox Remote Protocol (Implements parts of Chrome dev-tools protocol). Based on Mochitest browser-chrome.

SM(...), SM(pkg)

SpiderMonkey automation

Desktop

N/A

JSShell

N/A

Low

SpiderMonkey engine shell tests and JSAPI tests.

W

web-platform-tests

Desktop

Child

Content

Yes

Low

Standardized features exposed to ECMAScript in web content; tests are shared with other vendors.

Wr

web-platform-tests

All

Child

Content

Yes

Low

Layout and graphic correctness for standardized features; tests are shared with other vendors.

Mn

Marionette

Desktop

?

Content, Browser

?

High

Large out-of-process function integration tests and tests that do communication with multiple remote Gecko processes.

Fxfn

Firefox UI Tests

Desktop

?

Content, Browser

Yes

High

Integration tests with a focus on the user interface and localization.

tt(c)

telemetry-tests-client

Desktop

N/A

Content, Browser

Yes

High

Integration tests for the Firefox Telemetry client.

TV

Test Verification (test-verify)

All

Depends on test harness

?

?

?

Uses other test harnesses - mochitest, reftest, xpcshell - to perform extra testing on new/modified tests.

TVw

Test Verification for wpt (test-verify-wpt)

Desktop

Child

?

?

?

Uses wpt test harnesses to perform extra testing on new/modified web-platform tests.

WR(wrench)

WebRender standalone tests

All

N/A

Terminal

N/A

N/A

WebRender rust code (as a standalone module, with Gecko integration).

Note: there are preference-based variations of the previous testing suites. For example, mochitests on Treeherder can have gli, swr, spi, nofis, a11y-checks, spi-nw-1proc, and many others. Another example is GTest, which can use GTest-1proc. To learn more about these variations, you can mouse hover over these items to read a description of what these abbreviations mean.

Table key

Symbol

Abbreviation for the test suite used by Treeherder. The first letter generally indicates which of the test harnesses is used to execute the test. The letter in parentheses identifies the actual test suite.

Name

Common name used when referring to the test suite.

File type

When adding a new test, you will generally create a file of this type in the source tree and then declare it in a manifest or makefile.

Platform

Most test suites are supported only on a subset of the available plaforms and operating systems. Unless otherwise noted:

  • Desktop tests run on Windows, Mac OS X, and Linux.

  • Mobile tests run on Android emulators or remotely on Android devices.

Process

  • When Parent is indicated, the test file will always run in the parent process, even when the browser is running in Electrolysis (e10s) mode.

  • When Child is indicated, the test file will run in the child process when the browser is running in Electrolysis (e10s) mode.

  • The Allow label indicates that the test has access to mechanisms to run code in the other process.

Environment

  • The JSShell and XPCShell environments are limited JavaScript execution environments with no windows or user interface (note however that XPCShell tests on Android are run within a browser window.)

  • The Content indication means that the test is run inside a content page loaded by a browser window.

  • The Browser indication means that the test is loaded in the JavaScript context of a browser XUL window.

  • The Browser Profile column indicates whether a browser profile is loaded when the test starts. The Allow label means that the test can optionally load a profile using a special command.

Privilege

Indicates whether the tests normally run with low (content) or high (chrome) JavaScript privileges. The Allow label means that the test can optionally run code in a privileged environment using a special command.

Performance testing

There are many test harnesses used to test performance. For more information on the various performance harnesses, check out the perf docs.

So which should I use?

Generally, you should pick the lowest-level framework that you can. If you are testing JavaScript but don’t need a window, use XPCShell or even JSShell. If you’re testing page layout, try to use web-platform-test reftest. The advantage in lower level testing is that you don’t drag in a lot of other components that might have their own problems, so you can home in quickly on any bugs in what you are specifically testing.

Here’s a series of questions to ask about your work when you want to write some tests.

Is it low-level code?

If the functionality is exposed to JavaScript, and you don’t need a window, consider XPCShell. If not, you’ll probably have to use GTest, which can test pretty much anything. In general, this should be your last option for a new test, unless you have to test something that is not exposed to JavaScript.

Does it cause a crash?

If you’ve found pages that crash Firefox, add a crashtest to make sure future versions don’t experience this crash (assertion or leak) again. Note that this may lead to more tests once the core problem is found.

Is it a layout/graphics feature?

Reftest is your best bet, if possible.

Do you need to verify performance?

Use an appropriate performance test suite from this list.

Are you testing UI features?

Try one of the flavors of mochitest, or Marionette if the application also needs to be restarted, or tested with localized builds.

Are you testing Mobile/Android?

If you are testing GeckoView, you will need to need to use JUnit integration tests.

There are some specific features that Mochitest or Reftest can cover. Browser-chrome tests do not run on Android. If you want to test performance, Raptor will be a good choice.

Are you doing none of the above?

Need to get more data out of your tests?

Most test jobs now expose an environment variable named $MOZ_UPLOAD_DIR. If this variable is set during automated test runs, you can drop additional files into this directory, and they will be uploaded to a web server when the test finishes. The URLs to retrieve the files will be output in the test log.

Passing $MOZ_RECORD_TEST=1 as an environment variable when running some tests (e.g. mochitests) on Linux platform will trigger a recording of the desktop with GNOME Screencast.

Need to set preferences for test-suites?

First ask yourself if these prefs need to be enabled for all tests or just a subset of tests (e.g to enable a feature).

Setting prefs that only apply to certain tests

If the answer is the latter, try to set the pref as local to the tests that need it as possible. Here are some options:

  • If the test runs in chrome scope (e.g mochitest chrome or browser-chrome), you can use Services.prefs to set the prefs in your test’s setup function. Be sure to reset the pref back to its original value during teardown!

  • Mochitest plain tests can use SpecialPowers to set prefs.

  • All variants of mochitest can set prefs in their manifests. For example, to set a pref for all tests in a manifest:

    [DEFAULT]
prefs =
  my.awesome.pref=foo,
  my.other.awesome.pref=bar,
[test_foo.js]
[test_bar.js]

  • All variants of reftest can also set prefs in their manifests.

  • All variants of web-platform-tests can also set prefs in their manifests.

Setting prefs that apply to the entire suite

Most test suites define prefs in user.js files that live under testing/profiles. Each directory is a profile that contains a user.js file with a number of prefs defined in it. Test suites will then merge one or more of these basic profiles into their own profile at runtime. To see which profiles apply to which test suites, you can inspect testing/profiles/profiles.json. Profiles at the beginning of the list get overridden by profiles at the end of the list.

Because this system makes it hard to get an overall view of which profiles are set for any given test suite, a handy profile utility was created:

$ cd testing/profiles
$ ./profile -- --help
usage: profile [-h] {diff,sort,show,rm} ...
$ ./profile show mochitest          # prints all prefs that will be set in mochitest
$ ./profile diff mochitest reftest  # prints differences between the mochitest and reftest suites

Note: JS engine tests do not use testing/profiles yet, instead set prefs here.

Adding New Context to Skip Conditions

Often when standing up new test configurations, it’s necessary to add new keys that can be used in skip-if annotations.