Using third-party Python packages¶
Mach and its associated commands have a variety of 3rd-party Python dependencies. Many of these
are vendored in
third_party/python, while others are installed at runtime via
The dependencies of Mach itself can be found at
python/sites/mach.txt. Mach commands
may have additional dependencies which are specified at
For example, the following Mach command would have its 3rd-party dependencies declared at
The format of
<site>.txt files are documented further in the
Adding a Python package¶
There’s two ways of using 3rd-party Python dependencies:
pip install the packages. Python dependencies with native code must be installed using
pip. This is the recommended technique for adding new Python dependencies.
Vendor the source of the Python package in-tree. Dependencies of the Mach core logic or of building Firefox itself must be vendored.
For dependencies that meet both restrictions (dependency of Mach/build, and has native code), see the Mach/Build Native 3rd-party Dependencies section below.
pip install the package¶
To add a
pip install-d package dependency, add it to your site’s
python/sites/<site>.txt manifest file:
Some tasks are not permitted to use external resources, and for those we can publish packages to an internal PyPI mirror. See how to upload to internal PyPI for more details.
Vendoring Python packages¶
To vendor a Python package, add it to
and then run
mach vendor python. This will update the tree of pinned
third_party/python/requirements.txt and download them all
Next, add that package and any new transitive dependencies (you’ll see them added in
third_party/python/requirements.txt) to the associated site’s dependency manifest in
The following policy applies to ALL vendored packages:
Vendored PyPI libraries MUST NOT be modified
Vendored libraries SHOULD be released copies of libraries available on PyPI.
When considering manually vendoring a package, discuss the situation with the
#buildteam to ensure that other, more maintainable options are exhausted.
We require that it is possible to build Firefox using only a checkout of the source, without depending on a package index. This ensures that building Firefox is deterministic and dependable, avoids packages from changing out from under us, and means we’re not affected when 3rd party services are offline. We don’t want a DoS against PyPI or a random package maintainer removing an old tarball to delay a Firefox chemspill. Therefore, packages required by Mach core logic or for building Firefox itself must be vendored.
Mach/Build Native 3rd-party Dependencies¶
There are cases where Firefox is built without being able to
pip install, but where
native 3rd party Python dependencies enable optional functionality. This can’t be solved
by vendoring the platform-specific libraries, as then each one would have to be stored
multiple times in-tree according to how many platforms we wish to support.
Instead, this is solved by pre-installing such native packages onto the host system in advance, then having Mach attempt to use such packages directly from the system. This feature is only viable in very specific environments, as the system Python packages have to be compatible with Mach’s vendored packages.
To control this behaviour, the
variable can be used:
Mach will perform the build using only vendored packages. No Python virtual environment will be created for Mach.
Mach will use the host system’s Python packages as part of doing the build. This option
allows the usage of native Python packages without leaning on a
Same behaviour as
There’s a couple restrictions here:
MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCEonly applies to the top-level
"common"site and the
"build"site. All other sites will use
pip installat run-time as needed.
MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"is not allowed when using any site other than
As described in Package compatibility below, packages used by Mach are still in scope when commands are run, and
The host system is practically guaranteed to be incompatible with commands’ dependency lockfiles.
MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE environment variable fits into the following use
Mozilla CI Builds¶
We need access to the native packages of
psutil to extract archives and
get OS information respectively. Use
Mozilla CI non-Build Tasks¶
We generally don’t want to create a Mach virtual environment to avoid redundant processing,
but it’s ok to
pip install for specific command sites as needed, so leave
MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE unset (
MOZ_AUTOMATION implies the default
In cases where native packages are needed by Mach, use
Downstream CI Builds¶
Sometimes these builds happen in sandboxed, network-less environments, and usually these builds
don’t need any of the behaviour enabled by installing native Python dependencies.
When installing Firefox via the package manager, Gentoo generally builds it from source rather than
distributing a compiled binary artifact. Accordingly, users doing a build of Firefox in this
context don’t want stray files created in
~/.mozbuild or unnecessary
pip install calls.
MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE unset so that all Mach commands can be run,
Python dependency lockfiles are respected, and optional behaviour is enabled by installing
Mach requires that all commands’ package requirements be compatible with those of Mach itself. (This is because functions and state created by Mach are still usable from within the commands, and they may still need access to their associated 3rd-party modules).
However, it is OK for Mach commands to have package requirements which are incompatible with each other. This allows the flexibility for some Mach commands to depend on modern dependencies while other, more mature commands may still only be compatible with a much older version.
Only one version of a package may be vendored at any given time. If two Mach commands need to
have conflicting packages, then at least one of them must
pip install the package instead
If a Mach command’s dependency conflicts with a vendored package, and that vendored package
isn’t needed by Mach itself, then that vendored dependency should be moved from
python/sites/mach.txt to its associated environment.