moz.build files are the mechanism by which tree metadata (notably
the build configuration) is defined.
Directories in the tree contain
moz.build files which declare
functionality for their respective part of the tree. This includes
things such as the list of C++ files to compile, where to find tests,
moz.build files are actually Python scripts. However, their
execution is governed by special rules. This is explained below.
moz.build Python Sandbox¶
As mentioned above,
moz.build files are Python scripts. However,
they are executed in a special Python sandbox that significantly
changes and limits the execution environment. The environment is so
different, it’s doubtful most
moz.build files would execute without
error if executed by a vanilla Python interpreter (e.g.
The following properties make execution of
moz.build files special:
- The execution environment exposes a limited subset of Python.
- There is a special set of global symbols and an enforced naming convention of symbols.
- Some symbols are inherited from previously-executed
The limited subset of Python is actually an extremely limited subset.
Only a few symbols from
__builtin__ are exposed. These include
open aren’t available.
moz.build files can do very little. This is by design.
The execution sandbox treats all
UPPERCASE variables specially. Any
UPPERCASE variable must be known to the sandbox before the script
executes. Any attempt to read or write to an unknown
variable will result in an exception being raised. Furthermore, the
types of all
UPPERCASE variables is strictly enforced. Attempts to
assign an incompatible type to an
UPPERCASE variable will result in
an exception being raised.
The strictness of behavior with
UPPERCASE variables is a very
intentional design decision. By ensuring strict behavior, any operation
UPPERCASE variable is guaranteed to have well-defined
side-effects. Previously, when the build configuration was defined in
Makefiles, assignments to variables that did nothing would go
moz.build files fix this problem by eliminating the
potential for false promises.
moz.build file has completed execution, only the
UPPERCASE variables are used to retrieve state.
The set of variables and functions available to the Python sandbox is
defined by the
mozbuild.frontend.context module. The
data structures in this module are consumed by the
mozbuild.frontend.reader.MozbuildSandbox class to construct
the sandbox. There are tests to ensure that the set of symbols exposed
to an empty sandbox are all defined in the
This module also contains documentation for each symbol, so nothing can
sneak into the sandbox without being explicitly defined and documented.
Reading and Traversing moz.build Files¶
The process for reading
moz.build files roughly consists of:
- Start at the root
- Evaluate the
moz.buildfile in a new sandbox.
- Emit the main context and any sub-contexts from the executed sandbox.
- Extract a set of
moz.buildfiles to execute next.
- For each additional
moz.buildfile, goto #2 and repeat until all referenced files have executed.
From the perspective of the consumer, the output of reading is a stream
mozbuild.frontend.reader.context.Context instances. Each
Context defines a particular aspect of data. Consumers iterate over
these objects and do something with the data inside. Each object is
essentially a dictionary of all the
UPPERCASE variables populated
during its execution.
Historically, there was only one
As the number of things tracked by
moz.build files grew and more
and more complex processing was desired, it was necessary to split these
contexts into multiple logical parts. It is now common to emit
multiple contexts per
Build System Reading Mode¶
The traditional mode of evaluation of
moz.build files is what’s
called build system traversal mode. In this mode, the
variable in each
moz.build sandbox is populated from data coming
config.status, which is produced by
moz.build files often make decisions conditional
on the state of the build configuration. e.g. only compile foo.cpp if
feature X is enabled.
In this mode, traversal of
moz.build files is governed by variables
TEST_DIRS. For example, to execute a child
foo, you would add
DIRS += ['foo'] to a
foo/moz.build would be evaluated.
Filesystem Reading Mode¶
There is an alternative reading mode that doesn’t involve the build
system and doesn’t use
DIRS variables to control traversal into
child directories. This mode is called filesystem reading mode.
In this reading mode, the
CONFIG variable is a dummy, mostly empty
object. Accessing all but a few special variables will return an empty
value. This means that nearly all
if CONFIG['FOO']: branches will
not be taken.
Instead of using content from within the evaluated
file to drive traversal into subsequent
moz.build files, the set
of files to evaluate is controlled by the thing doing the reading.
moz.build file is not guaranteed to be executable in
isolation. Instead, we must evaluate all parent
first. For example, in order to evaluate
/foo/moz.build, one must
/moz.build and have its state influence the execution of
Filesystem reading mode is utilized to power the Files Metadata feature.
The code for reading
moz.build files lives in
mozbuild.frontend.reader. The Python sandboxes evaluation results
mozbuild.frontend.context.Context) are passed into
mozbuild.frontend.emitter, which converts them to classes defined
mozbuild.frontend.data. Each class in this module defines a
domain-specific component of tree metadata. e.g. there will be separate
manifests. This means downstream consumers of this data can filter on class
types to only consume what they are interested in.
There is no well-defined mapping between
moz.build file instances
and the number of
mozbuild.frontend.data classes derived from
each. Depending on the content of the
moz.build file, there may be 1
object derived or 100.
The purpose of the
emitter layer between low-level sandbox execution
and metadata representation is to facilitate a unified normalization and
verification step. There are multiple downstream consumers of the
moz.build-derived data and many will perform the same actions. This
logic can be complicated, so we have a component dedicated to it.
mozbuild.frontend.reader.TreeMetadataEmitter` have a
stream-based API courtesy of generators. When you hook them up properly,
mozbuild.frontend.data classes are emitted before all
moz.build files have been read. This means that downstream errors
are raised soon after sandbox execution.
Lots of the code for evaluating Python sandboxes is applicable to non-Mozilla systems. In theory, it could be extracted into a standalone and generic package. However, until there is a need, there will likely be some tightly coupled bits.