mozbuild.frontend package

Submodules

mozbuild.frontend.context module

This module contains the data structure (context) holding the configuration from a moz.build. The data emitted by the frontend derives from those contexts.

It also defines the set of variables and functions available in moz.build. If you are looking for the absolute authority on what moz.build files can contain, you’ve come to the right place.

class mozbuild.frontend.context.AbsolutePath(context, value=None)

Bases: Path

Like Path, but allows arbitrary paths outside the source and object directories.

class mozbuild.frontend.context.AsmFlags(context): Bases: BaseCompileFlags

class mozbuild.frontend.context.BaseCompileFlags(context): Bases: ContextDerivedValue, dict

class mozbuild.frontend.context.CompileFlags(context): Bases: TargetCompileFlags

class mozbuild.frontend.context.Context(allowed_variables={}, config=None, finder=None)

Bases: KeyedDefaultDict

Represents a moz.build configuration context.

Instances of this class are filled by the execution of sandboxes. At the core, a Context is a dict, with a defined set of possible keys we’ll call variables. Each variable is associated with a type.

When reading a value for a given key, we first try to read the existing value. If a value is not found and it is defined in the allowed variables set, we return a new instance of the class for that variable. We don’t assign default instances until they are accessed because this makes debugging the end-result much simpler. Instead of a data structure with lots of empty/default values, you have a data structure with only the values that were read or touched.

Instances of variables classes are created by invoking class_name(), except when class_name derives from ContextDerivedValue or SubContext, in which case class_name(instance_of_the_context) or class_name(self) is invoked. A value is added to those calls when instances are created during assignment (setitem).

allowed_variables is a dict of the variables that can be set and read in this context instance. Keys in this dict are the strings representing keys in this context which are valid. Values are tuples of stored type, assigned type, default value, a docstring describing the purpose of the variable, and a tier indicator (see comment above the VARIABLES declaration in this module).

config is the ConfigEnvironment for this context.

add_source(path): Adds the given path as source of the data from this context.

property all_paths: Returns all paths ever added to the context.

property error_is_fatal: Returns True if the error function should be fatal.

objdir: A specialized version of the memoize decorator that works for class instance properties.

pop_source(): Get back to the previous current path for the context.

push_source(path): Adds the given path as source of the data from this context and make it the current path for the context.

relobjdir: A specialized version of the memoize decorator that works for class instance properties.

property relsrcdir

property source_stack: Returns the current stack of pushed sources.

property srcdir

update(iterable={}, **kwargs)

Like dict.update(), but using the context’s setitem.

This function is transactional: if setitem fails for one of the values, the context is not updated at all.

mozbuild.frontend.context.ContextDerivedTypedHierarchicalStringList(type): Specialized HierarchicalStringList for use with ContextDerivedValue types.

mozbuild.frontend.context.ContextDerivedTypedList(klass, base_class=<class 'mozbuild.util.List'>): Specialized TypedList for use with ContextDerivedValue types.

mozbuild.frontend.context.ContextDerivedTypedListWithItems(type, base_class=<class 'mozbuild.util.List'>): Specialized TypedList for use with ContextDerivedValue types.

mozbuild.frontend.context.ContextDerivedTypedRecord(*fields)

Factory for objects with certain properties and dynamic type checks.

This API is extremely similar to the TypedNamedTuple API, except that properties may be mutated. This supports syntax like:

VARIABLE_NAME.property += [
  'item1',
  'item2',
]

class mozbuild.frontend.context.ContextDerivedValue

Bases: object

Classes deriving from this one receive a special treatment in a Context. See Context documentation.

mozbuild.frontend.context.Enum(*values)

class mozbuild.frontend.context.Files(parent, *patterns)

Bases: SubContext

Metadata attached to files.

It is common to want to annotate files with metadata, such as which Bugzilla component tracks issues with certain files. This sub-context is where we stick that metadata.

The argument to this sub-context is a file matching pattern that is applied against the host file’s directory. If the pattern matches a file whose info is currently being sought, the metadata attached to this instance will be applied to that file.

Patterns are collections of filename characters with / used as the directory separate (UNIX-style paths) and * and ** used to denote wildcard matching.

Patterns without the * character are literal matches and will match at most one entity.

Patterns with * or ** are wildcard matches. * matches files at least within a single directory. ** matches files across several directories.

foo.html: Will match only the foo.html file in the current directory.
*.mjs: Will match all .mjs files in the current directory.
**/*.cpp: Will match all .cpp files in this and all child directories.
foo/*.css: Will match all .css files in the foo/ directory.
bar/*: Will match all files in the bar/ directory and all of its children directories.
bar/**: This is equivalent to bar/* above.
bar/**/foo: Will match all foo files in the bar/ directory and all of its children directories.

The difference in behavior between * and ** is only evident if a pattern follows the * or **. A pattern ending with * is greedy. ** is needed when you need an additional pattern after the wildcard. e.g. **/foo.

VARIABLES = {'BUG_COMPONENT': (<class 'mozbuild.util.TypedNamedTuple.<locals>.TypedTuple'>, <class 'tuple'>, "The bug component that tracks changes to these files.\n\n Values are a 2-tuple of unicode describing the Bugzilla product and\n component. e.g. ``('Firefox Build System', 'General')``.\n "), 'FINAL': (<class 'bool'>, <class 'bool'>, 'Mark variable assignments as finalized.\n\n During normal processing, values from newer Files contexts\n overwrite previously set values. Last write wins. This behavior is\n not always desired. ``FINAL`` provides a mechanism to prevent\n further updates to a variable.\n\n When ``FINAL`` is set, the value of all variables defined in this\n context are marked as frozen and all subsequent writes to them\n are ignored during metadata reading.\n\n See :ref:`mozbuild_files_metadata_finalizing` for more info.\n '), 'SCHEDULES': (<class 'mozbuild.frontend.context.Schedules'>, <class 'list'>, 'Maps source files to the CI tasks that should be scheduled when\n they change. The tasks are grouped by named components, and those\n names appear again in the taskgraph configuration\n `($topsrcdir/taskgraph/).\n\n Some components are "inclusive", meaning that changes to most files\n do not schedule them, aside from those described in a Files\n subcontext. For example, py-lint tasks need not be scheduled for\n most changes, but should be scheduled when any Python file changes.\n Such components are named by appending to `SCHEDULES.inclusive`:\n\n with Files(\'**.py\'):\n SCHEDULES.inclusive += [\'py-lint\']\n\n Other components are \'exclusive\', meaning that changes to most\n files schedule them, but some files affect only one or two\n components. For example, most files schedule builds and tests of\n Firefox for Android, OS X, Windows, and Linux, but files under\n `mobile/android/` affect Android builds and tests exclusively, so\n builds for other operating systems are not needed. Test suites\n provide another example: most files schedule reftests, but changes\n to reftest scripts need only schedule reftests and no other suites.\n\n Exclusive components are named by setting `SCHEDULES.exclusive`:\n\n with Files(\'mobile/android/**\'):\n SCHEDULES.exclusive = [\'android\']\n ')}

static aggregate(files)

Given a mapping of path to Files, obtain aggregate results.

Consumers may want to extract useful information from a collection of Files describing paths. e.g. given the files info data for N paths, recommend a single bug component based on the most frequent one. This function provides logic for deriving aggregate knowledge from a collection of path File metadata.

Note: the intent of this function is to operate on the result of mozbuild.frontend.reader.BuildReader.files_info(). The mozbuild.frontend.context.Files() instances passed in are thus the “collapsed” (__iadd__``ed) results of all ``Files from all moz.build files relevant to a specific path, not individual Files instances from a single moz.build file.

asdict()

Return this instance as a dict with built-in data structures.

Call this to obtain an object suitable for serializing.

class mozbuild.frontend.context.FinalTargetValue(context, value=''): Bases: ContextDerivedValue, str

class mozbuild.frontend.context.HostCompileFlags(context): Bases: BaseCompileFlags

class mozbuild.frontend.context.InitializedDefines(context, value=None)

Bases: ContextDerivedValue, OrderedDict

update([E, ]**F) → None. Update D from dict/iterable E and F.: If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

class mozbuild.frontend.context.LinkFlags(context): Bases: BaseCompileFlags

mozbuild.frontend.context.ManifestparserManifestList: alias of _OrderedListWithAction

class mozbuild.frontend.context.ObjDirPath(context, value=None)

Bases: Path

Like Path, but limited to paths in the object directory.

mozbuild.frontend.context.OrderedPathListWithAction(action)

Returns a class which behaves as a StrictOrderingOnAppendList, but invokes the given callable with each input and a context as it is read, storing a tuple including the result and the original item.

This used to extend moz.build reading to make more data available in filesystem-reading mode.

class mozbuild.frontend.context.Path(context, value=None)

Bases: ContextDerivedValue, str

Stores and resolves a source path relative to a given context

This class is used as a backing type for some of the sandbox variables. It expresses paths relative to a context. Supported paths are:

‘/topsrcdir/relative/paths’

‘srcdir/relative/paths’

‘!/topobjdir/relative/paths’

‘!objdir/relative/paths’

‘%/filesystem/absolute/paths’

join(*p): ContextDerived equivalent of mozpath.join(self, *p), returning a new Path instance.

target_basename: A specialized version of the memoize decorator that works for class instance properties.

class mozbuild.frontend.context.PathMeta

Bases: type

Meta class for the Path family of classes.

It handles calling __new__ with the right arguments in cases where a Path is instantiated with another instance of Path instead of having received a context.

It also makes Path(context, value) instantiate one of the subclasses depending on the value, allowing callers to do standard type checking (isinstance(path, ObjDirPath)) instead of checking the value itself (path.startswith(‘!’)).

mozbuild.frontend.context.ReftestManifestList: alias of _OrderedListWithAction

class mozbuild.frontend.context.RenamedSourcePath(context, value=None)

Bases: SourcePath

Like SourcePath, but with a different base name when installed.

The constructor takes a tuple of (source, target_basename).

This class is not meant to be exposed to moz.build sandboxes as of now, and is not supported by the RecursiveMake backend.

property target_basename

class mozbuild.frontend.context.Schedules(inclusive=None, exclusive=None)

Bases: object

Similar to a ContextDerivedTypedRecord, but with different behavior for the properties:

VAR.inclusive can only be appended to (+=), and can only contain values from mozbuild.schedules.INCLUSIVE_COMPONENTS

VAR.exclusive can only be assigned to (no +=), and can only contain values from mozbuild.schedules.ALL_COMPONENTS

property components

property exclusive

property inclusive

mozbuild.frontend.context.SchedulingComponents: alias of _TypedRecord

class mozbuild.frontend.context.SourcePath(context, value=None)

Bases: Path

Like Path, but limited to paths in the source directory.

translated: A specialized version of the memoize decorator that works for class instance properties.

class mozbuild.frontend.context.SubContext(parent)

Bases: Context, ContextDerivedValue

A Context derived from another Context.

Sub-contexts are intended to be used as context managers.

Sub-contexts inherit paths and other relevant state from the parent context.

class mozbuild.frontend.context.TargetCompileFlags(context)

Bases: BaseCompileFlags

Base class that encapsulates some common logic between CompileFlags and WasmCompileFlags.

class mozbuild.frontend.context.TemplateContext(template=None, allowed_variables={}, config=None): Bases: Context

class mozbuild.frontend.context.WasmFlags(context): Bases: TargetCompileFlags

mozbuild.frontend.context.cls: alias of Files

mozbuild.frontend.data module

Data structures representing Mozilla’s source tree.

The frontend files are parsed into static data structures. These data structures are defined in this module.

All data structures of interest are children of the TreeMetadata class.

Logic for populating these data structures is not defined in this class. Instead, what we have here are dumb container classes. The emitter module contains the code for converting executed mozbuild files into these data structures.

class mozbuild.frontend.data.BaseConfigSubstitution(context)

Bases: ContextDerived

Base class describing autogenerated files as part of config.status.

input_path

output_path

relpath

class mozbuild.frontend.data.BaseDefines(context, defines)

Bases: ContextDerived

Context derived container object for DEFINES/HOST_DEFINES, which are OrderedDicts.

defines

get_defines()

update(more_defines)

class mozbuild.frontend.data.BaseLibrary(context, basename)

Bases: Linkable

Generic context derived container object for libraries.

basename

import_name

property import_path

lib_name

property name

refs

class mozbuild.frontend.data.BaseProgram(context, program, is_unit_test=False)

Bases: Linkable

Context derived container object for programs, which is a unicode string.

This class handles automatically appending a binary suffix to the program name. If the suffix is not defined, the program name is unchanged. Otherwise, if the program name ends with the given suffix, it is unchanged Otherwise, the suffix is appended to the program name.

DICT_ATTRS = {'KIND', 'install_target', 'program', 'relobjdir'}

property name

program

class mozbuild.frontend.data.BaseRustLibrary

Bases: object

property import_path

init(context, basename, cargo_file, crate_type, dependencies, features, is_gkrust)

slots = ('cargo_file', 'crate_type', 'dependencies', 'features', 'output_category', 'is_gkrust')

class mozbuild.frontend.data.BaseRustProgram(context, name, cargo_file)

Bases: Linkable

KIND

SUFFIX_VAR

TARGET_SUBST_VAR

cargo_file

location

name

class mozbuild.frontend.data.BaseSources(context, static_files, generated_files, canonical_suffix)

Bases: ContextDerived

Base class for files to be compiled during the build.

canonical_suffix

files

generated_files

static_files

class mozbuild.frontend.data.ChromeManifestEntry(context, manifest_path, entry)

Bases: ContextDerived

Represents a chrome.manifest entry.

entry

path

class mozbuild.frontend.data.ComputedFlags(context, reader_flags)

Bases: ContextDerived

Aggregate flags for consumption by various backends.

flags

get_flags()

resolve_flags(key, value)

class mozbuild.frontend.data.ConfigFileSubstitution(context)

Bases: BaseConfigSubstitution

Describes a config file that will be generated using substitutions.

input_path

output_path

relpath

class mozbuild.frontend.data.ContextDerived(context)

Bases: TreeMetadata

Build object derived from a single Context instance.

It holds fields common to all context derived classes. This class is likely never instantiated directly but is instead derived from.

config

context_all_paths

context_main_path

property defines

property install_target

property installed

objdir

property relobjdir

relsrcdir

srcdir

topobjdir

topsrcdir

class mozbuild.frontend.data.Defines(context, defines)

Bases: BaseDefines

defines

class mozbuild.frontend.data.DirectoryTraversal(context)

Bases: ContextDerived

Describes how directory traversal for building should work.

This build object is likely only of interest to the recursive make backend. Other build backends should (ideally) not attempt to mimic the behavior of the recursive make backend. The only reason this exists is to support the existing recursive make backend while the transition to mozbuild frontend files is complete and we move to a more optimal build backend.

Fields in this class correspond to similarly named variables in the frontend files.

dirs

class mozbuild.frontend.data.Exports(sandbox, files)

Bases: FinalTargetFiles

Context derived container object for EXPORTS, which is a HierarchicalStringList.

We need an object derived from ContextDerived for use in the backend, so this object fills that role. It just has a reference to the underlying HierarchicalStringList, which is created when parsing EXPORTS.

files

property install_target

class mozbuild.frontend.data.ExternalLibrary

Bases: object

Empty mixin for libraries built by an external build system.

class mozbuild.frontend.data.ExternalSharedLibrary(context, basename, real_name=None, soname=None, variant=None, symbols_file=False)

Bases: SharedLibrary, ExternalLibrary

Context derived container for shared libraries built by an external build system.

output_category

soname

symbols_file

symbols_link_arg

variant

class mozbuild.frontend.data.ExternalStaticLibrary(context, basename, real_name=None, link_into=None, no_expand_lib=False)

Bases: StaticLibrary, ExternalLibrary

Context derived container for static libraries built by an external build system.

link_into

no_expand_lib

class mozbuild.frontend.data.FinalTargetFiles(sandbox, files)

Bases: ContextDerived

Sandbox container object for FINAL_TARGET_FILES, which is a HierarchicalStringList.

We need an object derived from ContextDerived for use in the backend, so this object fills that role. It just has a reference to the underlying HierarchicalStringList, which is created when parsing FINAL_TARGET_FILES.

files

class mozbuild.frontend.data.FinalTargetPreprocessedFiles(sandbox, files)

Bases: ContextDerived

Sandbox container object for FINAL_TARGET_PP_FILES, which is a HierarchicalStringList.

We need an object derived from ContextDerived for use in the backend, so this object fills that role. It just has a reference to the underlying HierarchicalStringList, which is created when parsing FINAL_TARGET_PP_FILES.

files

class mozbuild.frontend.data.GeneratedFile(context, script, method, outputs, inputs, flags=(), localized=False, force=False, py2=False, required_during_compile=None)

Bases: ContextDerived

Represents a generated file.

flags

force

inputs

localized

method

outputs

py2

required_before_compile

required_before_export

required_during_compile

script

class mozbuild.frontend.data.HostDefines(context, defines)

Bases: BaseDefines

defines

class mozbuild.frontend.data.HostLibrary(context, basename)

Bases: HostMixin, BaseLibrary

Context derived container object for a host library

KIND = 'host'

basename

import_name

lib_name

no_expand_lib = False

refs

class mozbuild.frontend.data.HostMixin

Bases: object

property defines

class mozbuild.frontend.data.HostProgram(context, program, is_unit_test=False)

Bases: HostMixin, BaseProgram

Context derived container object for HOST_PROGRAM

KIND = 'host'

SUFFIX_VAR = 'HOST_BIN_SUFFIX'

property install_target

program

class mozbuild.frontend.data.HostRustLibrary(context, basename, cargo_file, crate_type, dependencies, features, is_gkrust)

Bases: BaseRustLibrary, HostLibrary

Context derived container object for a host rust library

FEATURES_VAR = 'HOST_RUST_LIBRARY_FEATURES'

KIND = 'host'

LIB_FILE_VAR = 'HOST_RUST_LIBRARY_FILE'

TARGET_SUBST_VAR = 'RUST_HOST_TARGET'

cargo_file

crate_type

dependencies

features

is_gkrust

no_expand_lib = True

output_category

class mozbuild.frontend.data.HostRustProgram(context, name, cargo_file)

Bases: BaseRustProgram

KIND

SUFFIX_VAR

TARGET_SUBST_VAR

cargo_file

location

name

class mozbuild.frontend.data.HostSharedLibrary(context, basename)

Bases: HostMixin, Library

Context derived container object for a host shared library.

This class supports less things than SharedLibrary does for target shared libraries. Currently has enough build system support to build the clang plugin.

KIND = 'host'

class mozbuild.frontend.data.HostSimpleProgram(context, program, is_unit_test=False)

Bases: HostMixin, BaseProgram

Context derived container object for each program in HOST_SIMPLE_PROGRAMS

KIND = 'host'

SUFFIX_VAR = 'HOST_BIN_SUFFIX'

program

source_files()

class mozbuild.frontend.data.HostSources(context, static_files, generated_files, canonical_suffix)

Bases: HostMixin, BaseSources

Represents files to be compiled for the host during the build.

canonical_suffix

files

generated_files

static_files

class mozbuild.frontend.data.IPDLCollection(context)

Bases: ContextDerived

Collects IPDL files during the build.

all_preprocessed_sources()

all_regular_sources()

all_source_files()

all_sources()

config

context_all_paths

context_main_path

objdir

relsrcdir

srcdir

topobjdir

topsrcdir

class mozbuild.frontend.data.InstallationTarget(context)

Bases: ContextDerived

Describes the rules that affect where files get installed to.

enabled

is_custom(): Returns whether or not the target is not derived from the default given xpiname and subdir.

subdir

target

xpiname

class mozbuild.frontend.data.JARManifest(context, path)

Bases: ContextDerived

Describes an individual JAR manifest file and how to process it.

This class isn’t very useful for optimizing backends yet because we don’t capture defines. We can’t capture defines safely until all of them are defined in moz.build and not Makefile.in files.

path

class mozbuild.frontend.data.Library(context, basename, real_name=None)

Bases: BaseLibrary

Context derived container object for a library

KIND = 'target'

class mozbuild.frontend.data.Linkable(context)

Bases: ContextDerived

Generic context derived container object for programs and libraries

cxx_link

lib_defines

link_library(obj)

link_system_library(lib)

linked_libraries

linked_system_libs

property objs

property output_path

source_files()

sources

exception mozbuild.frontend.data.LinkageWrongKindError

Bases: Exception

Error thrown when trying to link objects of the wrong kind

class mozbuild.frontend.data.LocalInclude(context, path)

Bases: ContextDerived

Describes an individual local include path.

path

class mozbuild.frontend.data.LocalizedFiles(sandbox, files)

Bases: FinalTargetFiles

Sandbox container object for LOCALIZED_FILES, which is a HierarchicalStringList.

files

class mozbuild.frontend.data.LocalizedPreprocessedFiles(sandbox, files)

Bases: FinalTargetPreprocessedFiles

Sandbox container object for LOCALIZED_PP_FILES, which is a HierarchicalStringList.

files

class mozbuild.frontend.data.ObjdirFiles(sandbox, files)

Bases: FinalTargetFiles

Sandbox container object for OBJDIR_FILES, which is a HierarchicalStringList.

files

property install_target

class mozbuild.frontend.data.ObjdirPreprocessedFiles(sandbox, files)

Bases: FinalTargetPreprocessedFiles

Sandbox container object for OBJDIR_PP_FILES, which is a HierarchicalStringList.

files

property install_target

class mozbuild.frontend.data.PerSourceFlag(context, file_name, flags)

Bases: ContextDerived

Describes compiler flags specified for individual source files.

file_name

flags

class mozbuild.frontend.data.PgoGenerateOnlySources(context, files)

Bases: BaseSources

Represents files to be compiled during the build.

These files are only used during the PGO generation phase.

canonical_suffix

files

generated_files

static_files

class mozbuild.frontend.data.Program(context, program, is_unit_test=False)

Bases: BaseProgram

Context derived container object for PROGRAM

KIND = 'target'

SUFFIX_VAR = 'BIN_SUFFIX'

program

class mozbuild.frontend.data.RustLibrary(context, basename, cargo_file, crate_type, dependencies, features, is_gkrust=False, link_into=None)

Bases: BaseRustLibrary, StaticLibrary

Context derived container object for a rust static library

FEATURES_VAR = 'RUST_LIBRARY_FEATURES'

KIND = 'target'

LIB_FILE_VAR = 'RUST_LIBRARY_FILE'

TARGET_SUBST_VAR = 'RUST_TARGET'

cargo_file

crate_type

dependencies

features

is_gkrust

output_category

class mozbuild.frontend.data.RustProgram(context, name, cargo_file)

Bases: BaseRustProgram

KIND

SUFFIX_VAR

TARGET_SUBST_VAR

cargo_file

location

name

class mozbuild.frontend.data.RustTests(context, names, features)

Bases: ContextDerived

features

names

output_category

class mozbuild.frontend.data.SandboxedWasmLibrary(context, basename, real_name=None)

Bases: Library

Context derived container object for a static sandboxed wasm library

KIND = 'wasm'

no_expand_lib = True

class mozbuild.frontend.data.SharedLibrary(context, basename, real_name=None, soname=None, variant=None, symbols_file=False)

Bases: Library

Context derived container object for a shared library

DICT_ATTRS = {'basename', 'import_name', 'install_target', 'lib_name', 'relobjdir', 'soname'}

FRAMEWORK = 1

MAX_VARIANT = 2

property import_path

output_category

soname

symbols_file

symbols_link_arg

variant

class mozbuild.frontend.data.SimpleProgram(context, program, is_unit_test=False)

Bases: BaseProgram

Context derived container object for each program in SIMPLE_PROGRAMS

KIND = 'target'

SUFFIX_VAR = 'BIN_SUFFIX'

program

source_files()

class mozbuild.frontend.data.Sources(context, static_files, generated_files, canonical_suffix)

Bases: BaseSources

Represents files to be compiled during the build.

canonical_suffix

files

generated_files

static_files

class mozbuild.frontend.data.StaticLibrary(context, basename, real_name=None, link_into=None, no_expand_lib=False)

Bases: Library

Context derived container object for a static library

link_into

no_expand_lib

class mozbuild.frontend.data.TestHarnessFiles(sandbox, files)

Bases: FinalTargetFiles

Sandbox container object for TEST_HARNESS_FILES, which is a HierarchicalStringList.

files

property install_target

class mozbuild.frontend.data.TestManifest(context, path, manifest, flavor=None, install_prefix=None, relpath=None, sources=(), dupe_manifest=False)

Bases: ContextDerived

Represents a manifest file containing information about tests.

deferred_installs

directory

dupe_manifest

external_installs

flavor

install_prefix

installs

manifest

manifest_obj_relpath

manifest_relpath

path

pattern_installs

source_relpaths

tests

class mozbuild.frontend.data.TreeMetadata

Bases: object

Base class for all data being captured.

to_dict()

class mozbuild.frontend.data.UnifiedSources(context, static_files, generated_files, canonical_suffix)

Bases: BaseSources

Represents files to be compiled in a unified fashion during the build.

have_unified_mapping

unified_source_mapping

class mozbuild.frontend.data.VariablePassthru(context)

Bases: ContextDerived

A dict of variables to pass through to backend.mk unaltered.

The purpose of this object is to facilitate rapid transitioning of variables from Makefile.in to moz.build. In the ideal world, this class does not exist and every variable has a richer class representing it. As long as we rely on this class, we lose the ability to have flexibility in our build backends since we will continue to be tied to our rules.mk.

variables

class mozbuild.frontend.data.WasmDefines(context, defines)

Bases: BaseDefines

defines

class mozbuild.frontend.data.WasmSources(context, static_files, generated_files, canonical_suffix)

Bases: BaseSources

Represents files to be compiled with the wasm compiler during the build.

canonical_suffix

files

generated_files

static_files

class mozbuild.frontend.data.WebIDLCollection(context)

Bases: ContextDerived

Collects WebIDL info referenced during the build.

all_basenames()

all_non_static_basenames()

all_non_static_sources()

all_preprocessed_sources()

all_regular_basenames()

all_regular_bindinggen_stems()

all_regular_cpp_basenames()

all_regular_sources()

all_regular_stems()

all_source_files()

all_sources()

all_static_sources()

all_stems()

all_test_basenames()

all_test_cpp_basenames()

all_test_sources()

all_test_stems()

config

context_all_paths

context_main_path

generated_events_basenames()

generated_events_stems()

objdir

relsrcdir

srcdir

topobjdir

topsrcdir

property unified_source_mapping

class mozbuild.frontend.data.XPCOMComponentManifests(context)

Bases: ContextDerived

Collects XPCOM manifest files during the build.

all_source_files()

all_sources()

config

context_all_paths

context_main_path

objdir

relsrcdir

srcdir

topobjdir

topsrcdir

class mozbuild.frontend.data.XPIDLModule(context, name, idl_files)

Bases: ContextDerived

Describes an XPIDL module to be compiled.

idl_files

name

mozbuild.frontend.data.cargo_output_directory(context, target_var)

mozbuild.frontend.emitter module

class mozbuild.frontend.emitter.TreeMetadataEmitter(config)

Bases: LoggingMixin

Converts the executed mozbuild files into data structures.

This is a bridge between reader.py and data.py. It takes what was read by reader.BuildReader and converts it into the classes defined in the data module.

ARCH_VAR = {'host': 'HOST_OS_ARCH', 'target': 'OS_TARGET'}

LIBRARY_NAME_VAR = {'host': 'HOST_LIBRARY_NAME', 'target': 'LIBRARY_NAME', 'wasm': 'SANDBOXED_WASM_LIBRARY_NAME'}

STDCXXCOMPAT_NAME = {'host': 'host_stdc++compat', 'target': 'stdc++compat'}

emit(output, emitfn=None)

Convert the BuildReader output into data structures.

The return value from BuildReader.read_topsrcdir() (a generator) is typically fed into this function.

emit_from_context(context)

Convert a Context to tree metadata objects.

This is a generator of mozbuild.frontend.data.ContextDerived instances.

summary()

mozbuild.frontend.gyp_reader module

class mozbuild.frontend.gyp_reader.GypContext(config, relobjdir)

Bases: TemplateContext

Specialized Context for use with data extracted from Gyp.

config is the ConfigEnvironment for this context. relobjdir is the object directory that will be used for this context, relative to the topobjdir defined in the ConfigEnvironment.

class mozbuild.frontend.gyp_reader.GypProcessor(config, gyp_dir_attrs, path, output, executor, action_overrides, non_unified_sources)

Bases: object

Reads a gyp configuration in the background using the given executor and emits GypContexts for the backend to process.

config is a ConfigEnvironment, path is the path to a root gyp configuration file, and output is the base path under which the objdir for the various gyp dependencies will be. gyp_dir_attrs are attributes set for the dir from moz.build.

property results

mozbuild.frontend.gyp_reader.handle_actions(actions, context, action_overrides)

mozbuild.frontend.gyp_reader.handle_copies(copies, context)

mozbuild.frontend.gyp_reader.load_gyp(*args)

mozbuild.frontend.gyp_reader.process_gyp_result(gyp_result, gyp_dir_attrs, path, config, output, non_unified_sources, action_overrides)

mozbuild.frontend.mach_commands module

exception mozbuild.frontend.mach_commands.InvalidPathException

Bases: Exception

Represents an error due to an invalid path.

mozbuild.frontend.mach_commands.bugzilla_automation(command_context, out_dir)

Analyze and validate Bugzilla metadata as required by automation.

This will write out JSON and gzipped JSON files for Bugzilla metadata.

The exit code will be non-0 if Bugzilla metadata fails validation.

mozbuild.frontend.mach_commands.file_info(command_context)

Show files metadata derived from moz.build files.

moz.build files contain “Files” sub-contexts for declaring metadata against file patterns. This command suite is used to query that data.

mozbuild.frontend.mach_commands.file_info_bugzilla(command_context, paths, rev=None, fmt=None)

Show Bugzilla component for a set of files.

Given a requested set of files (which can be specified using wildcards), print the Bugzilla component for each file.

mozbuild.frontend.mach_commands.file_info_missing_bugzilla(command_context, paths, rev=None, fmt=None)

mozbuild.frontend.mach_commands.file_info_schedules(command_context, paths)

Show what is scheduled by the given files.

Given a requested set of files (which can be specified using wildcards), print the total set of scheduled components.

mozbuild.frontend.mach_commands.reference(command_context, symbol, name_only=False)

mozbuild.frontend.reader module

Read build frontend files into data structures.

In terms of code architecture, the main interface is BuildReader. BuildReader starts with a root mozbuild file. It creates a new execution environment for this file, which is represented by the Sandbox class. The Sandbox class is used to fill a Context, representing the output of an individual mozbuild file. The

The BuildReader contains basic logic for traversing a tree of mozbuild files. It does this by examining specific variables populated during execution.

class mozbuild.frontend.reader.BuildReader(config, finder=<mozpack.files.FileFinder object>)

Bases: object

Read a tree of mozbuild files into data structures.

This is where the build system starts. You give it a tree configuration (the output of configuration) and it executes the moz.build files and collects the data they define.

The reader can optionally call a callable after each sandbox is evaluated but before its evaluated content is processed. This gives callers the opportunity to modify contexts before side-effects occur from their content. This callback receives the Context containing the result of each sandbox evaluation. Its return value is ignored.

all_mozbuild_paths()

Iterator over all available moz.build files.

This method has little to do with the reader. It should arguably belong elsewhere.

files_info(paths)

Obtain aggregate data from Files for a set of files.

Given a set of input paths, determine which moz.build files may define metadata for them, evaluate those moz.build files, and apply file metadata rules defined within to determine metadata values for each file requested.

Essentially, for each input path:

Determine the set of moz.build files relevant to that file by looking for moz.build files in ancestor directories.
Evaluate moz.build files starting with the most distant.
Iterate over Files sub-contexts.
If the file pattern matches the file we’re seeking info on, apply attribute updates.
Return the most recent value of attributes.

find_variables_from_ast(variables, path=None)

Finds all assignments to the specified variables by parsing moz.build abstract syntax trees.

This function only supports two cases, as detailed below.

A dict. Keys and values should both be strings, e.g:

VARIABLE[‘foo’] = ‘bar’

This is an Assign node with a Subscript target. The Subscript’s value is a Name node with id “VARIABLE”. The slice of this target is an Index node and its value is a Str with value “foo”.

2) A simple list. Values should be strings, e.g: The target of the assignment should be a Name node. Values should be a List node, whose elements are Str nodes. e.g:

VARIABLE += [‘foo’]

This is an AugAssign node with a Name target with id “VARIABLE”. The value is a List node containing one Str element whose value is “foo”.

With a little work, this function could support other types of assignment. But if we end up writing a lot of AST code, it might be best to import a high-level AST manipulation library into the tree.

Parameters:

variables (list) – A list of variable assignments to capture.
path (str) – A path relative to the source dir. If specified, only moz.build files relevant to this path will be parsed. Otherwise all moz.build files are parsed.

Returns:

A generator that generates tuples of the form (<moz.build path>, <variable name>, <key>, <value>). The key will only be defined if the variable is an object, otherwise it is None.

gyp_summary()

read_mozbuild(path, config, descend=True, metadata={})

Read and process a mozbuild file, descending into children.

This starts with a single mozbuild file, executes it, and descends into other referenced files per our traversal logic.

The traversal logic is to iterate over the *DIRS variables, treating each element as a relative directory path. For each encountered directory, we will open the moz.build file located in that directory in a new Sandbox and process it.

If descend is True (the default), we will descend into child directories and files per variable values.

Arbitrary metadata in the form of a dict can be passed into this function. This feature is intended to facilitate the build reader injecting state and annotations into moz.build files that is independent of the sandbox’s execution context.

Traversal is performed depth first (for no particular reason).

read_relevant_mozbuilds(paths)

Read and process moz.build files relevant for a set of paths.

For an iterable of relative-to-root filesystem paths paths, find all moz.build files that may apply to them based on filesystem hierarchy and read those moz.build files.

The return value is a 2-tuple. The first item is a dict mapping each input filesystem path to a list of Context instances that are relevant to that path. The second item is a list of all Context instances. Each Context instance is in both data structures.

read_topsrcdir()

Read the tree of linked moz.build files.

This starts with the tree’s top-most moz.build file and descends into all linked moz.build files until all relevant files have been evaluated.

This is a generator of Context instances. As each moz.build file is read, a new Context is created and emitted.

summary()

exception mozbuild.frontend.reader.BuildReaderError(file_stack, trace, sandbox_exec_error=None, sandbox_load_error=None, validation_error=None, other_error=None, sandbox_called_error=None)

Bases: Exception

Represents errors encountered during BuildReader execution.

The main purpose of this class is to facilitate user-actionable error messages. Execution errors should say:

Why they failed

Where they failed

What can be done to prevent the error

A lot of the code in this class should arguably be inside sandbox.py. However, extraction is somewhat difficult given the additions MozbuildSandbox has over Sandbox (e.g. the concept of included files - which affect error messages, of course).

property actual_file

property main_file

property sandbox_error

class mozbuild.frontend.reader.EmptyConfig(topsrcdir, substs=None)

Bases: object

A config object that is empty.

This config object is suitable for using with a BuildReader on a vanilla checkout, without any existing configuration. The config is simply bootstrapped from a top source directory path.

class PopulateOnGetDict(default_factory, *args, **kwargs)

Bases: ReadOnlyDefaultDict

A variation on ReadOnlyDefaultDict that populates during .get().

This variation is needed because CONFIG uses .get() to access members. Without it, None (instead of our EmptyValue types) would be returned.

get(key, default=None): Return the value for key if key is in the dictionary, else default.

default_substs = {'JS_STANDALONE': '1', 'MOZ_APP_NAME': 'empty', 'MOZ_CHILD_PROCESS_NAME': 'empty'}

class mozbuild.frontend.reader.MozbuildSandbox(context, metadata={}, finder=<mozpack.files.FileFinder object>)

Bases: Sandbox

Implementation of a Sandbox tailored for mozbuild files.

We expose a few useful functions and expose the set of variables defining Mozilla’s build system.

context is a Context instance.

metadata is a dict of metadata that can be used during the sandbox evaluation.

exec_file(path, becomes_current_path=True)

Override exec_file to normalize paths and restrict file loading.

Paths will be rejected if they do not fall under topsrcdir or one of the external roots.

recompute_exports(): Recompute the variables to export to subdirectories with the current values in the subdirectory.

exception mozbuild.frontend.reader.SandboxCalledError(file_stack, message)

Bases: SandboxError

Represents an error resulting from calling the error() function.

exception mozbuild.frontend.reader.SandboxValidationError(message, context)

Bases: Exception

Represents an error encountered when validating sandbox results.

class mozbuild.frontend.reader.TemplateFunction(func, sandbox)

Bases: object

class RewriteName(sandbox, global_name)

Bases: NodeTransformer

AST Node Transformer to rewrite variable accesses to go through a dict.

visit_Name(node)

exec_in_sandbox(sandbox, *args, **kwargs): Executes the template function in the given sandbox.

mozbuild.frontend.reader.is_read_allowed(path, config)

Whether we are allowed to load a mozbuild file at the specified path.

This is used as cheap security to ensure the build is isolated to known source directories.

We are allowed to read from the main source directory and any defined external source directories. The latter is to allow 3rd party applications to hook into our build system.

mozbuild.frontend.reader.log(logger, level, action, params, formatter)

mozbuild.frontend.sandbox module

Python sandbox implementation for build files.

This module contains classes for Python sandboxes that execute in a highly-controlled environment.

The main class is Sandbox. This provides an execution environment for Python code and is used to fill a Context instance for the takeaway information from the execution.

Code in this module takes a different approach to exception handling compared to what you’d see elsewhere in Python. Arguments to built-in exceptions like KeyError are machine parseable. This machine-friendly data is used to present user-friendly error messages in the case of errors.

class mozbuild.frontend.sandbox.Sandbox(context, finder=<mozpack.files.FileFinder object>)

Bases: dict

Represents a sandbox for executing Python code.

This class provides a sandbox for execution of a single mozbuild frontend file. The results of that execution is stored in the Context instance given as the context argument.

Sandbox is effectively a glorified wrapper around compile() + exec(). You point it at some Python code and it executes it. The main difference from executing Python code like normal is that the executed code is very limited in what it can do: the sandbox only exposes a very limited set of Python functionality. Only specific types and functions are available. This prevents executed code from doing things like import modules, open files, etc.

Sandbox instances act as global namespace for the sandboxed execution itself. They shall not be used to access the results of the execution. Those results are available in the given Context instance after execution.

The Sandbox itself is responsible for enforcing rules such as forbidding reassignment of variables.

Implementation note: Sandbox derives from dict because exec() insists that what it is given for namespaces is a dict.

BUILTINS = {'False': False, 'None': None, 'True': True, 'int': <class 'int'>, 'set': <class 'set'>, 'sorted': <function alphabetical_sorted>, 'tuple': <class 'tuple'>}

exec_file(path, becomes_current_path=True)

Execute code at a path in the sandbox.

The path must be absolute.

exec_function(func, args=(), kwargs={}, path='', becomes_current_path=True): Execute function with the given arguments in the sandbox.

exec_source(source, path='', becomes_current_path=True)

Execute Python code within a string.

The passed string should contain Python code to be executed. The string will be compiled and executed.

You should almost always go through exec_file() because exec_source() does not perform extra path normalization. This can cause relative paths to behave weirdly.

get(key, default=None): Return the value for key if key is in the dictionary, else default.

pop_subcontext(context)

Pop a SubContext off the execution stack.

SubContexts must be pushed and popped in opposite order. This is validated as part of the function call to ensure proper consumer API use.

push_subcontext(context)

Push a SubContext onto the execution stack.

When called, the active context will be set to the specified context, meaning all variable accesses will go through it. We also record this SubContext as having been executed as part of this sandbox.

exception mozbuild.frontend.sandbox.SandboxError(file_stack): Bases: Exception

exception mozbuild.frontend.sandbox.SandboxExecutionError(file_stack, exc_type, exc_value, trace)

Bases: SandboxError

Represents errors encountered during execution of a Sandbox.

This is a simple container exception. It’s purpose is to capture state so something else can report on it.

exception mozbuild.frontend.sandbox.SandboxLoadError(file_stack, trace, illegal_path=None, read_error=None)

Bases: SandboxError

Represents errors encountered when loading a file for execution.

This exception represents errors in a Sandbox that occurred as part of loading a file. The error could have occurred in the course of executing a file. If so, the file_stack will be non-empty and the file that caused the load will be on top of the stack.

mozbuild.frontend.sandbox.alphabetical_sorted(iterable, key=<function <lambda>>, reverse=False): sorted() replacement for the sandbox, ordering alphabetically by default.

mozbuild.frontend package

Submodules

mozbuild.frontend.context module

mozbuild.frontend.data module

mozbuild.frontend.emitter module

mozbuild.frontend.gyp_reader module

mozbuild.frontend.mach_commands module

mozbuild.frontend.reader module

mozbuild.frontend.sandbox module

Module contents