Creating Toolchain Archives

There are various scripts in the repository for producing archives of the build tools (e.g. compilers and linkers) required to build.

Clang and Rust

To modify the toolchains used for a particular task, you may need several things:

  1. A build task

  2. Which uses a toolchain task

  3. Which uses a git fetch

  4. (clang only) Which uses a config json

  5. Which takes patches you may want to apply.

For the most part, you should be able to accomplish what you want by copying/editing the existing examples in those files.

Clang

Building clang is handled by build-clang.py, which uses several resources in the build-clang directory. Read the build-clang README for more details.

Note for local builds: build-clang.py can be run on developer machines but its lengthy multi-stage build process is unnecessary for most local development. The upstream LLVM Getting Started Guide has instructions on how to build clang more directly.

Rust

Rust builds are handled by repack_rust.py. The primary purpose of that script is to download prebuilt tarballs from the Rust project.

It uses the same basic format as rustup for specifying the toolchain (via --channel):

  • request a stable build with 1.xx.y (e.g. 1.47.0)

  • request a beta build with beta-yyyy-mm-dd (e.g. beta-2020-08-26)

  • request a nightly build with nightly-yyyy-mm-dd (e.g. nightly-2020-08-26)

  • request a build from Rust’s ci with bors-$sha (e.g. bors-796a2a9bbe7614610bd67d4cd0cf0dfff0468778)

  • request a from-source build with dev

Rust From Source

As of this writing, from-source builds for Rust are a new feature, and not used anywhere by default. The feature was added so that we can test patches to rustc against the tree. Expect things to be a bit hacky and limited.

Most importantly, building from source requires your toolchain to have a fetch of the rust tree as well as clang and binutils toolchains. It is also recommended to upgrade the worker-type to e.g. b-linux-large.

Rust’s build dependencies are fairly minimal, and it has a sanity check that should catch any missing or too-old dependencies. See the Rust README for more details.

Patches are set via the –patch flag (passed via toolchain/rust.yml). Patch paths are assumed to be relative to /build/build-rust/, and may be optionally prefixed with module-path: to specify they apply to that git submodule in the Rust source. e.g. --patch src/llvm-project:mypatch.diff patches rust’s llvm with /build/build-rust/mypatch.diff. There are no currently checked in rust patches to use as an example, but they should be the same format as the clang ones.

Rust builds are not currently configurable, and uses a hardcoded config.toml, which you may need to edit for your purposes. See Rust’s example config for details/defaults. Note that these options do occasionally change, so be sure you’re using options for the version you’re targeting. For instance, there was a large change around Rust ~1.48, and the currently checked in config was for 1.47, so it may not work properly when building the latest version of Rust.

Rust builds are currently limited to targeting only the host platform. Although the machinery is in place to request additional targets, the cross-compilation fails for some unknown reason. We have not yet investigated what needs to be done to get this working.

While Rust generally maintains a clean tree for building rustc and cargo, other tools like rustfmt or miri are allowed to be transiently broken. This means not every commit in the Rust tree will be able to build the tools we require.

Although repack_rust considers rustfmt an optional package, Rust builds do not currently implement this and will fail if rustfmt is busted. Some attempt was made to work around it, but more work is needed.

Windows

The build/vs/generate_yaml.py and build/vs/pack_vs.py scripts are used to manage and build Windows toolchain archives containing Visual Studio executables, SDKs, etc.

The build/vs/generate_yaml.py script is used to generate one of the YAML files used in the relevant toolchain task. The exact command line used to generate the file is stored in the header of the YAML file itself. Each YAML file records the necessary downloads from Microsoft servers to install the required Visual Studio components given on the command line.

The build/vs/pack_vs.py script takes a YAML file as input and generates the corresponding toolchain artifact.

Both scripts should be run via mach python.

Firefox for Android with Gradle

To build Firefox for Android with Gradle in automation, archives containing both the Gradle executable and a Maven repository comprising the exact build dependencies are produced and uploaded to an internal Mozilla server. The build automation will download, verify, and extract these archive before building. These archives provide a self-contained Gradle and Maven repository so that machines don’t need to fetch additional Maven dependencies at build time. (Gradle and the downloaded Maven dependencies can be both redistributed publicly.)

Archiving the Gradle executable is straight-forward, but archiving a local Maven repository is not. Therefore a toolchain job exists for producing the required archives, android-gradle-dependencies. The job runs in a container based on a custom Docker image and spawns a Sonatype Nexus proxying Maven repository process in the background. The job builds Firefox for Android using Gradle and the in-tree Gradle configuration rooted at build.gradle. The spawned proxying Maven repository downloads external dependencies and collects them. After the Gradle build completes, the job archives the Gradle version used to build, and the downloaded Maven repository, and exposes them as Task Cluster artifacts.

To update the version of Gradle in the archive produced, update gradle/wrapper/gradle-wrapper.properties. Be sure to also update the SHA256 checksum to prevent poisoning the build machines!

To update the versions of Gradle dependencies used, update dependencies sections in the in-tree Gradle configuration rooted at build.gradle. Once you are confident your changes build locally, push a fresh build to try. The android-gradle-dependencies toolchain should run automatically, fetching your new dependencies and wiring them into the appropriate try build jobs.

To update the version of Sonatype Nexus, update the sonatype-nexus fetch task definition.

To modify the Sonatype Nexus configuration, typically to proxy a new remote Maven repository, modify taskcluster/scripts/misc/android-gradle-dependencies/nexus.xml.

There is also a toolchain job that fetches the Android SDK and related packages. To update the versions of packaged fetched, modify python/mozboot/mozboot/android-packages.txt and update the various in-tree versions accordingly.