Firefox CI and Taskgraph¶
Firefox’s continuous integration (CI) system is made up of three parts.
First there’s Taskcluster, a task execution framework developed by Mozilla. Taskcluster is capable of building complex, scalable and highly customizable CI systems. Taskcluster is more like a set of building blocks that can be used to create CI systems, rather than a fully-fledged CI system itself.
The second part is the Firefox CI instance of Taskcluster. This is the
Taskcluster deployment responsible for running most of Mozilla’s CI needs. This
component is comprised of a Kubernetes cluster to run the Taskcluster services
(maintained by SRE Services), some customizations to support Taskcluster
on hg.mozilla.org
and access control in the fxci-config repo
(maintained by Release Engineering).
The third part is Taskgraph. Taskgraph is a Python library that can generate a DAG of tasks and submit them to a Taskcluster instance. This is the component that most Gecko and Mozilla developers will interact with when working with tasks, and will be the focal point of the rest of this documentation.
Note
Historically Taskgraph started out inside mozilla-central
. It was then
forked to standalone Taskgraph in order to support projects on Github.
Over time maintaining two forks grew cumbersome so they are in the process
of being merged back together.
Today the version of Taskgraph under taskcluster/gecko_taskgraph
depends
on the standalone version, which is vendored under
third_party/python/taskcluster_taskgraph
. There is still a lot of
duplication between these places, but gecko_taskgraph
is slowly being
re-written to consume standalone Taskgraph.
The taskcluster
directory contains all the files needed to define the graph
of tasks that must be executed to build and test the Gecko tree. This is more
complex than you think! There are 30,000+ tasks and counting in Gecko’s CI
graphs.
Taskgraph supports:
A huge array of tasks
Different behavior for different repositories
“Try” pushes, with special means to select a subset of the graph for execution
Optimization – skipping tasks that have already been performed
Extremely flexible generation of a variety of tasks using an approach of incrementally transforming job descriptions into task definitions.
The most comprehensive resource on Taskgraph is Taskgraph’s documentation. These docs
will refer there when appropriate and expand on topics specific to gecko_taskgraph
where necessary.
If you are reading this with a particular goal in mind and would rather avoid becoming a task-graph expert, check out the how-to section.
- Taskgraph Overview
- How Tos
- Transforms
- Optimization Strategies
- Periodic Taskgraphs
- Try
- Release Promotion
- Version Control in CI
- Taskcluster Configuration
- Reference
- Task Kinds
- build
- build-fat-aar
- build-signing
- build-mac-signing
- build-mac-notarization
- artifact-build
- bootstrap
- hazard
- l10n
- shippable-l10n
- shippable-l10n-signing
- shippable-l10n-mac-signing
- shippable-l10n-mac-notarization
- source-test
- code-review
- upload-symbols
- upload-symbols-dummy
- upload-generated-sources
- upload-generated-sources-dummy
- valgrind
- searchfox
- static-analysis-autotest
- toolchain
- spidermonkey
- test
- docker-image
- balrog
- beetmover
- beetmover-l10n
- beetmover-repackage
- release-beetmover-push-to-release
- beetmover-source
- beetmover-geckoview
- beetmover-apt
- condprof
- release-source-checksums-signing
- beetmover-checksums
- release-beetmover-source-checksums
- perftest
- release-balrog-submit-toplevel
- release-secondary-balrog-submit-toplevel
- release-balrog-scheduling
- release-secondary-balrog-scheduling
- release-binary-transparency
- release-flatpak-repackage
- release-flatpak-push
- release-secondary-flatpak-push
- release-notify-av-announce
- release-notify-push
- release-notify-ship
- release-secondary-notify-ship
- release-notify-promote
- release-notify-started
- release-notify-testrail
- release-bouncer-sub
- release-mark-as-shipped
- release-bouncer-aliases
- cron-bouncer-check
- bouncer-locations
- release-bouncer-check
- release-generate-checksums
- release-generate-checksums-signing
- release-generate-checksums-beetmover
- release-final-verify
- release-secondary-final-verify
- release-push-langpacks
- release-beetmover-signed-langpacks
- release-beetmover-signed-langpacks-checksums
- release-update-verify
- release-update-verify-next
- release-update-verify-config
- release-secondary-update-verify-config
- release-update-verify-config-next
- release-updates-builder
- release-version-bump
- release-source
- release-source-signing
- release-partner-repack
- release-partner-attribution
- release-partner-repack-chunking-dummy
- release-partner-repack-signing
- release-partner-repack-mac-signing
- release-partner-repack-mac-notarization
- release-partner-repack-repackage
- release-partner-repack-repackage-signing
- release-partner-repack-beetmover
- release-partner-attribution-beetmover
- release-partner-repack-bouncer-sub
- release-early-tagging
- release-eme-free-repack
- release-eme-free-repack-signing
- release-eme-free-repack-repackage
- release-eme-free-repack-repackage-signing
- release-eme-free-repack-beetmover
- release-eme-free-repack-beetmover-checksums
- release-eme-free-repack-mac-signing
- release-eme-free-repack-mac-notarization
- repackage
- repackage-l10n
- repackage-deb
- repackage-deb-l10n
- repackage-signing
- repackage-signing-l10n
- mar-signing
- mar-signing-l10n
- mar-signing-autograph-stage
- repackage-msi
- repackage-signing-msi
- repackage-msix
- repackage-shippable-l10n-msix
- repackage-signing-msix
- repackage-signing-shippable-l10n-msix
- repackage-snap
- release-msix-push
- repo-update
- partials
- partials-zucchini
- partials-signing
- post-balrog-dummy
- post-beetmover-dummy
- post-beetmover-checksums-dummy
- post-beetmover-components-dummy
- post-langpack-dummy
- post-update-verify-dummy
- fetch
- packages
- diffoscope
- addon
- openh264-plugin
- openh264-signing
- webrender
- instrumented-build
- generate-profile
- geckodriver-signing
- geckodriver-mac-notarization
- maybe-release
- l10n-bump
- merge-automation
- sentry
- system-symbols
- system-symbols-upload
- system-symbols-reprocess
- scriptworker-canary
- updatebot
- fuzzing
- startup-test
- attribution
- attribution-l10n
- snap-upstream-build
- snap-upstream-test
- trigger-comm-central
- build-components
- build-bundle
- build-apk
- build-samples-browser
- signing
- signing-bundle
- signing-apk
- test-components
- test-apk
- ui-test-apk
- android-browsertime
- android-startup-test
- post-signing-dummy
- beetmover-components
- beetmover-android-app
- push-bundle
- android-l10n
- release-update-product-channel-version
- instrumented-build-apk
- generate-baseline-profile-firebase
- Parameters
- Task Attributes
- kind
- run_on_projects
- run_on_hg_branches
- task_duplicates
- build_platform
- build_type
- test_platform
- unittest_suite
- unittest_category
- unittest_try_name
- unittest_variant
- talos_try_name
- raptor_try_name
- job_try_name
- test_chunk
- test_manifests
- image_name
- nightly
- shippable
- all_locales
- all_locales_with_changesets
- l10n_chunk
- chunk_locales
- locale
- signed
- stub-installer
- repackage_type
- fetch-artifact
- fetch-alias
- toolchain-artifact
- toolchain-alias
- toolchain-env
- toolchain-command
- always_target
- shipping_product
- shipping_phase
- artifact_prefix
- artifact_map
- release_artifacts
- batch
- enable-full-crashsymbols
- skip-upload-crashsymbols
- upload-generated-sources
- cron
- cached_task
- eager_indexes
- required_signoffs
- update-channel
- mar-channel-id
- accepted-mar-channel-ids
- openh264_rev
- code-review
- retrigger
- disable-push-apk
- disable-build-signing
- enable-build-signing
- run-visual-metrics
- skip-verify-test-packaging
- geckodriver
- rebuild-on-release
- local-toolchain
- artifact-build
- maven_packages
- supports-artifact-builds
- primary-kind-dependency
- snap_test_type
- snap_test_release
- build-type
- component
- apks
- aab
- release-type
- shipping-product
- artifacts
- is_final_chunked_task
- legacy
- nightly-test
- screenshots
- abi
- apk
- test-manifests
- lull-schedule
- this_chunk
- total_chunks
- toolchain-extract
- Caches
- Task Kinds