Taskgraph Overview

Taskgraph builds a directed acyclic graph, where each node in the graph represents a task, and each edge represents the dependency relationships between them.

See Taskgraph’s graph generation documentation for more details.

Decision Task

The decision task is the first task created when a new graph begins. It is responsible for creating the rest of the task graph.

The decision task for pushes is defined in-tree, in .taskcluster.yml. That task description invokes mach taskcluster decision with some metadata about the push. That mach command determines the required graph of tasks to submit, then calls the Taskcluster API to create the tasks.

Note

mach taskgraph decision is not meant to be invoked by humans. Instead, follow this guide (prepending mach to all commands) to invoke Taskgraph locally without submitting anything to Taskcluster.

Kinds

Generation starts with “kinds”. These are yaml files that denote groupings of tasks that are loosely related to one another. For example, in Gecko there are build and test kinds. Each kind has its own directory under taskcluster/kinds which contains a kind.yml file.

For more information on kinds, see Taskgraph’s kind documentation. For a list of available kinds in mozilla-central, see the kinds reference.

Loader

Next, a “loader” is responsible for parsing each kind.yml file and turning it into an initial set of tasks. Loaders will always parse kinds in an ordering satisfying the kind-dependencies key.

See Taskgraph’s loader documentation for more details.

Transforms

After the initial set of tasks are loaded, transformations are applied to each task. Transforms are Python functions that take a generator of tasks as input, and yields a generator of tasks as output. Possibly modifying, adding or removing tasks along the way.

See Taskgrpah’s transforms documentation for more details on transforms, or the transforms section for information on some of the transforms available in mozilla-central.

Target Tasks

After transformation is finished, the target_tasks module filters out any tasks that aren’t applicable to the current parameter set.

Optimization

After the “target graph” is generated, an optimization process looks to remove or replace unnecessary tasks in the graph. For instance, a task may be removed if the push doesn’t modify any of the files that could affect it.

See Taskgraph’s optimization documentation for more details, or the optimization strategies available in mozilla-central.

Task Parameterization

A few components of tasks are only known at the very end of the decision task – just before the queue.createTask call is made. These are specified using simple parameterized values, as follows:

{"relative-datestamp": "certain number of seconds/hours/days/years"}

Objects of this form will be replaced with an offset from the current time just before the queue.createTask call is made. For example, an artifact expiration might be specified as {"relative-datestamp": "1 year"}.

{"task-reference": "string containing <dep-name>"}

The task definition may contain “task references” of this form. These will be replaced during the optimization step, with the appropriate taskId for the named dependency substituted for <dep-name> in the string. Additionally, decision and self can be used a dependency names to refer to the decision task, and the task itself. Multiple labels may be substituted in a single string, and <<> can be used to escape a literal <.

{"artifact-reference": "..<dep-name/artifact/name>.."}

Similar to a task-reference, but this substitutes a URL to the queue’s getLatestArtifact API method (for which a GET will redirect to the artifact itself).

Graph Configuration

There are several configuration settings that are pertain to the entire taskgraph. These are specified in config.yml at the root of the taskgraph configuration (typically taskcluster). The available settings are documented inline in taskcluster/gecko_taskgraph/config.py.

Action Tasks

Action Tasks are tasks which perform an action based on a manual trigger (e.g clicking a button in Treeherder). Actions are how it is possible to retrigger or “Add New Jobs”.

For more information, see Taskgraph’s actions documentation.