gecko_taskgraph.util package¶

Submodules¶

gecko_taskgraph.util.attributes module¶

gecko_taskgraph.util.attributes.attrmatch(attributes, **kwargs)¶: Determine whether the given set of task attributes matches. The conditions are given as keyword arguments, where each keyword names an attribute. The keyword value can be a literal, a set, or a callable. A literal must match the attribute exactly. Given a set, the attribute value must be in the set. A callable is called with the attribute value. If an attribute is specified as a keyword argument but not present in the attributes, the result is False.

gecko_taskgraph.util.attributes.copy_attributes_from_dependent_job(dep_job, denylist=())¶

gecko_taskgraph.util.attributes.is_try(params)¶: Determine whether this graph is being built on a try project or for mach try fuzzy.

gecko_taskgraph.util.attributes.keymatch(attributes, target)¶: Determine if any keys in attributes are a match to target, then return a list of matching values. First exact matches will be checked. Failing that, regex matches and finally a default key.

gecko_taskgraph.util.attributes.match_run_on_hg_branches(hg_branch, run_on_hg_branches)¶: Determine whether the given project is included in the run-on-hg-branches parameter. Allows ‘all’.

gecko_taskgraph.util.attributes.match_run_on_projects(project, run_on_projects)¶: Determine whether the given project is included in the run-on-projects parameter, applying expansions for things like “integration” mentioned in the attribute documentation.

gecko_taskgraph.util.attributes.release_level(project)¶

Whether this is a staging release or not.

Return str: One of “production” or “staging”.

gecko_taskgraph.util.attributes.sorted_unique_list(*args)¶: Join one or more lists, and return a sorted list of unique members

gecko_taskgraph.util.attributes.task_name(task)¶

gecko_taskgraph.util.backstop module¶

gecko_taskgraph.util.backstop.is_backstop(params, push_interval=20, time_interval=240, trust_domain='gecko', integration_projects={'autoland'})¶

Determines whether the given parameters represent a backstop push.

Parameters

push_interval (int) – Number of pushes
time_interval (int) – Minutes between forced schedules. Use 0 to disable.
trust_domain (str) – “gecko” for Firefox, “comm” for Thunderbird
integration_projects (set) – project that uses backstop optimization

Returns

True if this is a backstop, otherwise False.

Return type

bool

gecko_taskgraph.util.bugbug module¶

exception gecko_taskgraph.util.bugbug.BugbugTimeoutException¶: Bases: Exception

gecko_taskgraph.util.bugbug.get_session()¶

gecko_taskgraph.util.bugbug.push_schedules(branch, rev)¶

gecko_taskgraph.util.bugbug.translate_group(group)¶

gecko_taskgraph.util.cached_tasks module¶

gecko_taskgraph.util.cached_tasks.add_optimization(config, taskdesc, cache_type, cache_name, digest=None, digest_data=None)¶

Allow the results of this task to be cached. This adds index routes to the task so it can be looked up for future runs, and optimization hints so that cached artifacts can be found. Exactly one of digest and digest_data must be passed.

Parameters

config (TransformConfig) – The configuration for the kind being transformed.
taskdesc (dict) – The description of the current task.
cache_type (str) – The type of task result being cached.
cache_name (str) – The name of the object being cached.
digest (bytes or None) – A unique string indentifying this version of the artifacts being generated. Typically this will be the hash of inputs to the task.
digest_data (list of bytes or None) – A list of bytes representing the inputs of this task. They will be concatenated and hashed to create the digest for this task.

gecko_taskgraph.util.chunking module¶

Utility functions to handle test chunking.

class gecko_taskgraph.util.chunking.BaseManifestLoader(params)¶

Bases: object

abstract get_manifests(flavor, subsuite, mozinfo)¶

Compute which manifests should run for the given flavor, subsuite and mozinfo.

This function returns skipped manifests separately so that more balanced chunks can be achieved by only considering “active” manifests in the chunking algorithm.

Parameters

flavor (str) – The suite to run. Values are defined by the ‘build_flavor’ key in moztest.resolve.TEST_SUITES.
subsuite (str) – The subsuite to run or ‘undefined’ to denote no subsuite.
mozinfo (frozenset) – Set of data in the form of (<key>, <value>) used for filtering.

Returns

A tuple of two manifest lists. The first is the set of active manifests (will run at least one test. The second is a list of skipped manifests (all tests are skipped).

class gecko_taskgraph.util.chunking.BugbugLoader(*args, **kwargs)¶

Bases: gecko_taskgraph.util.chunking.DefaultLoader

Load manifests using metadata from the TestResolver, and then filter them based on a query to bugbug.

CONFIDENCE_THRESHOLD = 0.7¶

get_manifests(suite, mozinfo)¶

Compute which manifests should run for the given flavor, subsuite and mozinfo.

This function returns skipped manifests separately so that more balanced chunks can be achieved by only considering “active” manifests in the chunking algorithm.

Parameters

flavor (str) – The suite to run. Values are defined by the ‘build_flavor’ key in moztest.resolve.TEST_SUITES.
subsuite (str) – The subsuite to run or ‘undefined’ to denote no subsuite.
mozinfo (frozenset) – Set of data in the form of (<key>, <value>) used for filtering.

Returns

A tuple of two manifest lists. The first is the set of active manifests (will run at least one test. The second is a list of skipped manifests (all tests are skipped).

class gecko_taskgraph.util.chunking.DefaultLoader(params)¶

Bases: gecko_taskgraph.util.chunking.BaseManifestLoader

Load manifests using metadata from the TestResolver.

get_manifests(suite, mozinfo)¶

Compute which manifests should run for the given flavor, subsuite and mozinfo.

This function returns skipped manifests separately so that more balanced chunks can be achieved by only considering “active” manifests in the chunking algorithm.

Parameters

flavor (str) – The suite to run. Values are defined by the ‘build_flavor’ key in moztest.resolve.TEST_SUITES.
subsuite (str) – The subsuite to run or ‘undefined’ to denote no subsuite.
mozinfo (frozenset) – Set of data in the form of (<key>, <value>) used for filtering.

Returns

A tuple of two manifest lists. The first is the set of active manifests (will run at least one test. The second is a list of skipped manifests (all tests are skipped).

get_tests(suite)¶

gecko_taskgraph.util.chunking.chunk_manifests(suite, platform, chunks, manifests)¶

Run the chunking algorithm.

Parameters

platform (str) – Platform used to find runtime info.
chunks (int) – Number of chunks to split manifests into.
manifests (list) – Manifests to chunk.

Returns

A list of length chunks where each item contains a list of manifests that run in that chunk.

gecko_taskgraph.util.chunking.get_manifest_loader(name, params)¶

gecko_taskgraph.util.chunking.get_runtimes(platform, suite_name)¶

gecko_taskgraph.util.chunking.guess_mozinfo_from_task(task)¶

Attempt to build a mozinfo dict from a task definition.

This won’t be perfect and many values used in the manifests will be missing. But it should cover most of the major ones and be “good enough” for chunking in the taskgraph.

Parameters: task (dict) – A task definition.
Returns: A dict that can be used as a mozinfo replacement.

gecko_taskgraph.util.declarative_artifacts module¶

gecko_taskgraph.util.declarative_artifacts.get_geckoview_artifact_id(config, platform, package, update_channel=None)¶

gecko_taskgraph.util.declarative_artifacts.get_geckoview_artifact_map(config, job)¶

gecko_taskgraph.util.declarative_artifacts.get_geckoview_template_vars(config, platform, package, update_channel)¶

gecko_taskgraph.util.declarative_artifacts.get_geckoview_upstream_artifacts(config, job, package, platform='')¶

gecko_taskgraph.util.docker module¶

gecko_taskgraph.util.hash module¶

gecko_taskgraph.util.hash.get_file_finder(base_path)¶

gecko_taskgraph.util.hash.hash_path(path)¶

Hash a single file.

Returns the SHA-256 hash in hex form.

gecko_taskgraph.util.hash.hash_paths(base_path, patterns)¶

Give a list of path patterns, return a digest of the contents of all the corresponding files, similarly to git tree objects or mercurial manifests.

Each file is hashed. The list of all hashes and file paths is then itself hashed to produce the result.

gecko_taskgraph.util.hg module¶

gecko_taskgraph.util.hg.calculate_head_rev(root)¶

gecko_taskgraph.util.hg.find_hg_revision_push_info(repository, revision)¶: Given the parameters for this action and a revision, find the pushlog_id of the revision.

gecko_taskgraph.util.hg.get_hg_commit_message(root, rev='.')¶

gecko_taskgraph.util.hg.get_hg_revision_branch(root, revision)¶: Given the parameters for a revision, find the hg_branch (aka relbranch) of the revision.

gecko_taskgraph.util.hg.get_json_automationrelevance(repository, revision)¶

gecko_taskgraph.util.hg.get_push_data(repository, project, push_id_start, push_id_end)¶

gecko_taskgraph.util.keyed_by module¶

gecko_taskgraph.util.keyed_by.evaluate_keyed_by(value, item_name, attributes, defer=None, enforce_single_match=True)¶

For values which can either accept a literal value, or be keyed by some attributes, perform that lookup and return the result.

For example, given item:

by-test-platform:
    macosx-10.11/debug: 13
    win.*: 6
    default: 12

a call to evaluate_keyed_by(item, ‘thing-name’, {‘test-platform’: ‘linux96’) would return 12.

Items can be nested as deeply as desired:

by-test-platform:
    win.*:
        by-project:
            ash: ..
            cedar: ..
    linux: 13
    default: 12

Parameters

value (str) – Name of the value to perform evaluation on.
item_name (str) – Used to generate useful error messages.
attributes (dict) – Dictionary of attributes used to lookup ‘by-<key>’ with.
defer (list) – Allows evaluating a by-* entry at a later time. In the example above it’s possible that the project attribute hasn’t been set yet, in which case we’d want to stop before resolving that subkey and then call this function again later. This can be accomplished by setting defer=[“project”] in this example.
enforce_single_match (bool) – If True (default), each task may only match a single arm of the evaluation.

gecko_taskgraph.util.partials module¶

gecko_taskgraph.util.partials.find_localtest(fileUrls)¶

gecko_taskgraph.util.partials.get_balrog_platform_name(platform)¶

Convert build platform names into balrog platform names.

Remove known values instead to catch aarch64 and other platforms that may be added.

gecko_taskgraph.util.partials.get_builds(release_history, platform, locale)¶: Examine cached balrog release history and return the list of builds we need to generate diffs from

gecko_taskgraph.util.partials.get_partials_artifacts_from_params(release_history, platform, locale)¶

gecko_taskgraph.util.partials.get_partials_info_from_params(release_history, platform, locale)¶

gecko_taskgraph.util.partials.get_release_builds(release, branch)¶

gecko_taskgraph.util.partials.get_sorted_releases(product, branch)¶: Returns a list of release names from Balrog. :param product: product name, AKA appName :param branch: branch name, e.g. mozilla-central :return: a sorted list of release names, most recent first.

gecko_taskgraph.util.partials.populate_release_history(product, branch, maxbuilds=4, maxsearch=10, partial_updates=None)¶

gecko_taskgraph.util.partners module¶

gecko_taskgraph.util.partners.GITHUB_API_ENDPOINT = 'https://api.github.com/graphql'¶: LOGIN_QUERY, MANIFEST_QUERY, and REPACK_CFG_QUERY are all written to the Github v4 API, which users GraphQL. See https://developer.github.com/v4/

gecko_taskgraph.util.partners.apply_partner_priority(config, jobs)¶

gecko_taskgraph.util.partners.check_if_partners_enabled(config, tasks)¶

gecko_taskgraph.util.partners.check_login(token)¶

gecko_taskgraph.util.partners.fix_partner_config(orig_config)¶

gecko_taskgraph.util.partners.generate_attribution_code(defaults, partner)¶

gecko_taskgraph.util.partners.get_attribution_config(manifestRepo, token)¶

gecko_taskgraph.util.partners.get_ftp_platform(platform)¶

gecko_taskgraph.util.partners.get_partner_config_by_kind(config, kind)¶

Retrieve partner data starting from the manifest url, which points to a repository containing a default.xml that is intended to be drive the Google tool ‘repo’. It descends into each partner repo to lookup and parse the repack.cfg file(s).

Supports caching data by kind to avoid repeated requests, relying on the related kinds for partner repacking, signing, repackage, repackage signing all having the same kind prefix.

gecko_taskgraph.util.partners.get_partner_config_by_url(manifest_url, kind, token, partner_subset=None)¶

Retrieve partner data starting from the manifest url, which points to a repository containing a default.xml that is intended to be drive the Google tool ‘repo’. It descends into each partner repo to lookup and parse the repack.cfg file(s).

If partner_subset is a list of sub_config names only return data for those.

Supports caching data by kind to avoid repeated requests, relying on the related kinds for partner repacking, signing, repackage, repackage signing all having the same kind prefix.

gecko_taskgraph.util.partners.get_partner_url_config(parameters, graph_config)¶

gecko_taskgraph.util.partners.get_partners(manifestRepo, token)¶: Given the url to a manifest repository, retrieve the default.xml and parse it into a list of partner repos.

gecko_taskgraph.util.partners.get_partners_to_be_published(config)¶

gecko_taskgraph.util.partners.get_repack_configs(repackRepo, token)¶: For a partner repository, retrieve all the repack.cfg files and parse them into a dict

gecko_taskgraph.util.partners.get_repack_ids_by_platform(config, build_platform)¶

gecko_taskgraph.util.partners.get_repo_params(repo)¶: Parse the organisation and repo name from an https or git url for a repo

gecko_taskgraph.util.partners.get_token(params)¶: We use a Personal Access Token from Github to lookup partner config. No extra scopes are needed on the token to read public repositories, but need the ‘repo’ scope to see private repositories. This is not fine grained and also grants r/w access, but is revoked at the repo level.

gecko_taskgraph.util.partners.locales_per_build_platform(build_platform, locales)¶

gecko_taskgraph.util.partners.parse_config(data)¶

Parse a single repack.cfg file into a python dictionary.: data is contents of the file, in “foo=bar
baz=buzz” style. We do some translation on: locales and platforms data, otherwise passthrough

gecko_taskgraph.util.partners.query_api(query, token)¶: Make a query with a Github auth header, returning the json

gecko_taskgraph.util.perfile module¶

gecko_taskgraph.util.perfile.perfile_number_of_chunks(is_try, try_task_config, head_repository, head_rev, type)¶

gecko_taskgraph.util.platforms module¶

gecko_taskgraph.util.platforms.architecture(build_platform)¶

gecko_taskgraph.util.platforms.archive_format(build_platform)¶: Given a build platform, return the archive format used on the platform.

gecko_taskgraph.util.platforms.executable_extension(build_platform)¶: Given a build platform, return the executable extension used on the platform.

gecko_taskgraph.util.platforms.platform_family(build_platform)¶: Given a build platform, return the platform family (linux, macosx, etc.)

gecko_taskgraph.util.python_path module¶

gecko_taskgraph.util.python_path.find_object(path)¶

Find a Python object given a path of the form <modulepath>:<objectpath>. Conceptually equivalent to

def find_object(modulepath, objectpath):
import <modulepath> as mod return mod.<objectpath>

gecko_taskgraph.util.python_path.import_sibling_modules(exceptions=None)¶

Import all Python modules that are siblings of the calling module.

Parameters: exceptions (list) – A list of file names to exclude (caller and __init__.py are implicitly excluded).

gecko_taskgraph.util.schema module¶

class gecko_taskgraph.util.schema.Schema(*args, check=True, **kwargs)¶

Bases: voluptuous.schema_builder.Schema

Operates identically to voluptuous.Schema, but applying some taskgraph-specific checks in the process.

extend(*args, **kwargs)¶

Create a new Schema by merging this and the provided schema.

Neither this Schema nor the provided schema are modified. The resulting Schema inherits the required and extra parameters of this, unless overridden.

Both schemas must be dictionary-based.

Parameters

schema – dictionary to extend this Schema with
required – if set, overrides required of this Schema
extra – if set, overrides extra of this Schema

gecko_taskgraph.util.schema.check_schema(schema)¶

gecko_taskgraph.util.schema.optionally_keyed_by(*arguments)¶

Mark a schema value as optionally keyed by any of a number of fields. The schema is the last argument, and the remaining fields are taken to be the field names. For example:

‘some-value’: optionally_keyed_by(
‘test-platform’, ‘build-platform’, Any(‘a’, ‘b’, ‘c’))

The resulting schema will allow nesting of by-test-platform and by-build-platform in either order.

gecko_taskgraph.util.schema.resolve_keyed_by(item, field, item_name, defer=None, enforce_single_match=True, **extra_values)¶

For values which can either accept a literal value, or be keyed by some other attribute of the item, perform that lookup and replacement in-place (modifying item directly). The field is specified using dotted notation to traverse dictionaries.

For example, given item:

job:
    test-platform: linux128
    chunks:
        by-test-platform:
            macosx-10.11/debug: 13
            win.*: 6
            default: 12

a call to resolve_keyed_by(item, ‘job.chunks’, item[‘thing-name’]) would mutate item in-place to:

job:
    test-platform: linux128
    chunks: 12

Items can be nested as deeply as the schema will allow:

chunks:
    by-test-platform:
        win.*:
            by-project:
                ash: ..
                cedar: ..
        linux: 13
        default: 12

Parameters

item (dict) – Object being evaluated.
field (str) – Name of the key to perform evaluation on.
item_name (str) – Used to generate useful error messages.
defer (list) – Allows evaluating a by-* entry at a later time. In the example above it’s possible that the project attribute hasn’t been set yet, in which case we’d want to stop before resolving that subkey and then call this function again later. This can be accomplished by setting defer=[“project”] in this example.
enforce_single_match (bool) – If True (default), each task may only match a single arm of the evaluation.
extra_values (kwargs) – If supplied, represent additional values available for reference from by-<field>.

Returns

item which has also been modified in-place.

Return type

dict

gecko_taskgraph.util.schema.validate_schema(schema, obj, msg_prefix)¶: Validate that object satisfies schema. If not, generate a useful exception beginning with msg_prefix.

gecko_taskgraph.util.scriptworker module¶

Make scriptworker.cot.verify more user friendly by making scopes dynamic.

Scriptworker uses certain scopes to determine which sets of credentials to use. Certain scopes are restricted by branch in chain of trust verification, and are checked again at the script level. This file provides functions to adjust these scopes automatically by project; this makes pushing to try, forking a project branch, and merge day uplifts more user friendly.

In the future, we may adjust scopes by other settings as well, e.g. different scopes for push-to-candidates rather than push-to-releases, even if both happen on mozilla-beta and mozilla-release.

Additional configuration is found in the graph config.

gecko_taskgraph.util.scriptworker.BALROG_ACTIONS = ('submit-locale', 'submit-toplevel', 'schedule', 'v2-submit-locale', 'v2-submit-toplevel')¶

Map balrog scope aliases to sets of projects.

This is a list of list-pairs, for ordering.

gecko_taskgraph.util.scriptworker.BALROG_SCOPE_ALIAS_TO_PROJECT = [['nightly', {'mozilla-central', 'oak', 'comm-central'}], ['beta', {'mozilla-beta', 'comm-beta'}], ['release', {'comm-esr91', 'comm-esr102', 'mozilla-release'}], ['esr91', {'mozilla-esr91'}], ['esr102', {'mozilla-esr102'}]]¶: Map the balrog scope aliases to the actual scopes.

gecko_taskgraph.util.scriptworker.BALROG_SERVER_SCOPES = {'aurora': 'balrog:server:aurora', 'beta': 'balrog:server:beta', 'default': 'balrog:server:dep', 'esr102': 'balrog:server:esr', 'esr91': 'balrog:server:esr', 'nightly': 'balrog:server:nightly', 'release': 'balrog:server:release'}¶: The list of the release promotion phases which we send notifications for

gecko_taskgraph.util.scriptworker.BEETMOVER_ACTION_SCOPES = {'default': 'beetmover:action:push-to-candidates', 'nightly': 'beetmover:action:push-to-nightly', 'nightly-oak': 'beetmover:action:push-to-nightly'}¶: Known balrog actions.

gecko_taskgraph.util.scriptworker.BEETMOVER_BUCKET_SCOPES = {'all-nightly-branches': 'beetmover:bucket:nightly', 'all-release-branches': 'beetmover:bucket:release', 'default': 'beetmover:bucket:dep'}¶: Map the beetmover tasks aliases to the actual action scopes.

gecko_taskgraph.util.scriptworker.BEETMOVER_SCOPE_ALIAS_TO_PROJECT = [['all-nightly-branches', {'mozilla-central', 'oak', 'comm-central'}], ['all-release-branches', {'comm-esr102', 'comm-beta', 'mozilla-release', 'mozilla-esr102', 'mozilla-beta', 'mozilla-esr91', 'comm-esr91'}]]¶: Map the beetmover scope aliases to the actual scopes.

gecko_taskgraph.util.scriptworker.DEVEDITION_SIGNING_CERT_SCOPES = {'beta': 'signing:cert:nightly-signing', 'default': 'signing:cert:dep-signing'}¶: Map beetmover scope aliases to sets of projects.

gecko_taskgraph.util.scriptworker.SIGNING_SCOPE_ALIAS_TO_PROJECT = [['all-nightly-branches', {'mozilla-central', 'oak', 'comm-central'}], ['all-release-branches', {'comm-esr102', 'comm-beta', 'mozilla-release', 'mozilla-esr102', 'mozilla-beta', 'mozilla-esr91', 'comm-esr91'}]]¶: Map the signing scope aliases to the actual scopes.

gecko_taskgraph.util.scriptworker.add_scope_prefix(config, scope)¶

Prepends the scriptworker scope prefix from the graph config.

Parameters

config (TransformConfig) – The configuration for the kind being transformed.
scope (string) – The suffix of the scope

Returns

the scope to use.

Return type

string

gecko_taskgraph.util.scriptworker.generate_beetmover_artifact_map(config, job, **kwargs)¶

Generate the beetmover artifact map.

Currently only applies to beetmover tasks.

Parameters

() (config) – Current taskgraph configuration.
job (dict) – The current job being generated

Common kwargs:: platform (str): The current build platform locale (str): The current locale being beetmoved.

Returns

A list of dictionaries containing source->destination: maps for beetmover.

Return type

list

gecko_taskgraph.util.scriptworker.generate_beetmover_partials_artifact_map(config, job, partials_info, **kwargs)¶

Generate the beetmover partials artifact map.

Currently only applies to beetmover tasks.

Parameters

() (config) – Current taskgraph configuration.
job (dict) – The current job being generated
partials_info (dict) – Current partials and information about them in a dict

Common kwargs:: platform (str): The current build platform locale (str): The current locale being beetmoved.

Returns

A list of dictionaries containing source->destination: maps for beetmover.

Return type

list

gecko_taskgraph.util.scriptworker.generate_beetmover_upstream_artifacts(config, job, platform, locale=None, dependencies=None, **kwargs)¶

Generate the upstream artifacts for beetmover, using the artifact map.

Currently only applies to beetmover tasks.

Parameters

job (dict) – The current job being generated
dependencies (list) – A list of the job’s dependency labels.
platform (str) – The current build platform
locale (str) – The current locale being beetmoved.

Returns

A list of dictionaries conforming to the upstream_artifacts spec.

Return type

list

gecko_taskgraph.util.scriptworker.get_balrog_server_scope(config, *, alias_to_project_map=[['nightly', {'mozilla-central', 'oak', 'comm-central'}], ['beta', {'mozilla-beta', 'comm-beta'}], ['release', {'comm-esr91', 'comm-esr102', 'mozilla-release'}], ['esr91', {'mozilla-esr91'}], ['esr102', {'mozilla-esr102'}]], alias_to_scope_map={'aurora': 'balrog:server:aurora', 'beta': 'balrog:server:beta', 'default': 'balrog:server:dep', 'esr102': 'balrog:server:esr', 'esr91': 'balrog:server:esr', 'nightly': 'balrog:server:nightly', 'release': 'balrog:server:release'})¶

Determine the restricted scope from config.params[‘project’].

Parameters

config (TransformConfig) – The configuration for the kind being transformed.
alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
alias_to_scope_map (dict) – the alias alias to scope

Returns

the scope to use.

Return type

string

gecko_taskgraph.util.scriptworker.get_beetmover_action_scope(config, *, release_type_to_scope_map={'default': 'beetmover:action:push-to-candidates', 'nightly': 'beetmover:action:push-to-nightly', 'nightly-oak': 'beetmover:action:push-to-nightly'})¶

Determine the restricted scope from config.params[‘target_tasks_method’].

Parameters

config (TransformConfig) – The configuration for the kind being transformed.
release_type_to_scope_map (dict) – the maps release types to scopes

Returns

the scope to use.

Return type

string

gecko_taskgraph.util.scriptworker.get_beetmover_bucket_scope(config, *, alias_to_project_map=[['all-nightly-branches', {'mozilla-central', 'oak', 'comm-central'}], ['all-release-branches', {'comm-esr102', 'comm-beta', 'mozilla-release', 'mozilla-esr102', 'mozilla-beta', 'mozilla-esr91', 'comm-esr91'}]], alias_to_scope_map={'all-nightly-branches': 'beetmover:bucket:nightly', 'all-release-branches': 'beetmover:bucket:release', 'default': 'beetmover:bucket:dep'})¶

Determine the restricted scope from config.params[‘project’].

Parameters

config (TransformConfig) – The configuration for the kind being transformed.
alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
alias_to_scope_map (dict) – the alias alias to scope

Returns

the scope to use.

Return type

string

gecko_taskgraph.util.scriptworker.get_devedition_signing_cert_scope(config, *, alias_to_project_map=[['beta', {'mozilla-beta'}]], alias_to_scope_map={'beta': 'signing:cert:nightly-signing', 'default': 'signing:cert:dep-signing'})¶

Determine the restricted scope from config.params[‘project’].

Parameters

config (TransformConfig) – The configuration for the kind being transformed.
alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
alias_to_scope_map (dict) – the alias alias to scope

Returns

the scope to use.

Return type

string

gecko_taskgraph.util.scriptworker.get_phase_from_target_method(config, alias_to_tasks_map, alias_to_phase_map)¶

Determine the phase from config.params[‘target_tasks_method’].

Parameters

config (TransformConfig) – The configuration for the kind being transformed.
alias_to_tasks_map (list of lists) – each list pair contains the alias and the set of target methods that match. This is ordered.
alias_to_phase_map (dict) – the alias to phase map

Returns

the phase to use.

Return type

string

gecko_taskgraph.util.scriptworker.get_release_config(config)¶

Get the build number and version for a release task.

Currently only applies to beetmover tasks.

Parameters

config (TransformConfig) – The configuration for the kind being transformed.

Returns

containing both build_number and version. This can be used to: update task.payload.

Return type

dict

gecko_taskgraph.util.scriptworker.get_scope_from_project(config, alias_to_project_map, alias_to_scope_map)¶

Determine the restricted scope from config.params[‘project’].

Parameters

config (TransformConfig) – The configuration for the kind being transformed.
alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
alias_to_scope_map (dict) – the alias alias to scope

Returns

the scope to use.

Return type

string

gecko_taskgraph.util.scriptworker.get_scope_from_release_type(config, release_type_to_scope_map)¶

Determine the restricted scope from config.params[‘target_tasks_method’].

Parameters

config (TransformConfig) – The configuration for the kind being transformed.
release_type_to_scope_map (dict) – the maps release types to scopes

Returns

the scope to use.

Return type

string

gecko_taskgraph.util.scriptworker.get_signing_cert_scope(config, *, alias_to_project_map=[['all-nightly-branches', {'mozilla-central', 'oak', 'comm-central'}], ['all-release-branches', {'comm-esr102', 'comm-beta', 'mozilla-release', 'mozilla-esr102', 'mozilla-beta', 'mozilla-esr91', 'comm-esr91'}]], alias_to_scope_map={'all-nightly-branches': 'signing:cert:nightly-signing', 'all-release-branches': 'signing:cert:release-signing', 'default': 'signing:cert:dep-signing'})¶

Determine the restricted scope from config.params[‘project’].

Parameters

config (TransformConfig) – The configuration for the kind being transformed.
alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
alias_to_scope_map (dict) – the alias alias to scope

Returns

the scope to use.

Return type

string

gecko_taskgraph.util.scriptworker.get_signing_cert_scope_per_platform(build_platform, is_shippable, config)¶

gecko_taskgraph.util.scriptworker.with_scope_prefix(f)¶

Wraps a function, calling add_scope_prefix() on the result of calling the wrapped function.

Parameters: f (callable) – A function that takes a config and some keyword arguments, and returns a scope suffix.
Returns: the wrapped function
Return type: callable

gecko_taskgraph.util.signed_artifacts module¶

Defines artifacts to sign before repackage.

gecko_taskgraph.util.signed_artifacts.generate_specifications_of_artifacts_to_sign(config, job, keep_locale_template=True, kind=None, dep_kind=None)¶

gecko_taskgraph.util.signed_artifacts.get_geckoview_artifacts_to_sign(config, job)¶

gecko_taskgraph.util.signed_artifacts.get_signed_artifacts(input, formats, behavior=None)¶: Get the list of signed artifacts for the given input and formats.

gecko_taskgraph.util.signed_artifacts.is_notarization_kind(kind)¶

gecko_taskgraph.util.signed_artifacts.is_partner_kind(kind)¶

gecko_taskgraph.util.taskcluster module¶

gecko_taskgraph.util.taskcluster.insert_index(index_path, task_id, data=None, use_proxy=False)¶

gecko_taskgraph.util.taskcluster.list_task_group_incomplete_task_ids(task_group_id)¶

gecko_taskgraph.util.taskcluster.list_task_group_tasks(task_group_id)¶: Generate the tasks in a task group

gecko_taskgraph.util.taskcluster.state_task(task_id, use_proxy=False)¶

Gets the state of a task given a task_id.

In testing mode, just logs that it would have retrieved state. This is a subset of the data returned by status_task().

Parameters

task_id (str) – A task id.
use_proxy (bool) – Whether to use taskcluster-proxy (default: False)

Returns

The state of the task, one of: pending, running, completed, failed, exception, unknown.

Return type

str

gecko_taskgraph.util.taskcluster.status_task(task_id, use_proxy=False)¶

Gets the status of a task given a task_id.

In testing mode, just logs that it would have retrieved status.

Parameters

task_id (str) – A task id.
use_proxy (bool) – Whether to use taskcluster-proxy (default: False)

Returns

A dictionary object as defined here:: https://docs.taskcluster.net/docs/reference/platform/queue/api#status

Return type

dict

gecko_taskgraph.util.taskcluster.trigger_hook(hook_group_id, hook_id, hook_payload)¶

gecko_taskgraph.util.taskgraph module¶

Tools for interacting with existing taskgraphs.

gecko_taskgraph.util.taskgraph.find_decision_task(parameters, graph_config)¶: Given the parameters for this action, find the taskId of the decision task

gecko_taskgraph.util.taskgraph.find_existing_tasks(previous_graph_ids)¶

gecko_taskgraph.util.taskgraph.find_existing_tasks_from_previous_kinds(full_task_graph, previous_graph_ids, rebuild_kinds)¶: Given a list of previous decision/action taskIds and kinds to ignore from the previous graphs, return a dictionary of labels-to-taskids to use as existing_tasks in the optimization step.

gecko_taskgraph.util.templates module¶

gecko_taskgraph.util.templates.merge(*objects)¶

Merge the given objects, using the semantics described for merge_to, with objects later in the list taking precedence. From an inheritance perspective, “parents” should be listed before “children”.

Returns the result without modifying any arguments.

gecko_taskgraph.util.templates.merge_to(source, dest)¶

Merge dict and arrays (override scalar values)

Keys from source override keys from dest, and elements from lists in source are appended to lists in dest.

Parameters

source (dict) – to copy from
dest (dict) – to copy to (modified in place)

gecko_taskgraph.util.treeherder module¶

gecko_taskgraph.util.treeherder.add_suffix(treeherder_symbol, suffix)¶: Add a suffix to a treeherder symbol that may contain a group.

gecko_taskgraph.util.treeherder.inherit_treeherder_from_dep(job, dep_job)¶: Inherit treeherder defaults from dep_job

gecko_taskgraph.util.treeherder.join_symbol(group, symbol)¶: Perform the reverse of split_symbol, combining the given group and symbol. If the group is ‘?’, then it is omitted.

gecko_taskgraph.util.treeherder.replace_group(treeherder_symbol, new_group)¶: Add a suffix to a treeherder symbol that may contain a group.

gecko_taskgraph.util.treeherder.split_symbol(treeherder_symbol)¶: Split a symbol expressed as grp(sym) into its two parts. If no group is given, the returned group is ‘?’

gecko_taskgraph.util.verify module¶

class gecko_taskgraph.util.verify.DocPaths(paths=NOTHING)¶

Bases: object

add(path)¶: Projects that make use of Firefox’s taskgraph can extend it with their own task kinds by registering additional paths for documentation. documentation_paths.add() needs to be called by the project’s Taskgraph registration function. See taskgraph.config.

get_files(filename)¶

gecko_taskgraph.util.verify.verify_always_optimized(task, taskgraph, scratch_pad, graph_config, parameters)¶: This function ensures that always-optimized tasks have been optimized.

gecko_taskgraph.util.verify.verify_attributes(task, taskgraph, scratch_pad, graph_config, parameters)¶

gecko_taskgraph.util.verify.verify_dependency_tiers(task, taskgraph, scratch_pad, graph_config, parameters)¶

gecko_taskgraph.util.verify.verify_docs(filename, identifiers, appearing_as)¶: Look for identifiers of the type appearing_as in the files returned by documentation_paths.get_files(). Firefox will have a single file in a list, but projects such as Thunderbird can have documentation in another location and may return multiple files.

gecko_taskgraph.util.verify.verify_kinds_docs(kinds)¶

gecko_taskgraph.util.verify.verify_local_toolchains(task, taskgraph, scratch_pad, graph_config, parameters)¶

Toolchains that are used for local development need to be built on a level-3 branch to installable via mach bootstrap. We ensure here that all such tasks run on at least trunk projects, even if they aren’t pulled in as a dependency of other tasks in the graph.

There is code in mach artifact toolchain that verifies that anything installed via mach bootstrap has the attribute set.

gecko_taskgraph.util.verify.verify_parameters_docs(parameters)¶

gecko_taskgraph.util.verify.verify_required_signoffs(task, taskgraph, scratch_pad, graph_config, parameters)¶: Task with required signoffs can’t be dependencies of tasks with less required signoffs.

gecko_taskgraph.util.verify.verify_routes_notification_filters(task, taskgraph, scratch_pad, graph_config, parameters)¶

This function ensures that only understood filters for notifications are specified.

See: https://firefox-ci-tc.services.mozilla.com/docs/manual/using/task-notifications

gecko_taskgraph.util.verify.verify_run_known_projects(task, taskgraph, scratch_pad, graph_config, parameters)¶

Validates the inputs in run-on-projects.

We should never let ‘try’ (or ‘try-comm-central’) be in run-on-projects even though it is valid because it is not considered for try pushes. While here we also validate for other unknown projects or typos.

gecko_taskgraph.util.verify.verify_run_using()¶

gecko_taskgraph.util.verify.verify_shippable_no_sccache(task, taskgraph, scratch_pad, graph_config, parameters)¶

gecko_taskgraph.util.verify.verify_task_graph_symbol(task, taskgraph, scratch_pad, graph_config, parameters)¶: This function verifies that tuple (collection.keys(), machine.platform, groupSymbol, symbol) is unique for a target task graph.

gecko_taskgraph.util.verify.verify_test_packaging(task, taskgraph, scratch_pad, graph_config, parameters)¶

gecko_taskgraph.util.verify.verify_toolchain_alias(task, taskgraph, scratch_pad, graph_config, parameters)¶: This function verifies that toolchain aliases are not reused.

gecko_taskgraph.util.verify.verify_trust_domain_v2_routes(task, taskgraph, scratch_pad, graph_config, parameters)¶: This function ensures that any two tasks have distinct index.{trust-domain}.v2 routes.

gecko_taskgraph.util.workertypes module¶

gecko_taskgraph.util.workertypes.get_worker_type(graph_config, worker_type, level, release_level)¶: Get the worker type provisioner and worker-type, optionally evaluating aliases from the graph config.

gecko_taskgraph.util.workertypes.worker_type_implementation(graph_config, worker_type)¶: Get the worker implementation and OS for the given workerType, where the OS represents the host system, not the target OS, in the case of cross-compiles.

gecko_taskgraph.util package¶

Submodules¶

gecko_taskgraph.util.attributes module¶

gecko_taskgraph.util.backstop module¶

gecko_taskgraph.util.bugbug module¶

gecko_taskgraph.util.cached_tasks module¶

gecko_taskgraph.util.chunking module¶

gecko_taskgraph.util.declarative_artifacts module¶

gecko_taskgraph.util.docker module¶

gecko_taskgraph.util.hash module¶

gecko_taskgraph.util.hg module¶

gecko_taskgraph.util.keyed_by module¶

gecko_taskgraph.util.partials module¶

gecko_taskgraph.util.partners module¶

gecko_taskgraph.util.perfile module¶

gecko_taskgraph.util.platforms module¶

gecko_taskgraph.util.python_path module¶

gecko_taskgraph.util.schema module¶

gecko_taskgraph.util.scriptworker module¶

gecko_taskgraph.util.signed_artifacts module¶

gecko_taskgraph.util.taskcluster module¶

gecko_taskgraph.util.taskgraph module¶

gecko_taskgraph.util.templates module¶

gecko_taskgraph.util.treeherder module¶

gecko_taskgraph.util.verify module¶

gecko_taskgraph.util.workertypes module¶

Module contents¶