taskgraph.transforms package

Submodules

taskgraph.transforms.balrog_submit module

Transform the per-locale balrog task into an actual task description.

taskgraph.transforms.balrog_submit.get_balrog_server_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.balrog_submit.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc.

taskgraph.transforms.balrog_submit.make_task_description(config, jobs)

taskgraph.transforms.balrog_toplevel module

Transform the beetmover task into an actual task description.

taskgraph.transforms.balrog_toplevel.generate_update_line(config, jobs)

Resolve fields that can be keyed by platform, etc.

taskgraph.transforms.base module

class taskgraph.transforms.base.TransformConfig(kind, path, config, params, kind_dependencies_tasks, graph_config)

Bases: object

A container for configuration affecting transforms. The config argument to transforms is an instance of this class.

class taskgraph.transforms.base.TransformSequence(transforms=NOTHING)

Bases: object

Container for a sequence of transforms. Each transform is represented as a callable taking (config, items) and returning a generator which will yield transformed items. The resulting sequence has the same interface.

This is convenient to use in a file full of transforms, as it provides a decorator, @transforms.add, that will add the decorated function to the sequence.

add(func)
add_validate(schema)
class taskgraph.transforms.base.ValidateSchema(schema)

Bases: object

taskgraph.transforms.beetmover module

Transform the beetmover task into an actual task description.

taskgraph.transforms.beetmover.craft_release_properties(config, job)
taskgraph.transforms.beetmover.get_beetmover_action_scope()

Determine the restricted scope from config.params[‘target_tasks_method’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • release_type_to_scope_map (dict) – the maps release types to scopes
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover.get_beetmover_bucket_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover.make_task_description(config, jobs)
taskgraph.transforms.beetmover.make_task_worker(config, jobs)

taskgraph.transforms.beetmover_checksums module

Transform the checksums signing task into an actual task description.

taskgraph.transforms.beetmover_checksums.get_beetmover_action_scope()

Determine the restricted scope from config.params[‘target_tasks_method’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • release_type_to_scope_map (dict) – the maps release types to scopes
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover_checksums.get_beetmover_bucket_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover_checksums.make_beetmover_checksums_description(config, jobs)
taskgraph.transforms.beetmover_checksums.make_beetmover_checksums_worker(config, jobs)

taskgraph.transforms.beetmover_emefree_checksums module

Transform release-beetmover-source-checksums into an actual task description.

taskgraph.transforms.beetmover_emefree_checksums.generate_upstream_artifacts(refs, partner_path)
taskgraph.transforms.beetmover_emefree_checksums.make_beetmover_checksums_description(config, jobs)
taskgraph.transforms.beetmover_emefree_checksums.make_beetmover_checksums_worker(config, jobs)

taskgraph.transforms.beetmover_geckoview module

Transform the beetmover task into an actual task description.

taskgraph.transforms.beetmover_geckoview.craft_release_properties(config, job)
taskgraph.transforms.beetmover_geckoview.make_task_description(config, jobs)
taskgraph.transforms.beetmover_geckoview.make_task_worker(config, jobs)
taskgraph.transforms.beetmover_geckoview.resolve_keys(config, jobs)

taskgraph.transforms.beetmover_l10n module

Transform the signing task into an actual task description.

taskgraph.transforms.beetmover_l10n.make_beetmover_description(config, jobs)

taskgraph.transforms.beetmover_langpack_checksums module

Transform release-beetmover-langpack-checksums into an actual task description.

taskgraph.transforms.beetmover_langpack_checksums.get_beetmover_action_scope()

Determine the restricted scope from config.params[‘target_tasks_method’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • release_type_to_scope_map (dict) – the maps release types to scopes
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover_langpack_checksums.get_beetmover_bucket_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover_langpack_checksums.make_beetmover_checksums_description(config, jobs)
taskgraph.transforms.beetmover_langpack_checksums.make_beetmover_checksums_worker(config, jobs)

taskgraph.transforms.beetmover_push_to_release module

Transform the beetmover-push-to-release task into a task description.

taskgraph.transforms.beetmover_push_to_release.get_beetmover_bucket_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover_push_to_release.make_beetmover_push_to_release_description(config, jobs)
taskgraph.transforms.beetmover_push_to_release.make_beetmover_push_to_release_worker(config, jobs)

taskgraph.transforms.beetmover_repackage module

Transform the beetmover task into an actual task description.

taskgraph.transforms.beetmover_repackage.convert_deps(config, jobs)
taskgraph.transforms.beetmover_repackage.generate_partials_upstream_artifacts(job, artifacts, platform, locale=None)
taskgraph.transforms.beetmover_repackage.get_beetmover_action_scope()

Determine the restricted scope from config.params[‘target_tasks_method’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • release_type_to_scope_map (dict) – the maps release types to scopes
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover_repackage.get_beetmover_bucket_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover_repackage.make_partials_artifacts(config, jobs)
taskgraph.transforms.beetmover_repackage.make_task_description(config, jobs)
taskgraph.transforms.beetmover_repackage.make_task_worker(config, jobs)
taskgraph.transforms.beetmover_repackage.strip_unwanted_langpacks_from_worker(config, jobs)

Strips out langpacks where we didn’t sign them.

This explicitly deletes langpacks from upstream artifacts and from artifact-maps. Due to limitations in declarative artifacts, doing this was our easiest way right now.

taskgraph.transforms.beetmover_repackage_l10n module

Transform the signing task into an actual task description.

taskgraph.transforms.beetmover_repackage_l10n.make_beetmover_description(config, jobs)

taskgraph.transforms.beetmover_repackage_partner module

Transform the beetmover task into an actual task description.

taskgraph.transforms.beetmover_repackage_partner.generate_upstream_artifacts(job, build_task_ref, repackage_task_ref, repackage_signing_task_ref, platform, repack_id, partner_path, repack_stub_installer=False)
taskgraph.transforms.beetmover_repackage_partner.get_beetmover_bucket_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover_repackage_partner.make_task_description(config, jobs)
taskgraph.transforms.beetmover_repackage_partner.make_task_worker(config, jobs)
taskgraph.transforms.beetmover_repackage_partner.populate_scopes_and_worker_type(config, job, bucket_scope, partner_public=False)
taskgraph.transforms.beetmover_repackage_partner.resolve_keys(config, jobs)
taskgraph.transforms.beetmover_repackage_partner.split_public_and_private(config, jobs)

taskgraph.transforms.beetmover_snap module

Transform the snap beetmover kind into an actual task description.

taskgraph.transforms.beetmover_snap.leave_snap_repackage_dependencies_only(config, jobs)
taskgraph.transforms.beetmover_snap.set_custom_treeherder_job_name(config, jobs)

taskgraph.transforms.beetmover_source module

Transform the beetmover-source task to also append build as dependency

taskgraph.transforms.beetmover_source.remove_build_dependency_in_beetmover_source(config, jobs)

taskgraph.transforms.beetmover_source_checksums module

Transform release-beetmover-source-checksums into an actual task description.

taskgraph.transforms.beetmover_source_checksums.get_beetmover_action_scope()

Determine the restricted scope from config.params[‘target_tasks_method’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • release_type_to_scope_map (dict) – the maps release types to scopes
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover_source_checksums.get_beetmover_bucket_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.beetmover_source_checksums.make_beetmover_checksums_description(config, jobs)
taskgraph.transforms.beetmover_source_checksums.make_beetmover_checksums_worker(config, jobs)

taskgraph.transforms.bouncer_aliases module

Add from parameters.yml into bouncer submission tasks.

taskgraph.transforms.bouncer_aliases.craft_bouncer_entries(config, job)
taskgraph.transforms.bouncer_aliases.make_task_worker(config, jobs)

taskgraph.transforms.bouncer_check module

taskgraph.transforms.bouncer_check.add_command(config, jobs)
taskgraph.transforms.bouncer_check.add_previous_versions(config, jobs)
taskgraph.transforms.bouncer_check.command_to_string(config, jobs)

Convert command to string to make it work properly with run-task

taskgraph.transforms.bouncer_check.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by project, etc.

taskgraph.transforms.bouncer_locations module

taskgraph.transforms.bouncer_locations.make_task_worker(config, jobs)

taskgraph.transforms.bouncer_submission module

Add from parameters.yml into bouncer submission tasks.

class taskgraph.transforms.bouncer_submission.InvalidSubstitution(error)

Bases: object

taskgraph.transforms.bouncer_submission.craft_add_locales(product)
taskgraph.transforms.bouncer_submission.craft_bouncer_entries(config, job)
taskgraph.transforms.bouncer_submission.craft_bouncer_product_name(product, bouncer_product, current_version, current_build_number=None, previous_version=None)
taskgraph.transforms.bouncer_submission.craft_paths_per_bouncer_platform(product, bouncer_product, bouncer_platforms, current_version, current_build_number, previous_version=None)
taskgraph.transforms.bouncer_submission.craft_ssl_only(bouncer_product, project)
taskgraph.transforms.bouncer_submission.make_task_worker(config, jobs)
taskgraph.transforms.bouncer_submission.split_build_data(version)

taskgraph.transforms.bouncer_submission_partners module

Add from parameters.yml into bouncer submission tasks.

taskgraph.transforms.bouncer_submission_partners.craft_bouncer_entries(config, job)
taskgraph.transforms.bouncer_submission_partners.craft_partner_bouncer_product_name(product, bouncer_product, current_version, partner, sub_config)
taskgraph.transforms.bouncer_submission_partners.craft_paths_per_bouncer_platform(product, bouncer_product, bouncer_platforms, current_version, partner, sub_config)
taskgraph.transforms.bouncer_submission_partners.craft_ssl_only(bouncer_product)
taskgraph.transforms.bouncer_submission_partners.make_task_worker(config, jobs)

taskgraph.transforms.build module

Apply some defaults and minor modifications to the jobs defined in the build kind.

taskgraph.transforms.build.enable_full_crashsymbols(config, jobs)

Enable full crashsymbols on jobs with ‘enable-full-crashsymbols’ set to True and on release branches, or on try

taskgraph.transforms.build.mozconfig(config, jobs)
taskgraph.transforms.build.resolve_shipping_product(config, jobs)
taskgraph.transforms.build.set_defaults(config, jobs)

Set defaults, including those that differ per worker implementation

taskgraph.transforms.build.set_env(config, jobs)

Set extra environment variables from try command line.

taskgraph.transforms.build.stub_installer(config, jobs)
taskgraph.transforms.build.update_channel(config, jobs)
taskgraph.transforms.build.use_artifact(config, jobs)
taskgraph.transforms.build.use_profile_data(config, jobs)

taskgraph.transforms.build_attrs module

taskgraph.transforms.build_attrs.set_build_attributes(config, jobs)

Set the build_platform and build_type attributes based on the job name. Although not all jobs using this transform are actual “builds”, the try option syntax treats them as such, and this arranges the attributes appropriately for that purpose.

taskgraph.transforms.build_attrs.set_schedules_optimization(config, jobs)

Set the skip-unless-affected optimization based on the build platform.

taskgraph.transforms.build_fat_aar module

taskgraph.transforms.build_fat_aar.set_fetches_and_locations(config, jobs)

Set defaults, including those that differ per worker implementation

taskgraph.transforms.build_lints module

Apply some defaults and minor modifications to the jobs defined in the build kind.

taskgraph.transforms.build_lints.check_mozharness_perfherder_options(config, jobs)

Verify that multiple jobs don’t use the same perfherder bucket.

Build jobs record perfherder metrics by default. Perfherder metrics go to a bucket derived by the platform by default. The name can further be customized by the presence of “extra options” either defined in mozharness sub-configs or in an environment variable.

This linter tries to verify that no 2 jobs will send Perfherder metrics to the same bucket by looking for jobs not defining extra options when their platform or mozharness config are otherwise similar.

taskgraph.transforms.build_signing module

Transform the signing task into an actual task description.

taskgraph.transforms.build_signing.add_signed_routes(config, jobs)

Add routes corresponding to the routes of the build task this corresponds to, with .signed inserted, for all gecko.v2 routes

taskgraph.transforms.build_signing.define_upstream_artifacts(config, jobs)

taskgraph.transforms.cached_tasks module

taskgraph.transforms.cached_tasks.cache_task(config, tasks)
taskgraph.transforms.cached_tasks.format_task_digest(cached_task)
taskgraph.transforms.cached_tasks.order_tasks(config, tasks)

Iterate image tasks in an order where parent tasks come first.

taskgraph.transforms.chunk_partners module

Chunk the partner repack tasks by subpartner and locale

taskgraph.transforms.chunk_partners.chunk_partners(config, jobs)

taskgraph.transforms.coalesce module

This transform enables coalescing for tasks with scm level > 1, by defining values for the coalesce settings of the job.

taskgraph.transforms.coalesce.enable_coalescing(config, jobs)

This transform enables coalescing for tasks with scm level > 1, setting the name to be equal to the the first 20 chars of the sha256 of the task name (label). The hashing is simply used to keep the name short, unique, and alphanumeric, since it is used in AMQP routing keys.

Note, the coalescing keys themselves are not intended to be readable strings or embed information. The tasks they represent contain all relevant metadata.

taskgraph.transforms.code_review module

Add soft dependencies and configuration to code-review tasks.

taskgraph.transforms.code_review.add_dependencies(config, jobs)
taskgraph.transforms.code_review.add_phabricator_config(config, jobs)

taskgraph.transforms.copy_attributes_from_dependent_task module

Transform the repackage task into an actual task description.

taskgraph.transforms.copy_attributes_from_dependent_task.copy_attributes(config, jobs)

taskgraph.transforms.diffoscope module

This transform construct tasks to perform diffs between builds, as defined in kind.yml

taskgraph.transforms.diffoscope.fill_template(config, tasks)

taskgraph.transforms.docker_image module

taskgraph.transforms.docker_image.fill_template(config, tasks)
taskgraph.transforms.docker_image.order_image_tasks(config, tasks)

Iterate image tasks in an order where parent images come first.

taskgraph.transforms.fetch module

taskgraph.transforms.fetch.create_chromium_fetch_task(config, job)
taskgraph.transforms.fetch.create_fetch_url_task(config, job)
taskgraph.transforms.fetch.create_git_fetch_task(config, job)
taskgraph.transforms.fetch.make_base_task(config, name, description, command)
taskgraph.transforms.fetch.process_fetch_job(config, jobs)

taskgraph.transforms.final_verify module

Transform the beetmover task into an actual task description.

taskgraph.transforms.final_verify.add_command(config, tasks)

taskgraph.transforms.geckodriver_signing module

Transform the repackage signing task into an actual task description.

taskgraph.transforms.geckodriver_signing.make_repackage_signing_description(config, jobs)

taskgraph.transforms.google_play_strings module

Transform the push-apk kind into an actual task description.

taskgraph.transforms.google_play_strings.set_label(config, jobs)
taskgraph.transforms.google_play_strings.set_worker_data(config, jobs)

taskgraph.transforms.l10n module

Do transforms specific to l10n kind

taskgraph.transforms.l10n.all_locales_attribute(config, jobs)
taskgraph.transforms.l10n.chunk_locales(config, jobs)

Utilizes chunking for l10n stuff

taskgraph.transforms.l10n.copy_in_useful_magic(config, jobs)
taskgraph.transforms.l10n.handle_artifact_prefix(config, jobs)

Resolve artifact_prefix in env vars

taskgraph.transforms.l10n.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc.

taskgraph.transforms.l10n.make_job_description(config, jobs)
taskgraph.transforms.l10n.parse_locales_file(locales_file, platform=None)

Parse the passed locales file for a list of locales.

taskgraph.transforms.l10n.set_extra_config(config, jobs)
taskgraph.transforms.l10n.setup_name(config, jobs)
taskgraph.transforms.l10n.setup_nightly_dependency(config, jobs)

Sets up a task dependency to the signing job this relates to

taskgraph.transforms.l10n.stub_installer(config, jobs)

taskgraph.transforms.mar_signing module

Transform the {partials,mar}-signing task into an actual task description.

taskgraph.transforms.mar_signing.generate_complete_artifacts(job, kind)
taskgraph.transforms.mar_signing.generate_partials_artifacts(job, release_history, platform, locale=None)
taskgraph.transforms.mar_signing.make_task_description(config, jobs)

taskgraph.transforms.name_sanity module

Generate labels for tasks without names, consistently. Uses attributes from primary-dependency.

taskgraph.transforms.name_sanity.make_label(config, jobs)

Generate a sane label for a new task constructed from a dependency Using attributes from the dependent job and the current task kind

taskgraph.transforms.nightly_l10n_signing module

Transform the signing task into an actual task description.

taskgraph.transforms.nightly_l10n_signing.define_upstream_artifacts(config, jobs)
taskgraph.transforms.nightly_l10n_signing.make_signing_description(config, jobs)

taskgraph.transforms.openh264 module

This transform is used to help populate mozharness options for openh264 jobs

taskgraph.transforms.openh264.set_mh_options(config, jobs)

This transform enables coalescing for tasks with scm level > 1, setting the name to be equal to the the first 20 chars of the sha256 of the task name (label). The hashing is simply used to keep the name short, unique, and alphanumeric, since it is used in AMQP routing keys.

Note, the coalescing keys themselves are not intended to be readable strings or embed information. The tasks they represent contain all relevant metadata.

taskgraph.transforms.openh264_signing module

Transform the repackage signing task into an actual task description.

taskgraph.transforms.openh264_signing.make_signing_description(config, jobs)

taskgraph.transforms.partials module

Transform the partials task into an actual task description.

taskgraph.transforms.partials.make_task_description(config, jobs)

taskgraph.transforms.partner_repack module

Transform the partner repack task into an actual task description.

taskgraph.transforms.partner_repack.add_command_arguments(config, tasks)
taskgraph.transforms.partner_repack.make_label(config, tasks)
taskgraph.transforms.partner_repack.populate_repack_manifests_url(config, tasks)

taskgraph.transforms.partner_signing module

Transform the signing task into an actual task description.

taskgraph.transforms.partner_signing.define_upstream_artifacts(config, jobs)

taskgraph.transforms.per_platform_dummy module

Transform the repackage task into an actual task description.

taskgraph.transforms.per_platform_dummy.one_task_per_product_and_platform(config, jobs)

taskgraph.transforms.pipfile_update module

Transform the repo-update task into an actual task description.

taskgraph.transforms.pipfile_update.resolve_keys(config, tasks)

taskgraph.transforms.push_apk module

Transform the push-apk kind into an actual task description.

taskgraph.transforms.push_apk.check_every_architecture_is_present_in_dependent_tasks(project, dependent_tasks)
taskgraph.transforms.push_apk.delete_non_required_fields(_, jobs)
taskgraph.transforms.push_apk.generate_dependencies(dependent_tasks)
taskgraph.transforms.push_apk.generate_upstream_artifacts(job, dependencies)
taskgraph.transforms.push_apk.get_push_apk_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.push_apk.make_task_description(config, jobs)
taskgraph.transforms.push_apk.validate_dependent_tasks(config, jobs)

taskgraph.transforms.push_apk_checks module

Transform the push-apk-checks kind into an actual task description.

taskgraph.transforms.push_apk_checks.delete_non_required_fields_(config, jobs)
taskgraph.transforms.push_apk_checks.make_task_description(config, jobs)

taskgraph.transforms.release module

Transforms for release tasks

taskgraph.transforms.release.run_on_releases(config, jobs)

Filter out jobs with run-on-releases set, and that don’t match the release_type paramater.

taskgraph.transforms.release_beetmover_signed_addons module

Transform the beetmover task into an actual task description.

taskgraph.transforms.release_beetmover_signed_addons.get_beetmover_action_scope()

Determine the restricted scope from config.params[‘target_tasks_method’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • release_type_to_scope_map (dict) – the maps release types to scopes
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.release_beetmover_signed_addons.get_beetmover_bucket_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.release_beetmover_signed_addons.make_task_description(config, jobs)
taskgraph.transforms.release_beetmover_signed_addons.make_task_worker(config, jobs)
taskgraph.transforms.release_beetmover_signed_addons.resolve_keys(config, jobs)
taskgraph.transforms.release_beetmover_signed_addons.strip_unused_data(config, jobs)
taskgraph.transforms.release_beetmover_signed_addons.yield_all_platform_jobs(config, jobs)

taskgraph.transforms.release_deps module

Add dependencies to release tasks.

taskgraph.transforms.release_deps.add_dependencies(config, jobs)

taskgraph.transforms.release_generate_checksums module

Transform the checksums task into an actual task description.

taskgraph.transforms.release_generate_checksums.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by project, etc.

taskgraph.transforms.release_generate_checksums.interpolate(config, jobs)

taskgraph.transforms.release_generate_checksums_beetmover module

Transform the release-generate-checksums-beetmover task to also append build as dependency

taskgraph.transforms.release_generate_checksums_beetmover.get_beetmover_action_scope()

Determine the restricted scope from config.params[‘target_tasks_method’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • release_type_to_scope_map (dict) – the maps release types to scopes
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.release_generate_checksums_beetmover.get_beetmover_bucket_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.release_generate_checksums_beetmover.make_task_description(config, jobs)
taskgraph.transforms.release_generate_checksums_beetmover.make_task_worker(config, jobs)

taskgraph.transforms.release_generate_checksums_signing module

Transform the release-generate-checksums-signing task into task description.

taskgraph.transforms.release_generate_checksums_signing.get_signing_cert_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.release_generate_checksums_signing.make_release_generate_checksums_signing_description(config, jobs)

taskgraph.transforms.release_mark_as_shipped module

Add from parameters.yml into Balrog publishing tasks.

taskgraph.transforms.release_mark_as_shipped.make_task_description(config, jobs)

taskgraph.transforms.release_notifications module

Add notifications via taskcluster-notify for release tasks

taskgraph.transforms.release_notifications.add_notifications(config, jobs)

taskgraph.transforms.release_sign_and_push_langpacks module

Transform the release-sign-and-push task into an actual task description.

taskgraph.transforms.release_sign_and_push_langpacks.copy_attributes(config, jobs)
taskgraph.transforms.release_sign_and_push_langpacks.filter_out_macos_jobs_but_mac_only_locales(config, jobs)
taskgraph.transforms.release_sign_and_push_langpacks.generate_upstream_artifacts(upstream_task_ref, locales)
taskgraph.transforms.release_sign_and_push_langpacks.get_upstream_task_ref(job, expected_kinds)
taskgraph.transforms.release_sign_and_push_langpacks.make_task_description(config, jobs)
taskgraph.transforms.release_sign_and_push_langpacks.make_task_worker(config, jobs)
taskgraph.transforms.release_sign_and_push_langpacks.resolve_keys(config, jobs)
taskgraph.transforms.release_sign_and_push_langpacks.set_label(config, jobs)
taskgraph.transforms.release_sign_and_push_langpacks.strip_unused_data(config, jobs)

taskgraph.transforms.release_snap_push module

Transform the release-snap-push kind into an actual task description.

taskgraph.transforms.release_snap_push.generate_upstream_artifacts(dependencies)
taskgraph.transforms.release_snap_push.make_task_description(config, jobs)

taskgraph.transforms.release_snap_repackage module

taskgraph.transforms.release_snap_repackage.format(config, tasks)

Apply format substitution to worker.env and worker.command.

taskgraph.transforms.release_started module

Add notifications via taskcluster-notify for release tasks

taskgraph.transforms.release_started.add_notifications(config, jobs)

taskgraph.transforms.release_version_bump module

Transform the update generation task into an actual task description.

taskgraph.transforms.release_version_bump.handle_keyed_by(config, tasks)

Resolve fields that can be keyed by platform, etc.

taskgraph.transforms.repackage module

Transform the repackage task into an actual task description.

taskgraph.transforms.repackage.copy_in_useful_magic(config, jobs)

Copy attributes from upstream task to be used for keyed configuration.

taskgraph.transforms.repackage.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc.

taskgraph.transforms.repackage.make_job_description(config, jobs)
taskgraph.transforms.repackage.make_repackage_description(config, jobs)

taskgraph.transforms.repackage_l10n module

Transform the repackage task into an actual task description.

taskgraph.transforms.repackage_l10n.split_locales(config, jobs)

taskgraph.transforms.repackage_partner module

Transform the repackage task into an actual task description.

taskgraph.transforms.repackage_partner.copy_in_useful_magic(config, jobs)

Copy attributes from upstream task to be used for keyed configuration.

taskgraph.transforms.repackage_partner.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc.

taskgraph.transforms.repackage_partner.make_job_description(config, jobs)
taskgraph.transforms.repackage_partner.make_repackage_description(config, jobs)

taskgraph.transforms.repackage_routes module

Add indexes to repackage kinds

taskgraph.transforms.repackage_routes.add_indexes(config, jobs)

taskgraph.transforms.repackage_signing module

Transform the repackage signing task into an actual task description.

taskgraph.transforms.repackage_signing.make_repackage_signing_description(config, jobs)

taskgraph.transforms.repackage_signing_partner module

Transform the repackage signing task into an actual task description.

taskgraph.transforms.repackage_signing_partner.make_repackage_signing_description(config, jobs)

taskgraph.transforms.repo_update module

Transform the repo-update task into an actual task description.

taskgraph.transforms.repo_update.resolve_keys(config, tasks)

taskgraph.transforms.reverse_chunk_deps module

Adjust dependencies to not exceed MAX_DEPENDENCIES

taskgraph.transforms.reverse_chunk_deps.add_dependencies(config, jobs)
taskgraph.transforms.reverse_chunk_deps.yield_job(orig_job, deps, count)

taskgraph.transforms.run_pgo_profile module

Apply some defaults and minor modifications to the pgo jobs.

taskgraph.transforms.run_pgo_profile.run_profile_data(config, jobs)

taskgraph.transforms.scriptworker module

Transforms for adding appropriate scopes to scriptworker tasks.

taskgraph.transforms.scriptworker.add_balrog_scopes(config, jobs)
taskgraph.transforms.scriptworker.get_balrog_server_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.signing module

Transform the signing task into an actual task description.

taskgraph.transforms.signing.make_task_description(config, jobs)
taskgraph.transforms.signing.set_defaults(config, jobs)

taskgraph.transforms.source_checksums_signing module

Transform the checksums signing task into an actual task description.

taskgraph.transforms.source_checksums_signing.get_signing_cert_scope()

Determine the restricted scope from config.params[‘project’].

Parameters:
  • config (TransformConfig) – The configuration for the kind being transformed.
  • alias_to_project_map (list of lists) – each list pair contains the alias and the set of projects that match. This is ordered.
  • alias_to_scope_map (dict) – the alias alias to scope
Returns:

the scope to use.

Return type:

string

taskgraph.transforms.source_checksums_signing.make_checksums_signing_description(config, jobs)

taskgraph.transforms.source_test module

Source-test jobs can run on multiple platforms. These transforms allow jobs with either platform or a list of platforms, and set the appropriate treeherder configuration and attributes for that platform.

taskgraph.transforms.source_test.add_build_dependency(config, job)

Add build dependency to the job and installer_url to env.

taskgraph.transforms.source_test.add_decision_task_id_to_env(config, jobs)

Creates the DECISION_TASK_ID environment variable in tasks that set the require-decision-task-id config.

taskgraph.transforms.source_test.expand_platforms(config, jobs)
taskgraph.transforms.source_test.handle_platform(config, jobs)

Handle the ‘platform’ property, setting up treeherder context as well as try-related attributes.

taskgraph.transforms.source_test.handle_shell(config, jobs)

Handle the ‘shell’ property.

taskgraph.transforms.source_test.set_defaults(config, jobs)
taskgraph.transforms.source_test.set_job_name(config, jobs)
taskgraph.transforms.source_test.split_jsshell(config, jobs)
taskgraph.transforms.source_test.split_python(config, jobs)

taskgraph.transforms.spidermonkey module

taskgraph.transforms.spidermonkey.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc.

taskgraph.transforms.strip_dependent_task module

FIXME

taskgraph.transforms.strip_dependent_task.strip_dependent_task(config, jobs)

taskgraph.transforms.task module

These transformations take a task description and turn it into a TaskCluster task definition (along with attributes, label, etc.). The input to these transformations is generic to any kind of task, but abstracts away some of the complexities of worker implementations, scopes, and treeherder annotations.

class taskgraph.transforms.task.PayloadBuilder(schema, builder)

Bases: object

taskgraph.transforms.task.add_android_nightly_index_routes(config, task)
taskgraph.transforms.task.add_android_nightly_multi_index_routes(config, task)
taskgraph.transforms.task.add_geckoview_index_routes(config, task)
taskgraph.transforms.task.add_generic_index_routes(config, task)
taskgraph.transforms.task.add_index_routes(config, tasks)
taskgraph.transforms.task.add_l10n_index_routes(config, task, force_locale=None)
taskgraph.transforms.task.add_nightly_index_routes(config, task)
taskgraph.transforms.task.add_nightly_l10n_index_routes(config, task, force_locale=None)
taskgraph.transforms.task.add_nightly_multi_index_routes(config, task)
taskgraph.transforms.task.add_shippable_index_routes(config, task)
taskgraph.transforms.task.add_shippable_l10n_index_routes(config, task, force_locale=None)
taskgraph.transforms.task.build_balrog_payload(config, task, task_def)
taskgraph.transforms.task.build_beetmover_maven_payload(config, task, task_def)
taskgraph.transforms.task.build_beetmover_payload(config, task, task_def)
taskgraph.transforms.task.build_beetmover_push_to_release_payload(config, task, task_def)
taskgraph.transforms.task.build_bouncer_aliases_payload(config, task, task_def)
taskgraph.transforms.task.build_bouncer_locations_payload(config, task, task_def)
taskgraph.transforms.task.build_bouncer_submission_payload(config, task, task_def)
taskgraph.transforms.task.build_docker_worker_payload(config, task, task_def)
taskgraph.transforms.task.build_dummy_payload(config, task, task_def)
taskgraph.transforms.task.build_generic_worker_payload(config, task, task_def)
taskgraph.transforms.task.build_invalid_payload(config, task, task_def)
taskgraph.transforms.task.build_push_addons_payload(config, task, task_def)
taskgraph.transforms.task.build_push_apk_payload(config, task, task_def)
taskgraph.transforms.task.build_push_snap_payload(config, task, task_def)
taskgraph.transforms.task.build_script_engine_autophone_payload(config, task, task_def)
taskgraph.transforms.task.build_scriptworker_signing_payload(config, task, task_def)
taskgraph.transforms.task.build_ship_it_shipped_payload(config, task, task_def)
taskgraph.transforms.task.build_task(config, tasks)
taskgraph.transforms.task.build_treescript_payload(config, task, task_def)
taskgraph.transforms.task.chain_of_trust(config, tasks)
taskgraph.transforms.task.check_caches_are_volumes(task)

Ensures that all cache paths are defined as volumes.

Caches and volumes are the only filesystem locations whose content isn’t defined by the Docker image itself. Some caches are optional depending on the job environment. We want paths that are potentially caches to have as similar behavior regardless of whether a cache is used. To help enforce this, we require that all paths used as caches to be declared as Docker volumes. This check won’t catch all offenders. But it is better than nothing.

taskgraph.transforms.task.check_run_task_caches(config, tasks)

Audit for caches requiring run-task.

run-task manages caches in certain ways. If a cache managed by run-task is used by a non run-task task, it could cause problems. So we audit for that and make sure certain cache names are exclusive to run-task.

IF YOU ARE TEMPTED TO MAKE EXCLUSIONS TO THIS POLICY, YOU ARE LIKELY CONTRIBUTING TECHNICAL DEBT AND WILL HAVE TO SOLVE MANY OF THE PROBLEMS THAT RUN-TASK ALREADY SOLVES. THINK LONG AND HARD BEFORE DOING THAT.

taskgraph.transforms.task.check_task_dependencies(config, tasks)

Ensures that tasks don’t have more than 100 dependencies.

taskgraph.transforms.task.check_task_identifiers(config, tasks)

Ensures that all tasks have well defined identifiers: ^[a-zA-Z0-9_-]{1,38}$

taskgraph.transforms.task.coalesce_key(config, task)
taskgraph.transforms.task.get_branch_repo(config)
taskgraph.transforms.task.get_branch_rev(config)
taskgraph.transforms.task.get_default_priority
taskgraph.transforms.task.index_builder(name)
taskgraph.transforms.task.payload_builder(name, schema)
taskgraph.transforms.task.set_defaults(config, tasks)
taskgraph.transforms.task.set_implementation(config, tasks)

Set the worker implementation based on the worker-type alias.

taskgraph.transforms.task.superseder_url(config, task)
taskgraph.transforms.task.task_name_from_label(config, tasks)
taskgraph.transforms.task.validate(config, tasks)
taskgraph.transforms.task.validate_shipping_product(config, product)
taskgraph.transforms.task.verify_index(config, index)

taskgraph.transforms.tests module

These transforms construct a task description to run the given test, based on a test description. The implementation here is shared among all test kinds, but contains specific support for how we run tests in Gecko (via mozharness, invoked in particular ways).

This is a good place to translate a test-description option such as single-core: true to the implementation of that option in a task description (worker options, mozharness commandline, environment variables, etc.)

The test description should be fully formed by the time it reaches these transforms, and these transforms should not embody any specific knowledge about what should run where. this is the wrong place for special-casing platforms, for example - use all_tests.py instead.

taskgraph.transforms.tests.allow_software_gl_layers(config, tests)

Handle the “allow-software-gl-layers” property for platforms where it applies.

taskgraph.transforms.tests.disable_fennec_e10s(config, tests)
taskgraph.transforms.tests.enable_code_coverage(config, tests)

Enable code coverage for the ccov build-platforms

taskgraph.transforms.tests.enable_fission_on_central(config, tests)

Enable select fission tasks on mozilla-central.

taskgraph.transforms.tests.enable_webrender(config, tests)

Handle the “webrender” property by passing a flag to mozharness if it is enabled.

taskgraph.transforms.tests.ensure_spi_disabled_on_all_but_spi(config, tests)
taskgraph.transforms.tests.fission_filter(test)
taskgraph.transforms.tests.get_firefox_version()
taskgraph.transforms.tests.get_mobile_project(test)

Returns the mobile project of the specified task or None.

taskgraph.transforms.tests.handle_artifact_prefix(config, tests)

Handle translating artifact_prefix appropriately

taskgraph.transforms.tests.handle_keyed_by(config, tests)

Resolve fields that can be keyed by platform, etc.

taskgraph.transforms.tests.handle_keyed_by_mozharness(config, tests)

Resolve a mozharness field if it is keyed by something

taskgraph.transforms.tests.handle_run_on_projects(config, tests)

Handle translating built-projects appropriately

taskgraph.transforms.tests.handle_suite_category(config, tests)
taskgraph.transforms.tests.make_job_description(config, tests)

Convert test descriptions to job descriptions (input to taskgraph.transforms.job)

taskgraph.transforms.tests.normpath(path)
taskgraph.transforms.tests.resolve_keys(config, tests)
taskgraph.transforms.tests.runs_on_central(test)
taskgraph.transforms.tests.set_defaults(config, tests)
taskgraph.transforms.tests.set_download_symbols(config, tests)

In general, we download symbols immediately for debug builds, but only on demand for everything else. ASAN builds shouldn’t download symbols since they don’t product symbol zips see bug 1283879

taskgraph.transforms.tests.set_expires_after(config, tests)

Try jobs expire after 2 weeks; everything else lasts 1 year. This helps keep storage costs low.

taskgraph.transforms.tests.set_profile(config, tests)

Set profiling mode for tests.

taskgraph.transforms.tests.set_retry_exit_status(config, tests)

Set the retry exit status to TBPL_RETRY, the value returned by mozharness scripts to indicate a transient failure that should be retried.

taskgraph.transforms.tests.set_schedules_for_webrender_android(config, tests)

android-hw has limited resources, we need webrender on phones

taskgraph.transforms.tests.set_tag(config, tests)

Set test for a specific tag.

taskgraph.transforms.tests.set_target(config, tests)
taskgraph.transforms.tests.set_test_type(config, tests)
taskgraph.transforms.tests.set_tier(config, tests)

Set the tier based on policy for all test descriptions that do not specify a tier otherwise.

taskgraph.transforms.tests.set_treeherder_machine_platform(config, tests)

Set the appropriate task.extra.treeherder.machine.platform

taskgraph.transforms.tests.set_worker_type(config, tests)

Set the worker type based on the test platform.

taskgraph.transforms.tests.setup_browsertime(config, tests)

Configure browsertime dependencies for Raptor pageload tests that have –browsertime extra option.

taskgraph.transforms.tests.setup_browsertime_flag(config, tests)

Optionally add –browsertime flag to Raptor pageload tests.

taskgraph.transforms.tests.setup_raptor(config, tests)

Add options that are specific to raptor jobs (identified by suite=raptor)

taskgraph.transforms.tests.setup_talos(config, tests)

Add options that are specific to talos jobs (identified by suite=talos)

taskgraph.transforms.tests.single_stylo_traversal_tests(config, tests)

Enable single traversal for all tests on the sequential Stylo platform.

taskgraph.transforms.tests.split_chunks(config, tests)

Based on the ‘chunks’ key, split tests up into chunks by duplicating them and assigning ‘this-chunk’ appropriately and updating the treeherder symbol.

taskgraph.transforms.tests.split_e10s(config, tests)
taskgraph.transforms.tests.split_variants(config, tests)

taskgraph.transforms.try_job module

taskgraph.transforms.try_job.set_job_try_name(config, jobs)

For a task which is governed by -j in try syntax, set the job_try_name attribute based on the job name.

taskgraph.transforms.update_verify module

Transform the beetmover task into an actual task description.

taskgraph.transforms.update_verify.add_command(config, tasks)

taskgraph.transforms.update_verify_config module

Transform the beetmover task into an actual task description.

taskgraph.transforms.update_verify_config.add_command(config, tasks)

taskgraph.transforms.upload_generated_sources module

Transform the upload-generated-files task description template, taskcluster/ci/upload-generated-sources/kind.yml, into an actual task description.

taskgraph.transforms.upload_generated_sources.add_task_info(config, jobs)

taskgraph.transforms.upload_symbols module

Transform the upload-symbols task description template, taskcluster/ci/upload-symbols/job-template.yml into an actual task description.

taskgraph.transforms.upload_symbols.check_nightlies(config, tasks)

Ensure that we upload symbols for all nightly builds, so that crash-stats can resolve any reports sent to it. Try may enable full symbols but not upload them.

Putting this check here (instead of the transforms for the build kind) lets us leverage the any not-for-build-platforms set in the update-symbols kind.

taskgraph.transforms.upload_symbols.fill_template(config, tasks)

taskgraph.transforms.visual_metrics module

These transformations take a task description for a visual metrics task and add the necessary environment variables to run on the given inputs.

taskgraph.transforms.visual_metrics.set_visual_metrics_jobs(config, jobs)

Set the visual metrics configuration for the given jobs.

taskgraph.transforms.webrender module

taskgraph.transforms.webrender.webrender(config, tasks)

Do transforms specific to webrender tasks.

Module contents