taskgraph.transforms package

Submodules

taskgraph.transforms.balrog module

taskgraph.transforms.base module

class taskgraph.transforms.base.TransformConfig(kind, path, config, params, kind_dependencies_tasks=None, graph_config=None)

Bases: object

A container for configuration affecting transforms. The config argument to transforms is an instance of this class, possibly with additional kind-specific attributes beyond those set here.

class taskgraph.transforms.base.TransformSequence(transforms=None)

Bases: object

Container for a sequence of transforms. Each transform is represented as a callable taking (config, items) and returning a generator which will yield transformed items. The resulting sequence has the same interface.

This is convenient to use in a file full of transforms, as it provides a decorator, @transforms.add, that will add the decorated function to the sequence.

add(func)

taskgraph.transforms.beetmover module

taskgraph.transforms.beetmover_cdns module

taskgraph.transforms.beetmover_checksums module

taskgraph.transforms.beetmover_l10n module

Transform the signing task into an actual task description.

taskgraph.transforms.beetmover_l10n.make_beetmover_description(config, jobs)

taskgraph.transforms.beetmover_repackage module

taskgraph.transforms.beetmover_repackage_l10n module

Transform the signing task into an actual task description.

taskgraph.transforms.beetmover_repackage_l10n.make_beetmover_description(config, jobs)

taskgraph.transforms.build module

Apply some defaults and minor modifications to the jobs defined in the build kind.

taskgraph.transforms.build.set_defaults(config, jobs)

Set defaults, including those that differ per worker implementation

taskgraph.transforms.build.set_env(config, jobs)

Set extra environment variables from try command line.

taskgraph.transforms.build_attrs module

taskgraph.transforms.build_attrs.set_build_attributes(config, jobs)

Set the build_platform and build_type attributes based on the job name. Although not all jobs using this transform are actual “builds”, the try option syntax treats them as such, and this arranges the attributes appropriately for that purpose.

taskgraph.transforms.build_attrs.set_schedules_optimization(config, jobs)

Set the skip-unless-affected optimization based on the build platform.

taskgraph.transforms.build_lints module

Apply some defaults and minor modifications to the jobs defined in the build kind.

taskgraph.transforms.build_lints.check_mozharness_perfherder_options(config, jobs)

Verify that multiple jobs don’t use the same perfherder bucket.

Build jobs record perfherder metrics by default. Perfherder metrics go to a bucket derived by the platform by default. The name can further be customized by the presence of “extra options” either defined in mozharness sub-configs or in an environment variable.

This linter tries to verify that no 2 jobs will send Perfherder metrics to the same bucket by looking for jobs not defining extra options when their platform or mozharness config are otherwise similar.

taskgraph.transforms.build_signing module

Transform the signing task into an actual task description.

taskgraph.transforms.build_signing.add_signed_routes(config, jobs)

Add routes corresponding to the routes of the build task this corresponds to, with .signed inserted, for all gecko.v2 routes

taskgraph.transforms.build_signing.define_upstream_artifacts(config, jobs)

taskgraph.transforms.checksums_signing module

taskgraph.transforms.coalesce module

This transform enables coalescing for tasks with scm level > 1, by defining values for the coalesce settings of the job.

taskgraph.transforms.coalesce.enable_coalescing(config, jobs)

This transform enables coalescing for tasks with scm level > 1, setting the name to be equal to the the first 20 chars of the sha256 of the task name (label). The hashing is simply used to keep the name short, unique, and alphanumeric, since it is used in AMQP routing keys.

Note, the coalescing keys themselves are not intended to be readable strings or embed information. The tasks they represent contain all relevant metadata.

taskgraph.transforms.docker_image module

Transform the upload-symbols task description template,
taskcluster/ci/upload-symbols/job-template.yml

into an actual task description.

taskgraph.transforms.docker_image.fill_template(config, tasks)

taskgraph.transforms.gecko_v2_whitelist module

This file contains a whitelist of gecko.v2 index route job names. The intent of this whitelist is to raise an alarm when new jobs are added. If those jobs already run in Buildbot, then it’s important that the generated index routes match (and that only one of Buildbot and TaskCluster be tier-1 at any time). If the jobs are new and never ran in Buildbot, then their job name can be added here without any further fuss.

Once all jobs have been ported from Buildbot, this file can be removed.

taskgraph.transforms.l10n module

taskgraph.transforms.marionette_harness module

Set dynamic task description properties of the marionette-harness task.

taskgraph.transforms.marionette_harness.setup_task(config, tasks)

taskgraph.transforms.name_sanity module

Generate labels for tasks without names, consistently. Uses attributes from ‘dependent-task’.

taskgraph.transforms.name_sanity.make_label(config, jobs)

Generate a sane label for a new task constructed from a dependency Using attributes from the dependent job and the current task kind

taskgraph.transforms.nightly_l10n_signing module

Transform the signing task into an actual task description.

taskgraph.transforms.nightly_l10n_signing.define_upstream_artifacts(config, jobs)
taskgraph.transforms.nightly_l10n_signing.make_signing_description(config, jobs)

taskgraph.transforms.partials module

Transform the partials task into an actual task description.

taskgraph.transforms.partials.make_task_description(config, jobs)

taskgraph.transforms.partials_signing module

Transform the partials task into an actual task description.

taskgraph.transforms.partials_signing.generate_upstream_artifacts(release_history, platform, locale=None)
taskgraph.transforms.partials_signing.make_task_description(config, jobs)

taskgraph.transforms.push_apk module

taskgraph.transforms.push_apk_breakpoint module

taskgraph.transforms.release_deps module

Add dependencies to release tasks.

taskgraph.transforms.release_deps.add_dependencies(config, jobs)

taskgraph.transforms.repackage module

taskgraph.transforms.repackage_l10n module

Transform the repackage task into an actual task description.

taskgraph.transforms.repackage_l10n.split_locales(config, jobs)

taskgraph.transforms.repackage_routes module

taskgraph.transforms.repackage_signing module

taskgraph.transforms.signing module

taskgraph.transforms.source_test module

taskgraph.transforms.spidermonkey module

taskgraph.transforms.spidermonkey.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc.

taskgraph.transforms.task module

taskgraph.transforms.tests module

These transforms construct a task description to run the given test, based on a test description. The implementation here is shared among all test kinds, but contains specific support for how we run tests in Gecko (via mozharness, invoked in particular ways).

This is a good place to translate a test-description option such as single-core: true to the implementation of that option in a task description (worker options, mozharness commandline, environment variables, etc.)

The test description should be fully formed by the time it reaches these transforms, and these transforms should not embody any specific knowledge about what should run where. this is the wrong place for special-casing platforms, for example - use all_tests.py instead.

taskgraph.transforms.tests.allow_software_gl_layers(config, tests)

Handle the “allow-software-gl-layers” property for platforms where it applies.

taskgraph.transforms.tests.disable_stylo(config, tests)

Disable Stylo for all jobs on -stylo-disabled platforms.

taskgraph.transforms.tests.enable_code_coverage(config, tests)

Enable code coverage for the linux64-ccov/opt & linux64-jsdcov/opt & win64-ccov/debug build-platforms

taskgraph.transforms.tests.enable_webrender(config, tests)

Handle the “webrender” property by passing a flag to mozharness if it is enabled.

taskgraph.transforms.tests.get_firefox_version()
taskgraph.transforms.tests.handle_keyed_by(config, tests)

Resolve fields that can be keyed by platform, etc.

taskgraph.transforms.tests.handle_keyed_by_mozharness(config, tests)

Resolve a mozharness field if it is keyed by something

taskgraph.transforms.tests.handle_run_on_projects(config, tests)

Handle translating built-projects appropriately

taskgraph.transforms.tests.handle_suite_category(config, tests)
taskgraph.transforms.tests.make_job_description(config, tests)

Convert test descriptions to job descriptions (input to taskgraph.transforms.job)

taskgraph.transforms.tests.normpath(path)
taskgraph.transforms.tests.set_defaults(config, tests)
taskgraph.transforms.tests.set_download_symbols(config, tests)

In general, we download symbols immediately for debug builds, but only on demand for everything else. ASAN builds shouldn’t download symbols since they don’t product symbol zips see bug 1283879

taskgraph.transforms.tests.set_expires_after(config, tests)

Try jobs expire after 2 weeks; everything else lasts 1 year. This helps keep storage costs low.

taskgraph.transforms.tests.set_profile(config, tests)

Set profiling mode for tests.

taskgraph.transforms.tests.set_retry_exit_status(config, tests)

Set the retry exit status to TBPL_RETRY, the value returned by mozharness scripts to indicate a transient failure that should be retried.

taskgraph.transforms.tests.set_tag(config, tests)

Set test for a specific tag.

taskgraph.transforms.tests.set_target(config, tests)
taskgraph.transforms.tests.set_test_type(config, tests)
taskgraph.transforms.tests.set_tier(config, tests)

Set the tier based on policy for all test descriptions that do not specify a tier otherwise.

taskgraph.transforms.tests.set_treeherder_machine_platform(config, tests)

Set the appropriate task.extra.treeherder.machine.platform

taskgraph.transforms.tests.set_worker_type(config, tests)

Set the worker type based on the test platform.

taskgraph.transforms.tests.setup_talos(config, tests)

Add options that are specific to talos jobs (identified by suite=talos)

taskgraph.transforms.tests.single_stylo_traversal_tests(config, tests)

Enable single traversal for all tests on the sequential Stylo platform.

taskgraph.transforms.tests.skip_win10_hardware(config, tests)

Windows 10 hardware isn’t ready yet, don’t even bother scheduling unless we’re on try

taskgraph.transforms.tests.split_chunks(config, tests)

Based on the ‘chunks’ key, split tests up into chunks by duplicating them and assigning ‘this-chunk’ appropriately and updating the treeherder symbol.

taskgraph.transforms.tests.split_e10s(config, tests)
taskgraph.transforms.tests.validate(config, tests)

taskgraph.transforms.try_job module

taskgraph.transforms.try_job.set_job_try_name(config, jobs)

For a task which is governed by -j in try syntax, set the job_try_name attribute based on the job name.

taskgraph.transforms.upload_generated_sources module

Transform the upload-generated-files task description template,
taskcluster/ci/upload-generated-sources/kind.yml

into an actual task description.

taskgraph.transforms.upload_generated_sources.add_task_info(config, jobs)

taskgraph.transforms.upload_symbols module

Transform the upload-symbols task description template,
taskcluster/ci/upload-symbols/job-template.yml

into an actual task description.

taskgraph.transforms.upload_symbols.fill_template(config, tasks)

taskgraph.transforms.use_toolchains module

taskgraph.transforms.use_toolchains.use_toolchains(config, jobs)

Add dependencies corresponding to toolchains to use, and pass a list of corresponding artifacts to jobs using toolchains.

Module contents