Test Kind ========= The ``test`` kind defines both desktop and mobile tests for builds. Each YAML file referenced in ``kind.yml`` defines the full set of tests for the associated suite. The process of generating tests goes like this, based on a set of YAML files named in ``kind.yml``: * For each build task, determine the related test platforms based on the build platform. For example, a Windows 2010 build might be tested on Windows 7 and Windows 10. Each test platform specifies "test sets" indicating which tests to run. This is configured in the file named ``test-platforms.yml``. * Each test set is expanded to a list of tests to run. This is configured in the file named by ``test-sets.yml``. A platform may specify several test sets, in which case the union of those sets is used. * Each named test is looked up in the file named by ``tests.yml`` to find a test description. This test description indicates what the test does, how it is reported to treeherder, and how to perform the test, all in a platform-independent fashion. * Each test description is converted into one or more tasks. This is performed by a sequence of transforms defined in the ``transforms`` key in ``kind.yml``. See :ref:`transforms` for more information. * The resulting tasks become a part of the task graph. .. important:: This process generates *all* test jobs, regardless of tree or try syntax. It is up to a later stages of the task-graph generation (the target set and optimization) to select the tests that will actually be performed. Variants -------- Sometimes we want to run the same tests under a different Firefox context, usually this means with a pref set. The concept of ``variants`` was invented to handle this use case. A variant is a stanza of configuration that can be merged into each test definition. Variants are defined in the `variants.yml`_ file. See this file for an up to date list of active variants and the pref(s) they set. Each variant must conform to the :py:data:`~gecko_taskgraph.transforms.test.variant_description_schema`: * **description** (required) - A description explaining what the variant is for. * **component** (required) - The name of the component that owns the variant. It should be formatted as ``PRODUCT``::``COMPONENT``. * **expiration** (required) - The date when the variant will be expired (maximum 6 months). * **suffix** (required) - A suffix to apply to the task label and treeherder symbol. * **when** - A `json-e`_ expression that must evaluate to ``true`` for the variant to be applied. The ``task`` definition is passed in as context. * **replace** - A dictionary that will overwrite keys in the task definition. * **merge** - A dictionary that will be merged into the task definition using the :py:func:`~gecko_taskgraph.util.templates.merge` function. .. note:: Exceptions can be requested to have a variant without expiration (using "never") if this is a shipped mode we support. Teams should contact the CI team to discuss this before submitting a patch if they think their variant qualifies. All exceptions will require director approval. Defining Variants ~~~~~~~~~~~~~~~~~ Variants can be defined in the test YAML files using the ``variants`` key. E.g: .. code-block:: yaml example-suite: variants: - foo - bar This will split the task into three. The original task, the task with the config from the variant named 'foo' merged in and the task with the config from the variant named 'bar' merged in. Composite Variants ~~~~~~~~~~~~~~~~~~ Sometimes we want to run tasks with multiple variants enabled at once. This can be achieved with "composite variants". Composite variants are simply two or more variant names joined with the ``+`` sign. Using the previous example, if we wanted to run both the ``foo`` and ``bar`` variants together, we could do: .. code-block:: yaml example-suite: variants: - foo+bar This will first merge or replace the config of ``foo`` into the task, followed by the config of ``bar``. Care should be taken if both variants are replacing the same keys. The last variant's configuration will be the one that gets used. Expired Variants ~~~~~~~~~~~~~~~~ Ideally, when a variant is not needed anymore, it should be dropped (even if it has not expired). If you need to extend the expiration date, you can submit a patch to modify the expiration date in the `variants.yml`_ file. Variants will not be scheduled to run after the expiration date. If an expired variant is not dropped, the triage owner of the component will be notified. If the expired variant persists for an extended period, the autonag bot will escalate to notify the manager and director of the triage owner. Once at that point, we will submit a patch to remove the variant from Taskcluster and manifest conditions pending the triage owner / manager to review. Please subscribe to alerts from `firefox-ci ` group in order to be aware of changes to the CI, scheduling, or the policy. .. _variants.yml: https://searchfox.org/mozilla-central/source/taskcluster/kinds/test/variants.yml .. _json-e: https://json-e.js.org/ Setting ------- A test ``setting`` is the set of conditions under which a test is running. Aside from the chunk number, a ``setting`` uniquely distinguishes a task from another that is running the same set of tests. There are three types of inputs that make up a ``setting``: 1. Platform - Bits of information that describe the underlying platform the test is running on. This includes things like the operating system and version, CPU architecture, etc. 2. Build - Bits of information that describe the build being tested. This includes things like the build type and which build attributes (like ``asan``, ``ccov``, etc) are enabled. 3. Runtime - Bits of information that describe the configured state of Firefox. This includes things like prefs and environment variables. Note that tasks should only set runtime configuration via the variants system (see `Variants`_). Test ``settings`` are available in the ``task.extra.test-setting`` object in all test tasks. They are defined by the :py:func:`~gecko_taskgraph.transforms.test.set_test_setting` transform function. The full schema is defined in the :py:data:`~gecko_taskgraph.transforms.test.test_setting_description_schema`. Setting Hash ~~~~~~~~~~~~ In addition to the three top-level objects, there is also a ``_hash`` key which contains a hash of the rest of the setting object. This is a convenient way for consumers to group or compare tasks that run under the same setting.