mozperfcommon — Code shared between the performance test harnesses

Mozperfcommon contains code shared between the performance test harnesses, such as Raptor and Talos. It currently provides extreme regression detection which compares the results of a test run with the recent history of the same tests in Perfherder and raises these regressions as failures.

Extreme regression detection

XtremeRegression compares a perfherder data blob, following performance-artifact-schema.json, with the historical data of its suites and subtests on a Treeherder repository (autoland by default).

Initialize the class with a platform (optionally the branch too), then call detect with the perfherder data that was produced to compare it to historical data. By default, it compares the suites and subtests that alert in Perfherder with all the pushes of the last 7 days, and flags regressions when the given data exceeds 10 standard deviations of the historical data. This returns a list of comparisons, and entries with exceeds_variance set to True are extreme regressions. Errors can then be raised for them and XtremeRegressionError is provided for this.

In CI, the revision being tested is taken from the GECKO_HEAD_REV environment variable. Outside of CI, the history up to now is used.

Basic usage

from mozperfcommon.detect import XtremeRegression

detector = XtremeRegression(
    "windows11-64-24h2-shippable", revision_repository="autoland"
)
results = detector.detect(perfherder_data)

for result in results:
    if result["exceeds_variance"]:
        print(result["suite"], result["test"], result["percentage"])

detect returns one dict per compared value, with the statistics of the comparison:

{
    "suite": "speedometer3",
    "test": "Charts-chartjs/total",  # None for the suite value
    "value": 53.26,
    "new_count": 50,  # Number of new values (replicates)
    "count": 2400,  # Number of historical values
    "signature_id": 5257402,
    "expected_change": None,  # Bug of the matching expected change
    "new_mean": 53.26,
    "historical_mean": 35.73,
    "historical_stdev": 1.31,
    "difference": 17.53,
    "percentage": 49.06,
    "exceeds_variance": True,
    "new_outliers": 0,  # Number of new values ignored as outliers
}

exceeds_variance is None when there isn’t enough history to check the variance, or when an expected change matches.

Suite and subtest values are only compared when the run produced replicates for them, i.e. at least 2 values. A single value says nothing about the variance of the run, so those values are left out of the results.

At least 15 historical values are needed to check the variance, since a comparison with fewer values isn’t useful. Tests that don’t reach 15 values within the last 7 days are never flagged. This is expected for tests that run infrequently or that produce few values per run.

Failures to get the history from Treeherder, e.g. when it is down or when the revision isn’t found on its repository, are logged and the values are then compared without history.

The signatures are cached per framework, so an XtremeRegression instance can be reused for all the results of a test run.

Failing on regressions

Pass raise_for_regressions=True to detect to raise an XtremeRegressionError when any value is flagged. Harnesses that want to finish their run first can collect the results and call raise_for_regressions() later for the same behaviour.

The first line of the error gives the number of regressions and the 3 largest ones by percentage, and the following lines give the details of all of them:

2 extreme regression(s) detected, top 2: speedometer3 Charts-chartjs/total (+49.06%), speedometer3 (-20.07%)
  speedometer3: difference -5.44 (-20.07%)
  speedometer3 Charts-chartjs/total: difference 17.53 (+49.06%)

Comparisons

The comparison and its threshold are options of detect, so they can differ between perfherder-data blobs:

results = detector.detect(perfherder_data, comparison="median", threshold=4)

Two comparisons are available:

  • mean (default): compares the means, with a threshold in historical standard deviations. New values that are more than 5 median absolute deviations worse than the median of the new values are ignored as outliers, so that a few bad replicates in an otherwise normal run aren’t flagged. Their number is returned in new_outliers. The historical values are all used.

  • median: compares the medians, with a threshold in historical median absolute deviations, which is less sensitive to outliers.

Other comparisons can be added with add(). A comparison receives the new values, the historical values (possibly empty), and the threshold and lower_is_better keyword arguments. It returns the fields to add to the result, which must include exceeds_variance. The exceeds_variance should be True when something was detected, False when something was not, and None if a comparison was not possible for any reason.

from mozperfcommon.comparisons import Comparisons


@Comparisons.add("maximum")
def compare_to_maximum(new_values, historical_values, threshold, lower_is_better):
    ...
    return {"historical_maximum": maximum, "exceeds_variance": exceeds}

Expected changes

Regressions caused on purpose are listed in testing/performance/common/expected_changes.yml. A matching value is still compared, but never flagged (will be changed in the future). Add an entry in the patch causing the regression to prevent the regression failures:

- bug: 1234567
  # fnmatch pattern of the suite.
  suite: speedometer3
  # Optional fnmatch pattern of the subtests. Without it, the suite value and
  # all the subtests of the suite match.
  test: "TodoMVC-*"
  # Optional fnmatch pattern of the Treeherder machine platform. Without it,
  # all the platforms match.
  platform: "windows11-*"
  # Optional extra options of the suite, which must all match, in any order.
  # Without it, the suite matches whatever its extra options are.
  extra_options: ["fission", "webrender"]
  # Optional fnmatch pattern of the application. Without it, all the
  # applications match.
  application: firefox
  # The entry is ignored for revisions pushed after this date.
  expires: 2026-11-01

Remove entries once they have expired. The file is validated when it is loaded, and an invalid entry raises an InvalidExpectedChangeError.

Outside of a source checkout, e.g. from the test archive in CI, the path of the file must be given in the PERF_EXPECTED_CHANGES_PATH environment variable. Mozharness sets it for Raptor and Talos.

Usage in the harnesses

Raptor runs the detection on the results of its browsertime tests, when running in CI (RAPTOR_TASK_LABEL is set) and when the results validate. The platform comes from the task label, and the repository of the tested revision from GECKO_HEAD_REPOSITORY. Other harnesses will also need the task label so that it can be used to find historical data. It’s up to the harness to determine how to call the detection and raise the failures, some may want to complete some work prior to failing (such as Raptor).

API reference

class mozperfcommon.detect.XtremeRegression(platform, repository=DEFAULT_REPOSITORY, days=DEFAULT_DAYS, push_limit=DEFAULT_PUSH_LIMIT, revision=None, revision_repository=None, replicates=True)

Compares perfherder-data blobs with the historical data of a repository, see detect().

Parameters:
  • platform – Treeherder machine platform the data was produced on, e.g. windows11-64-24h2-shippable.

  • repository – Treeherder repository to compare with, defaults to autoland.

  • days – Number of days of history to use.

  • push_limit – Maximum number of pushes with data, the latest ones in the days of history, to compare with. None, the default, to use all of them from the previous days number of days.

  • revision – Revision the data was produced on. Only data pushed before this revision is used. Defaults to the revision being tested in CI, see get_ci_revision(). Outside of CI, the most recent data is used.

  • revision_repository – Repository revision was pushed to, e.g. try. Defaults to repository.

  • replicates – If True, the history contains every replicate of each past run. If False, it contains one value per past run. Runs without replicates always contribute their single value. The new values are always the replicates.

Regressions listed in the in-tree expected changes, see load_expected_changes(), are not checked against the variance.

detect(perfherder_data, comparison=DEFAULT_COMPARISON, threshold=DEFAULT_THRESHOLD, raise_for_regressions=False)

Compare a PERFHERDER_DATA blob with the historical data.

Parameters:
  • perfherder_data – Dict following performance-artifact-schema.json.

  • comparison – Name of the comparison to do with the historical values, see Comparisons. Currently, only mean and median are available.

  • threshold – Threshold of the variance check of the comparison, e.g. the number of historical standard deviations the difference must exceed for the mean comparison.

  • raise_for_regressions – Raise an XtremeRegressionError for the extreme regressions detected, see raise_for_regressions().

Returns:

A list with one dict per suite and subtest value with replicates, e.g. with the default mean comparison:

[
    {
        "suite": "speedometer3",
        "test": "Charts-chartjs/total",  # None for the suite
        "value": 53.26,
        "new_count": 50,  # Number of new values (replicates)
        "count": 2400,  # Number of historical values
        "signature_id": 5257402,
        # (suite, test, extra options, application)
        "signature_key": (
            "speedometer3",
            "Charts-chartjs/total",
            ("fission", "webrender"),
            "firefox",
        ),
        "expected_change": None,  # Bug of the expected change
        # Fields returned by the comparison.
        "new_mean": 53.26,
        "historical_mean": 35.73,
        "historical_stdev": 1.31,
        "difference": 17.53,
        "percentage": 49.06,
        "exceeds_variance": True,
        # Number of new values ignored as outliers
        "new_outliers": 0,
    },
    ...,
]

Suite and subtest values without replicates are not compared, since a single value says nothing about the variance. Values that don’t alert in Perfherder are not compared either.

The comparison fields are None when there’s no history, including when the push of the revision or the signatures can’t be retrieved from Treeherder.

raise_for_regressions(results)

Helper for raising on extreme regressions.

exception mozperfcommon.detect.XtremeRegressionError(detections)

Raised by test harnesses when XtremeRegression.detect() detected extreme changes.

First line provides the number of regressions and the top 3 by percentage, and the rest of the lines provide more details on all the regressions.

Parameters:

detections – The results of XtremeRegression.detect() that are extreme, e.g. those with exceeds_variance set.

mozperfcommon.detect.should_alert(suite, subtest=None)

Return whether Perfherder alerts on a suite value, or on one of its subtests when subtest is given.

Suite values alert unless shouldAlert is false. Subtests alert when shouldAlert is true, or when it’s unset and their suite has no value.

class mozperfcommon.comparisons.Comparisons

Registry of the comparisons.

classmethod add(name)

Decorator registering a comparison function under name.

classmethod get(name)

Return the comparison registered under name.

exception mozperfcommon.comparisons.UnknownComparisonError
mozperfcommon.comparisons.compare_to_mean(new_values, historical_values, threshold=DEFAULT_THRESHOLD, lower_is_better=True)

Compare the means. Uses standard deviation for the threshold.

The outliers of the new values, see remove_outliers(), are ignored.

mozperfcommon.comparisons.compare_to_median(new_values, historical_values, threshold=DEFAULT_THRESHOLD, lower_is_better=True)

Compare the medians. Uses MAD for the threshold.

mozperfcommon.comparisons.median_absolute_deviation(values)

Return the median absolute deviation of values.

exception mozperfcommon.expected_changes.InvalidExpectedChangeError
mozperfcommon.expected_changes.find_expected_change(expected_changes, key, platform, date)

Returns an expected change if it exists given the parameters.

Checks if it’s expired on date (a date or datetime), and if not, matches on the suite, test (if it exists), extra options and application of the signature key (see signature_key()), and then the platform.

mozperfcommon.expected_changes.get_expected_changes_path()

Return the path to the expected changes YAML.

Uses either the in-tree location or a custom location set by the env variable PERF_EXPECTED_CHANGES_PATH.

mozperfcommon.expected_changes.load_expected_changes(path=None)

Validates, and returns the contents of the expected changes YAML.

class mozperfcommon.perfherder.PerfherderClient(server_url=TREEHERDER_URL, repository=DEFAULT_REPOSITORY, timeout=30)

Helper client for getting performance data from Perfherder.

Parameters:
  • server_url – Root URL of the Treeherder instance.

  • repository – Treeherder repository (project) to query.

  • timeout – Timeout in seconds for each request.

get_push_time(revision, repository=None)

Return the time, as a UTC datetime, at which revision was pushed to repository (defaults to the client’s repository).

get_signatures(framework_id, platform, start_date, subtests=True)

Return the signatures, keyed by id, that received data on platform since start_date. Signatures that have a parent are only included when subtests is True.

get_summary(start_date, end_date, signature_id=None, parent_signature_id=None, replicates=True, all_data=False)

Return the summaries of a signature, or of all the subtests of a parent signature, with the values pushed after start_date and before end_date.

Each summary has the values in values. With replicates, those are the replicates of each data point, or the data point itself when it has none. With all_data, which only applies to signature_id, the values are instead in data, along with the push they come from.

exception mozperfcommon.perfherder.PushNotFoundError

Raised when a push is not found for a specific revision.

exception mozperfcommon.utils.UnknownFrameworkError

Raised when an unknown framework name is found.

mozperfcommon.utils.format_date(date)

Format a datetime as the UTC date string expected by Treeherder. Naive datetimes are treated as UTC.

mozperfcommon.utils.get_ci_revision()

Return the revision being tested in CI, or None outside of CI.

mozperfcommon.utils.signature_key(suite, test, extra_options, application)

Return the key identifying a Perfherder signature on a platform. The test is None for suite signatures.