mozperfcommon — Code shared between the performance test harnesses
Mozperfcommon contains code shared between the performance test harnesses, such as Raptor and Talos. It currently provides extreme regression detection which compares the results of a test run with the recent history of the same tests in Perfherder and raises these regressions as failures.
Extreme regression detection
XtremeRegression compares a perfherder data
blob, following performance-artifact-schema.json, with the historical data of
its suites and subtests on a Treeherder repository (autoland by default).
Initialize the class with a platform (optionally the branch too), then call detect
with the perfherder data that was produced to compare it to historical data. By default,
it compares the suites and subtests that alert in Perfherder with all the pushes of the
last 7 days, and flags regressions when the given data exceeds 10 standard deviations of the
historical data. This returns a list of comparisons, and entries with exceeds_variance
set to True are extreme regressions. Errors can then be raised for them and
XtremeRegressionError is provided for this.
In CI, the revision being tested is taken from the GECKO_HEAD_REV environment
variable. Outside of CI, the history up to now is used.
Basic usage
from mozperfcommon.detect import XtremeRegression
detector = XtremeRegression(
"windows11-64-24h2-shippable", revision_repository="autoland"
)
results = detector.detect(perfherder_data)
for result in results:
if result["exceeds_variance"]:
print(result["suite"], result["test"], result["percentage"])
detect returns one dict per compared value, with the statistics of the
comparison:
{
"suite": "speedometer3",
"test": "Charts-chartjs/total", # None for the suite value
"value": 53.26,
"new_count": 50, # Number of new values (replicates)
"count": 2400, # Number of historical values
"signature_id": 5257402,
"expected_change": None, # Bug of the matching expected change
"new_mean": 53.26,
"historical_mean": 35.73,
"historical_stdev": 1.31,
"difference": 17.53,
"percentage": 49.06,
"exceeds_variance": True,
"new_outliers": 0, # Number of new values ignored as outliers
}
exceeds_variance is None when there isn’t enough history to check the
variance, or when an expected change matches.
Suite and subtest values are only compared when the run produced replicates for them, i.e. at least 2 values. A single value says nothing about the variance of the run, so those values are left out of the results.
At least 15 historical values are needed to check the variance, since a comparison with fewer values isn’t useful. Tests that don’t reach 15 values within the last 7 days are never flagged. This is expected for tests that run infrequently or that produce few values per run.
Failures to get the history from Treeherder, e.g. when it is down or when the revision isn’t found on its repository, are logged and the values are then compared without history.
The signatures are cached per framework, so an XtremeRegression instance can
be reused for all the results of a test run.
Failing on regressions
Pass raise_for_regressions=True to detect to raise an
XtremeRegressionError when any value is flagged.
Harnesses that want to finish their run first can collect the results and call
raise_for_regressions() later for
the same behaviour.
The first line of the error gives the number of regressions and the 3 largest ones by percentage, and the following lines give the details of all of them:
2 extreme regression(s) detected, top 2: speedometer3 Charts-chartjs/total (+49.06%), speedometer3 (-20.07%)
speedometer3: difference -5.44 (-20.07%)
speedometer3 Charts-chartjs/total: difference 17.53 (+49.06%)
Comparisons
The comparison and its threshold are options of detect, so they can differ
between perfherder-data blobs:
results = detector.detect(perfherder_data, comparison="median", threshold=4)
Two comparisons are available:
mean(default): compares the means, with a threshold in historical standard deviations. New values that are more than 5 median absolute deviations worse than the median of the new values are ignored as outliers, so that a few bad replicates in an otherwise normal run aren’t flagged. Their number is returned innew_outliers. The historical values are all used.median: compares the medians, with a threshold in historical median absolute deviations, which is less sensitive to outliers.
Other comparisons can be added with
add(). A comparison receives the
new values, the historical values (possibly empty), and the threshold and
lower_is_better keyword arguments. It returns the fields to add to the
result, which must include exceeds_variance. The exceeds_variance should be
True when something was detected, False when something was not, and None if
a comparison was not possible for any reason.
from mozperfcommon.comparisons import Comparisons
@Comparisons.add("maximum")
def compare_to_maximum(new_values, historical_values, threshold, lower_is_better):
...
return {"historical_maximum": maximum, "exceeds_variance": exceeds}
Expected changes
Regressions caused on purpose are listed in
testing/performance/common/expected_changes.yml. A matching value is still
compared, but never flagged (will be changed in the future). Add an entry
in the patch causing the regression to prevent the regression failures:
- bug: 1234567
# fnmatch pattern of the suite.
suite: speedometer3
# Optional fnmatch pattern of the subtests. Without it, the suite value and
# all the subtests of the suite match.
test: "TodoMVC-*"
# Optional fnmatch pattern of the Treeherder machine platform. Without it,
# all the platforms match.
platform: "windows11-*"
# Optional extra options of the suite, which must all match, in any order.
# Without it, the suite matches whatever its extra options are.
extra_options: ["fission", "webrender"]
# Optional fnmatch pattern of the application. Without it, all the
# applications match.
application: firefox
# The entry is ignored for revisions pushed after this date.
expires: 2026-11-01
Remove entries once they have expired. The file is validated when it is
loaded, and an invalid entry raises an
InvalidExpectedChangeError.
Outside of a source checkout, e.g. from the test archive in CI, the path of the
file must be given in the PERF_EXPECTED_CHANGES_PATH environment variable.
Mozharness sets it for Raptor and Talos.
Usage in the harnesses
Raptor runs the detection on the results of its browsertime tests, when running
in CI (RAPTOR_TASK_LABEL is set) and when the results validate. The platform
comes from the task label, and the repository of the tested revision from
GECKO_HEAD_REPOSITORY. Other harnesses will also need the task label so that it
can be used to find historical data. It’s up to the harness to determine how to
call the detection and raise the failures, some may want to complete some work
prior to failing (such as Raptor).
API reference
- class mozperfcommon.detect.XtremeRegression(platform, repository=DEFAULT_REPOSITORY, days=DEFAULT_DAYS, push_limit=DEFAULT_PUSH_LIMIT, revision=None, revision_repository=None, replicates=True)
Compares perfherder-data blobs with the historical data of a repository, see
detect().- Parameters:
platform – Treeherder machine platform the data was produced on, e.g.
windows11-64-24h2-shippable.repository – Treeherder repository to compare with, defaults to autoland.
days – Number of days of history to use.
push_limit – Maximum number of pushes with data, the latest ones in the days of history, to compare with. None, the default, to use all of them from the previous days number of days.
revision – Revision the data was produced on. Only data pushed before this revision is used. Defaults to the revision being tested in CI, see
get_ci_revision(). Outside of CI, the most recent data is used.revision_repository – Repository revision was pushed to, e.g.
try. Defaults to repository.replicates – If True, the history contains every replicate of each past run. If False, it contains one value per past run. Runs without replicates always contribute their single value. The new values are always the replicates.
Regressions listed in the in-tree expected changes, see
load_expected_changes(), are not checked against the variance.- detect(perfherder_data, comparison=DEFAULT_COMPARISON, threshold=DEFAULT_THRESHOLD, raise_for_regressions=False)
Compare a PERFHERDER_DATA blob with the historical data.
- Parameters:
perfherder_data – Dict following performance-artifact-schema.json.
comparison – Name of the comparison to do with the historical values, see
Comparisons. Currently, only mean and median are available.threshold – Threshold of the variance check of the comparison, e.g. the number of historical standard deviations the difference must exceed for the
meancomparison.raise_for_regressions – Raise an
XtremeRegressionErrorfor the extreme regressions detected, seeraise_for_regressions().
- Returns:
A list with one dict per suite and subtest value with replicates, e.g. with the default
meancomparison:[ { "suite": "speedometer3", "test": "Charts-chartjs/total", # None for the suite "value": 53.26, "new_count": 50, # Number of new values (replicates) "count": 2400, # Number of historical values "signature_id": 5257402, # (suite, test, extra options, application) "signature_key": ( "speedometer3", "Charts-chartjs/total", ("fission", "webrender"), "firefox", ), "expected_change": None, # Bug of the expected change # Fields returned by the comparison. "new_mean": 53.26, "historical_mean": 35.73, "historical_stdev": 1.31, "difference": 17.53, "percentage": 49.06, "exceeds_variance": True, # Number of new values ignored as outliers "new_outliers": 0, }, ..., ]
Suite and subtest values without replicates are not compared, since a single value says nothing about the variance. Values that don’t alert in Perfherder are not compared either.
The comparison fields are None when there’s no history, including when the push of the revision or the signatures can’t be retrieved from Treeherder.
- raise_for_regressions(results)
Helper for raising on extreme regressions.
- exception mozperfcommon.detect.XtremeRegressionError(detections)
Raised by test harnesses when
XtremeRegression.detect()detected extreme changes.First line provides the number of regressions and the top 3 by percentage, and the rest of the lines provide more details on all the regressions.
- Parameters:
detections – The results of
XtremeRegression.detect()that are extreme, e.g. those withexceeds_varianceset.
- mozperfcommon.detect.should_alert(suite, subtest=None)
Return whether Perfherder alerts on a suite value, or on one of its subtests when subtest is given.
Suite values alert unless
shouldAlertis false. Subtests alert whenshouldAlertis true, or when it’s unset and their suite has no value.
- class mozperfcommon.comparisons.Comparisons
Registry of the comparisons.
- classmethod add(name)
Decorator registering a comparison function under name.
- classmethod get(name)
Return the comparison registered under name.
- exception mozperfcommon.comparisons.UnknownComparisonError
- mozperfcommon.comparisons.compare_to_mean(new_values, historical_values, threshold=DEFAULT_THRESHOLD, lower_is_better=True)
Compare the means. Uses standard deviation for the threshold.
The outliers of the new values, see
remove_outliers(), are ignored.
- mozperfcommon.comparisons.compare_to_median(new_values, historical_values, threshold=DEFAULT_THRESHOLD, lower_is_better=True)
Compare the medians. Uses MAD for the threshold.
- mozperfcommon.comparisons.median_absolute_deviation(values)
Return the median absolute deviation of values.
- exception mozperfcommon.expected_changes.InvalidExpectedChangeError
- mozperfcommon.expected_changes.find_expected_change(expected_changes, key, platform, date)
Returns an expected change if it exists given the parameters.
Checks if it’s expired on date (a date or datetime), and if not, matches on the suite, test (if it exists), extra options and application of the signature key (see
signature_key()), and then the platform.
- mozperfcommon.expected_changes.get_expected_changes_path()
Return the path to the expected changes YAML.
Uses either the in-tree location or a custom location set by the env variable PERF_EXPECTED_CHANGES_PATH.
- mozperfcommon.expected_changes.load_expected_changes(path=None)
Validates, and returns the contents of the expected changes YAML.
- class mozperfcommon.perfherder.PerfherderClient(server_url=TREEHERDER_URL, repository=DEFAULT_REPOSITORY, timeout=30)
Helper client for getting performance data from Perfherder.
- Parameters:
server_url – Root URL of the Treeherder instance.
repository – Treeherder repository (project) to query.
timeout – Timeout in seconds for each request.
- get_push_time(revision, repository=None)
Return the time, as a UTC datetime, at which revision was pushed to repository (defaults to the client’s repository).
- get_signatures(framework_id, platform, start_date, subtests=True)
Return the signatures, keyed by id, that received data on platform since start_date. Signatures that have a parent are only included when subtests is True.
- get_summary(start_date, end_date, signature_id=None, parent_signature_id=None, replicates=True, all_data=False)
Return the summaries of a signature, or of all the subtests of a parent signature, with the values pushed after start_date and before end_date.
Each summary has the values in
values. With replicates, those are the replicates of each data point, or the data point itself when it has none. With all_data, which only applies to signature_id, the values are instead indata, along with the push they come from.
- exception mozperfcommon.perfherder.PushNotFoundError
Raised when a push is not found for a specific revision.
- exception mozperfcommon.utils.UnknownFrameworkError
Raised when an unknown framework name is found.
- mozperfcommon.utils.format_date(date)
Format a datetime as the UTC date string expected by Treeherder. Naive datetimes are treated as UTC.
- mozperfcommon.utils.get_ci_revision()
Return the revision being tested in CI, or None outside of CI.
- mozperfcommon.utils.signature_key(suite, test, extra_options, application)
Return the key identifying a Perfherder signature on a platform. The test is None for suite signatures.