Data Collection
This document describes the types of data that Normandy collects.
Uptake
Normandy monitors the execution of recipes and reports to
Telemetry the amount of successful and failed runs. This data
is reported using Uptake Telemetry under the
normandy
namespace.
Runner Status
Once per-fetch and execution of recipes, one of the following statuses is
reported under the key normandy/runner
:
- RUNNER_SUCCESS
- Telemetry value:
success
The operation completed successfully. Individual failures with actions and recipes may have been reported separately.
- RUNNER_SERVER_ERROR
- Telemetry value:
server_error
The data returned by the server when fetching the recipe is invalid in some way.
Action Status
For each action available from the Normandy service, one of the
following statuses is reported under the key
normandy/action/<action name>
:
- ACTION_NETWORK_ERROR
- Telemetry value:
network_error
There was a network-related error while fetching actions
- ACTION_PRE_EXECUTION_ERROR
- Telemetry value:
custom_1_error
There was an error while running the pre-execution hook for the action.
- ACTION_POST_EXECUTION_ERROR
- Telemetry value:
custom_2_error
There was an error while running the post-execution hook for the action.
- ACTION_SERVER_ERROR
- Telemetry value:
server_error
The data returned by the server when fetching the action is invalid in some way.
- ACTION_SUCCESS
- Telemetry value:
success
The operation completed successfully. Individual failures with recipes may be reported separately.
Recipe Status
For each recipe that is fetched and executed, one of the following statuses is
reported under the key normandy/recipe/<recipe id>
:
- RECIPE_ACTION_DISABLED
- Telemetry value:
custom_1_error
The action for this recipe failed in some way and was disabled, so the recipe could not be executed.
- RECIPE_DIDNT_MATCH_FILTER
- Telemetry value:
backoff
The recipe included a Jexl filter that the client did not match, so the recipe was not executed.
- RECIPE_EXECUTION_ERROR
- Telemetry value:
apply_error
An error occurred while executing the recipe.
- RECIPE_FILTER_BROKEN
- Telemetry value:
content_error
An error occurred while evaluating the Jexl filter for this recipe. Sometimes this represents a bug in the Jexl filter parser/evaluator, such as in bug 1477156, or it can represent an error fetching some data that a Jexl recipe needs such as bug 1447804.
- RECIPE_INVALID_ACTION
- Telemetry value:
download_error
The action specified by the recipe was invalid and it could not be executed.
- RECIPE_SUCCESS
- Telemetry value:
success
The recipe was executed successfully.
- RECIPE_SIGNATURE_INVALID
- Telemetry value:
signature_error
Normandy failed to verify the signature of the recipe.
Additionally, Normandy reports a keyed scalar to measure recipe
freshness. This scalar is called normandy.recipe_freshness
, and it
corresponds to the last_modified
date of each recipe (using its ID
as the key), reported as seconds since 1970 in UTC.
Enrollment
Normandy records enrollment and unenrollment of users into studies, and
records that data using Telemetry Events. All data is stored in the
normandy
category.
Preference Studies
- Enrollment
- method
The string
"enroll"
- object
The string
"preference_study"
- value
The name of the study (
recipe.arguments.slug
).- extra
- branch
The name of the branch the user was assigned to (example:
"control"
or"experiment"
).- experimentType
The type of preference experiment. Currently this can take values “exp” and “exp-highpop”, the latter being for experiments targeting large numbers of users.
- Enrollment Failed
- method
The string
"enrollFailed"
- object
The string
"preference_study"
- value
The slug of the study (
recipe.arguments.slug
)- extra
- reason
The reason for unenrollment. Possible values are:
"invalid-branch"
: The recipe specifies an invalid preference branch (not to be confused with the experiment branch). Valid values are “default” and “user”.
- preferenceBranch
If the reason was
"invalid-branch"
, the branch that was specified, truncated to 80 characters.
- Unenrollment
- method
The string
"unenroll"
.- object
The string
"preference_study"
.- value
The name of the study (
recipe.arguments.slug
).- extra
- didResetValue
The string
"true"
if the preference was set back to its original value,"false"
if it was left as its current value. This can happen when, for example, the user changes a preference that was involved in a user-branch study.- reason
The reason for unenrollment. Possible values are:
"recipe-not-seen"
: The recipe was no longer applicable to this client This can be because the recipe was disabled, or the user no longer matches the recipe’s filter."unknown"
: A reason was not specified. This should be considered a bug.
- Unenroll Failed
- method
The string
"unenrollFailed"
.- object
The string
"preference_study"
.- value
The name of the study (
recipe.arguments.slug
)- extra
- reason
A code describing the reason the unenroll failed. Possible values are:
"does-not-exist"
: The system attempted to unenroll a study that does not exist. This is a bug."already-unenrolled"
: The system attempted to unenroll a study that has already been unenrolled. This is likely a bug.
- caller
On Nightly builds only, a string identifying the source of the requested stop.
- originalReason
The code that would had been used for the unenrollment, had it not failed.
- Experimental Preference Changed
- method
The string
"expPrefChanged"
- object
The string
"preference_study"
.- value
The name of the study (
recipe.arguments.slug
)- extra
- preferenceName
The name of the preference that changed. Note that the value of the preference (old or new) is not given.
- reason
A code describing the reason that Normandy detected the preference change. Possible values are:
"atEnroll"
: The preferences already had user value when the experiment started."sideload"
: A preference was changed while Normandy’s observers weren’t active, likely while the browser was shut down."observer"
: The preference was observed to change by Normandy at runtime.
Preference Rollouts
- Enrollment
Sent when a user first enrolls in a rollout.
- method
The string
"enroll"
- object
The string
"preference_rollout"
- value
The slug of the rollout (
recipe.arguments.slug
)
- Enroll Failed
Sent when a user attempts to enroll in a rollout, but the enrollment process fails.
- method
The string
"enrollFailed"
- object
The string
"preference_rollout"
- value
The slug of the rollout (
recipe.arguments.slug
)- extra
- reason
A code describing the reason the unenroll failed. Possible values are:
"invalid type"
: The preferences specified in the rollout do not match the preferences built in to the browser. The represents a misconfiguration of the preferences in the recipe on the server."would-be-no-op"
: All of the preference specified in the rollout already have the given values. This represents an error in targeting on the server."conflict"
: At least one of the preferences specified in the rollout is already managed by another active rollout.
- preference
For
reason="invalid type"
, the first preference that was invalid. Forreason="conflict"
, the first preference that is conflicting.
- Update
Sent when the preferences specified in the recipe have changed, and the client updates the preferences of the browser to match.
- method
The string
"update"
- object
The string
"preference_rollout"
- value
The slug of the rollout (
recipe.arguments.slug
)- extra
- previousState
The state the rollout was in before this update (such as
"active"
or"graduated"
).
- Graduation
Sent when Normandy determines that further intervention is no longer needed for this rollout. After this point, Normandy will stop making changes to the browser for this rollout, unless the rollout recipe changes to specify preferences different than the built-in.
- method
The string
"graduate"
- object
The string
"preference_rollout"
- value
The slug of the rollout (
recipe.arguments.slug
)- extra
- reason
A code describing the reason for the graduation. Possible values are:
"all-prefs-match"
: All preferences specified in the rollout now have built-in values that match the rollouts values."in-graduation-set"
: The browser has changed versions (usually updated) to one that specifies this rollout no longer applies and should be graduated regardless of the built-in preference values. This behavior is controlled by the constantPreferenceRollouts.GRADUATION_SET
.
Add-on Studies
- Enrollment
- method
The string
"enroll"
- object
The string
"addon_study"
- value
The name of the study (
recipe.arguments.name
).- extra
- addonId
The add-on’s ID (example:
"feature-study@shield.mozilla.com"
).- addonVersion
The add-on’s version (example:
"1.2.3"
).
- Enroll Failure
- method
The string
"enrollFailed"
- object
The string
"addon_study"
- value
The name of the study (
recipe.arguments.name
).- reason
A string containing the filename and line number of the code that failed, and the name of the error thrown. This information is purposely limited to avoid leaking personally identifiable information. This should be considered a bug.
- Update
- method
The string
"update"
,- object
The string
"addon_study"
,- value
The name of the study (
recipe.arguments.name
).- extra
- addonId
The add-on’s ID (example:
"feature-study@shield.mozilla.com"
).- addonVersion
The add-on’s version (example:
"1.2.3"
).
- Update Failure
- method
The string
"updateFailed"
- object
The string
"addon_study"
- value
The name of the study (
recipe.arguments.name
).- extra
- reason
A string containing the filename and line number of the code that failed, and the name of the error thrown. This information is purposely limited to avoid leaking personally identifiable information. This should be considered a bug.
- Unenrollment
- method
The string
"unenroll"
.- object
The string
"addon_study"
.- value
The name of the study (
recipe.arguments.name
).- extra
- addonId
The add-on’s ID (example:
"feature-study@shield.mozilla.com"
).- addonVersion
The add-on’s version (example:
"1.2.3"
).- reason
The reason for unenrollment. Possible values are:
"install-failure"
: The add-on failed to install."individual-opt-out"
: The user opted-out of this particular study."general-opt-out"
: The user opted-out of studies in general."recipe-not-seen"
: The recipe was no longer applicable to this client. This can be because the recipe was disabled, or the user no longer matches the recipe’s filter."uninstalled"
: The study’s add-on as uninstalled by some mechanism. For example, this could be a user action or the add-on self-uninstalling."uninstalled-sideload"
: The study’s add-on was uninstalled while Normandy was inactive. This could be that the add-on is no longer compatible, or was manually removed from a profile."unknown"
: A reason was not specified. This should be considered a bug.