Data Collection

This document describes the types of data that Shield collects.

Uptake

Shield monitors the execution of recipes and reports to Telemetry the amount of successful and failed runs. This data is reported using Uptake Telemetry under the shield-recipe-client namespace.

Runner Status

Once per-fetch and execution of recipes, one of the following statuses is reported under the key shield-recipe-client/runner:

RUNNER_INVALID_SIGNATURE

Shield failed to verify the signature of the fetched recipes.

RUNNER_NETWORK_ERROR

There was a network-related error while fetching recipes.

RUNNER_SERVER_ERROR

The data returned by the server when fetching the recipe is invalid in some way.

RUNNER_SUCCESS

The operation completed successfully. Individual failures with actions and recipes may have been reported separately.

Action Status

For each action available from the Shield service, one of the following statuses is reported under the key shield-recipe-client/action/<action name>:

ACTION_NETWORK_ERROR

There was a network-related error while fetching actions

ACTION_PRE_EXECUTION_ERROR

There was an error while running the pre-execution hook for the action.

ACTION_POST_EXECUTION_ERROR

There was an error while running the post-execution hook for the action.

ACTION_SERVER_ERROR

The data returned by the server when fetching the action is invalid in some way.

ACTION_SUCCESS

The operation completed successfully. Individual failures with recipes may be reported separately.

Recipe Status

For each recipe that is fetched and executed, one of the following statuses is reported under the key shield-recipe-client/recipe/<recipe id>:

RECIPE_ACTION_DISABLED

The action for this recipe failed in some way and was disabled, so the recipe could not be executed.

RECIPE_EXECUTION_ERROR

An error occurred while executing the recipe.

RECIPE_INVALID_ACTION

The action specified by the recipe was invalid and it could not be executed.

RECIPE_SUCCESS

The recipe was executed successfully.

Enrollment

Shield records enrollment and unenrollment of users into studies, and records that data using Telemetry Events. All data is stored in the normandy category.

Preference Studies

Enrollment
method
The string "enroll"
object
The string "preference_study"
value
The name of the study (recipe.arguments.slug).
extra
branch
The name of the branch the user was assigned to (example: "control" or "experiment").
experimentType
The type of preference experiment. Currently this can take values “exp” and “exp-highpop”, the latter being for experiments targetting large numbers of users.
Unenrollment
method
The string "unenroll".
object
The string "preference_study".
value
The name of the study (recipe.arguments.slug).
extra
didResetValue
The string "true" if the preference was set back to its original value, "false" if it was left as its current value. This can happen when, for example, the user changes a preference that was involved in a user-branch study.
reason

The reason for unenrollment. Possible values are:

  • "recipe-not-seen": The recipe was no longer applicable to this client This can be because the recipe was disabled, or the user no longer matches the recipe’s filter.
  • "user-preference-changed": The study preference was changed on the user branch. This could mean the user changed the preference, or that some other mechanism set a non-default value for the preference.
  • "user-preference-changed-sideload": The study preference was changed on the user branch while Shield was inactive. This could mean that the value was manually changed in a profile while Firefox was not running.
  • "unknown": A reason was not specificied. This should be considered a bug.

Add-on Studies

Enrollment
method
The string "enroll"
object
The string "addon_study"
value
The name of the study (recipe.arguments.slug).
extra
addonId
The add-on’s ID (example: "feature-study@shield.mozilla.com").
addonVersion
The add-on’s version (example: "1.2.3").
Enroll Failure
method
The string "enrollFailed"
object
The string "addon_study"
value
The name of the study (recipe.arguments.slug).
reason
A string containing the filename and line number of the code that failed, and the name of the error thrown. This information is purposely limited to avoid leaking personally identifiable information. This should be considered a bug.
Unenrollment
method
The string "unenroll".
object
The string "addon_study".
value
The name of the study (recipe.arguments.name).
extra
addonId
The add-on’s ID (example: "feature-study@shield.mozilla.com").
addonVersion
The add-on’s version (example: "1.2.3").
reason

The reason for unenrollment. Possible values are:

  • "install-failure": The add-on failed to install.
  • "individual-opt-out": The user opted-out of this particular study.
  • "general-opt-out": The user opted-out of studies in general.
  • "recipe-not-seen": The recipe was no longer applicable to this client. This can be because the recipe was disabled, or the user no longer matches the recipe’s filter.
  • "uninstalled": The study’s add-on as uninstalled by some mechanism. For example, this could be a user action or the add-on self-uninstalling.
  • "uninstalled-sideload": The study’s add-on was uninstalled while Shield was inactive. This could be that the add-on is no longer compatible, or was manually removed from a profile.
  • "unknown": A reason was not specified. This should be considered a bug.