Activity Stream Pings¶

The Activity Stream system add-on sends various types of pings to the backend (HTTPS POST) Onyx server :

a health ping that reports whether or not a user has a custom about:home or about:newtab page
a session ping that describes the ending of an Activity Stream session (a new tab is closed or refreshed), and
an event ping that records specific data about individual user interactions while interacting with Activity Stream
a performance ping that records specific performance related events
an undesired ping that records data about bad app states and missing data
an impression_stats ping that records data about Pocket impressions and user interactions

Schema definitions/validations that can be used for tests can be found in system-addon/test/schemas/pings.js.

Example Activity Stream `health` log¶

{
  "addon_version": "20180710100040",
  "client_id": "374dc4d8-0cb2-4ac5-a3cf-c5a9bc3c602e",
  "locale": "en-US",
  "version": "62.0a1",
  "release_channel": "nightly",
  "event": "AS_ENABLED",
  "value": 10,

  // These fields are generated on the server
  "date": "2016-03-07",
  "ip": "10.192.171.13",
  "ua": "python-requests/2.9.1",
  "receive_at": 1457396660000
}

Example Activity Stream `session` Log¶

{
  // These fields are sent from the client
  "action": "activity_stream_session",
  "addon_version": "20180710100040",
  "client_id": "374dc4d8-0cb2-4ac5-a3cf-c5a9bc3c602e",
  "locale": "en-US",
  "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"],
  "session_duration": 1635,
  "session_id": "{12dasd-213asda-213dkakj}",
  "profile_creation_date": 14786,
  "user_prefs": 7

  // These fields are generated on the server
  "date": "2016-03-07",
  "ip": "10.192.171.13",
  "ua": "python-requests/2.9.1",
  "receive_at": 1457396660000
}

Example Activity Stream `user_event` Log¶

{
  "action": "activity_stream_user_event",
  "action_position": "3",
  "addon_version": "20180710100040",
  "client_id": "374dc4d8-0cb2-4ac5-a3cf-c5a9bc3c602e",
  "event": "click or scroll or search or delete",
  "locale": "en-US",
  "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"],
  "source": "top sites, or bookmarks, or...",
  "session_id": "{12dasd-213asda-213dkakj}",
  "recommender_type": "pocket-trending",
  "metadata_source": "MetadataService or Local or TippyTopProvider",
  "user_prefs": 7

  // These fields are generated on the server
  "ip": "10.192.171.13",
  "ua": "python-requests/2.9.1",
  "receive_at": 1457396660000,
  "date": "2016-03-07",
}

Example Activity Stream `performance` Log¶

{
  "action": "activity_stream_performance_event",
  "addon_version": "20180710100040",
  "client_id": "374dc4d8-0cb2-4ac5-a3cf-c5a9bc3c602e",
  "event": "previewCacheHit",
  "event_id": "45f1912165ca4dfdb5c1c2337dbdc58f",
  "locale": "en-US",
  "page": "unknown", // all session-specific perf events should be part of the session perf object
  "receive_at": 1457396660000,
  "source": "TOP_FRECENT_SITES",
  "value": 1,
  "user_prefs": 7,

  // These fields are generated on the server
  "ip": "10.192.171.13",
  "ua": "python-requests/2.9.1",
  "receive_at": 1457396660000,
  "date": "2016-03-07"
}

Example Activity Stream `undesired event` Log¶

{
  "action": "activity_stream_undesired_event",
  "addon_version": "20180710100040",
  "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c",
  "event": "MISSING_IMAGE",
  "locale": "en-US",
  "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"]
  "source": "HIGHLIGHTS",
  "value": 0,
  "user_prefs": 7,

  // These fields are generated on the server
  "ip": "10.192.171.13",
  "ua": "python-requests/2.9.1",
  "receive_at": 1457396660000,
  "date": "2016-03-07"
}

Example Activity Stream `impression_stats` Logs¶

{
  "action": "activity_stream_impression_stats",
  "client_id": "n/a",
  "session_id": "n/a",
  "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}",
  "addon_version": "20180710100040",
  "locale": "en-US",
  "source": "pocket",
  "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"]
  "tiles": [{"id": 10000}, {"id": 10001}, {"id": 10002}]
  "user_prefs": 7
}

{
  "action": "activity_stream_impression_stats",
  "client_id": "n/a",
  "session_id": "n/a",
  "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}",
  "addon_version": "20180710100040",
  "locale": "en-US",
  "source": "pocket",
  "page": "unknown",
  "user_prefs": 7,

  // "pos" is the 0-based index to record the tile's position in the Pocket section.
  // "shim" is a base64 encoded shim attached to spocs, unique to the impression from the Ad server.
  "tiles": [{"id": 10000, "pos": 0, "shim": "enuYa1j73z3RzxgTexHNxYPC/b,9JT6w5KB0CRKYEU+"}],

  // A 0-based index to record which tile in the "tiles" list that the user just interacted with.
  "click|block|pocket": 0
}

Example Discovery Stream `SPOCS Fill` log¶

{
  // both "client_id" and "session_id" are set to "n/a" in this ping.
  "client_id": "n/a",
  "session_id": "n/a",
  "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}",
  "addon_version": "20180710100040",
  "locale": "en-US",
  "version": "68",
  "release_channel": "release",
  "spoc_fills": [
    {"id": 10000, displayed: 0, reason: "frequency_cap", full_recalc: 1},
    {"id": 10001, displayed: 0, reason: "blocked_by_user", full_recalc: 1},
    {"id": 10002, displayed: 0, reason: "below_min_score", full_recalc: 1},
    {"id": 10003, displayed: 0, reason: "flight_duplicate", full_recalc: 1},
    {"id": 10004, displayed: 0, reason: "probability_selection", full_recalc: 0},
    {"id": 10004, displayed: 0, reason: "out_of_position", full_recalc: 0},
    {"id": 10005, displayed: 1, reason: "n/a", full_recalc: 0}
  ]
}

Example Activity Stream `Router` Pings¶

{
  "client_id": "n/a",
  "action": ["snippets_user_event" | "onboarding_user_event"],
  "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}",
  "source": "pocket",
  "addon_version": "20180710100040",
  "locale": "en-US",
  "source": "NEWTAB_FOOTER_BAR",
  "message_id": "some_snippet_id",
  "event": "IMPRESSION",
  "event_context": "{\"view\":\"application_menu\"}"
}

KEY	DESCRIPTION
`action_position`	[Optional] The index of the element in the `source` that was clicked.	one
`action`	[Required] Either `activity_stream_event`, `activity_stream_session`, or `activity_stream_performance`.	one
`addon_version`	[Required] Firefox build ID, i.e. `Services.appinfo.appBuildID`.	one
`client_id`	[Required] An identifier for this client.	one
`card_type`	[Optional] (“bookmark”, “pocket”, “trending”, “pinned”, “search”, “spoc”, “organic”)	one
`search_vendor`	[Optional] the vendor of the search shortcut, one of (“google”, “amazon”, “wikipedia”, “duckduckgo”, “bing”, etc.). This field only exists when `card_type = "search"`	one
`date`	[Auto populated by Onyx] The date in YYYY-MM-DD format.	three
`shield_id`	[Optional] DEPRECATED: use experiments instead. The unique identifier for a specific experiment.	one
`event_id`	[Required] An identifier shared by multiple performance pings that describe an entire request flow.	one
`event`	[Required] The type of event. Any user defined string (“click”, “share”, “delete”, “more_items”)	one
`event_context`	[Optional] A string to record the context of an AS Router event ping. Compound context values will be stringified by JSON.stringify	one
`highlight_type`	[Optional] Either [“bookmarks”, “recommendation”, “history”].	one
`impression_id`	[Optional] The unique impression identifier for a specific client.	one
`ip`	[Auto populated by Onyx] The IP address of the client.	two
`locale`	[Auto populated by Onyx] The browser chrome’s language (eg. en-US).	two
`load_trigger_ts`	[Optional][Server Counter][Server Alert for too many omissions] DOMHighResTimeStamp of the action perceived by the user to trigger the load of this page.	one
`load_trigger_type`	[Server Counter][Server Alert for too many omissions] Either [“first_window_opened”, “menu_plus_or_keyboard”, “unexpected”].	one
`metadata_source`	[Optional] The source of which we computed metadata. Either (`MetadataService` or `Local` or `TippyTopProvider`).	one
`page`	[Required] One of [“about:newtab”, “about:home”, “about:welcome”, “unknown” (which either means not-applicable or is a bug)].	one
`recommender_type`	[Optional] The type of recommendation that is being shown, if any.	one
`session_duration`	[Optional][Server Counter][Server Alert for too many omissions] Time in (integer) milliseconds of the difference between the new tab becoming visible and losing focus	one
`session_id`	[Optional] The unique identifier for a specific session.	one
`source`	[Required] Either (“recent_links”, “recent_bookmarks”, “frecent_links”, “top_sites”, “spotlight”, “sidebar”) and indicates what `action`.	two
`received_at`	[Auto populated by Onyx] The time in ms since epoch.	three
`total_bookmarks`	[Optional] The total number of bookmarks in the user’s places db.	one
`total_history_size`	[Optional] The number of history items currently in the user’s places db.	one
`ua`	[Auto populated by Onyx] The user agent string.	two
`unload_reason`	[Required] The reason the Activity Stream page lost focus.	one
`url`	[Optional] The URL of the recommendation shown in one of the highlights spots, if any.	one
`value` (performance)	[Required] An integer that represents the measured performance value. Can store counts, times in milliseconds, and should always be a positive integer.	one
`value` (event)	[Optional] An object with keys “icon_type” and “card_type” to record the extra information for event ping	one
`ver`	[Auto populated by Onyx] The version of the Onyx API the ping was sent to.	one
`highlights_size`	[Optional] The size of the Highlights set.	one
highlights_data_late_by_ms	[Optional] Time in ms it took for Highlights to become initialized	one
topsites_data_late_by_ms	[Optional] Time in ms it took for TopSites to become initialized	one
topstories.domain.affinity .calculation.ms	[Optional] Time in ms it took for domain affinities to be calculated (topstories.domain.affinity.calculation.ms)	one
topsites_first_painted_ts	[Optional][Service Counter][Server Alert for too many omissions] Timestamp of when the Top Sites element finished painting (possibly with only placeholder screenshots)	one
`custom_screenshot`	[Optional] Number of topsites that display a custom screenshot.	one
`screenshot_with_icon`	[Optional] Number of topsites that display a screenshot and a favicon.	one
`screenshot`	[Optional] Number of topsites that display only a screenshot.	one
`tippytop`	[Optional] Number of topsites that display a tippytop icon.	one
`rich_icon`	[Optional] Number of topsites that display a high quality favicon.	one
`no_image`	[Optional] Number of topsites that have no screenshot.	one
`topsites_pinned`	[Optional] Number of topsites that are pinned.	one
topsites_search_shortcuts	[Optional] Number of search shortcut topsites.	one
visibility_event_rcvd_ts	[Optional][Server Counter][Server Alert for too many omissions] DOMHighResTimeStamp of when the page itself receives an event that document.visibilityState == visible.	one
`tiles`	[Required] A list of tile objects for the Pocket articles. Each tile object mush have a ID, optionally a “pos” property to indicate the tile position, and optionally a “shim” property unique to the impression from the Ad server	one
`click`	[Optional] An integer to record the 0-based index when user clicks on a Pocket tile.	one
`block`	[Optional] An integer to record the 0-based index when user blocks a Pocket tile.	one
`pocket`	[Optional] An integer to record the 0-based index when user saves a Pocket tile to Pocket.	one
`user_prefs`	[Required] The encoded integer of user’s preferences.	one & :four:
`is_preloaded`	[Required] A boolean to signify whether the page is preloaded or not	one
`icon_type`	[Optional] (“tippytop”, “rich_icon”, “screenshot_with_icon”, “screenshot”, “no_image”, “custom_screenshot”)	one
`region`	[Optional] A string maps to pref “browser.search.region”, which is essentially the two letter ISO 3166-1 country code populated by the Firefox search service. Note that: 1). it reports “OTHER” for those regions with smaller Firefox user base (less than 10000) so that users cannot be uniquely identified; 2). it reports “UNSET” if this pref is missing; 3). it reports “EMPTY” if the value of this pref is an empty string.	one
`profile_creation_date`	[Optional] An integer to record the age of the Firefox profile as the total number of days since the UNIX epoch.	one
`message_id`	[required] A string identifier of the message in Activity Stream Router.	one
`has_flow_params`	[required] One of [true, false]. A boolean identifier that indicates if Firefox Accounts flow parameters are set or unset.	one
`displayed`	[required] 1: a SPOC is displayed; 0: non-displayed	one
`reason`	[required] The reason if a SPOC is not displayed, “n/a” for the displayed, one of (“frequency_cap”, “blocked_by_user”, “flight_duplicate”, “probability_selection”, “below_min_score”, “out_of_position”, “n/a”)	one
`full_recalc`	[required] Is it a full SPOCS recalculation: 0: false; 1: true. Recalculation case: 1). fetch SPOCS from Pocket endpoint. Non-recalculation cases: 1). An impression updates the SPOCS; 2). Any action that triggers the `selectLayoutRender`	one
`experiments`	[Optional] An object to record all active experiments (an empty object will be sent if there is no active experiment). The experiments IDs are stored as keys, and the value object stores the branch information. Example: {“experiment_1”: {“branch”: “control”}, “experiment_2”: {“branch”: “treatment”}}. This deprecates the shield_id used in Activity Stream and Messaging System.	one
browser_session_id	[Optional] The unique identifier for a browser session, retrieved from TelemetrySession	one

Where:

:one: Firefox data
:two: HTTP protocol data
:three: server augmented data
:four: User preferences encoding table

Note: the following session-related fields are not yet implemented in the system-addon, but will likely be added in future versions:

{
  "total_bookmarks": 19,
  "total_history_size": 9,
  "highlights_size": 20
}

Encoding and decoding of `user_prefs`¶

This encoding mapping was defined in system-addon/lib/TelemetryFeed.jsm

Preference	Encoded value (binary)
showSearch	1 (00000001)
showTopSites	2 (00000010)
showTopStories	4 (00000100)
showHighlights	8 (00001000)
showSnippets	16 (00010000)
showSponsored	32 (00100000)
showCFRAddons	64 (01000000)
showCFRFeatures	128 (10000000)

Each item above could be combined with other items through bitwise OR (|) operation.

Examples:

Everything is on, user_prefs = 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 = 255
Everything is off, user_prefs = 0
Only show search and Top Stories, user_prefs = 1 | 4 = 5
Everything except Highlights, user_prefs = 1 | 2 | 4 | 16 | 32 | 64 | 128 = 247

Likewise, one can use bitwise AND (&) for decoding.

Check if everything is shown, user_prefs & (1 | 2 | 4 | 8 | 16 | 32 | 64 | 128) or user_prefs == 255
Check if everything is off, user_prefs == 0
Check if search is shown, user_prefs & 1
Check if both Top Sites and Top Stories are shown, (user_prefs & 2) && (user_prefs & 4), or (user_prefs & (2 | 4)) == (2 | 4)

Activity Stream Pings¶

Example Activity Stream health log¶

Example Activity Stream session Log¶

Example Activity Stream user_event Log¶

Example Activity Stream performance Log¶

Example Activity Stream undesired event Log¶

Example Activity Stream impression_stats Logs¶

Example Discovery Stream SPOCS Fill log¶

Example Activity Stream Router Pings¶

Encoding and decoding of user_prefs¶