How webext storage works

This document describes the implementation of the the storage.sync part of the WebExtensions Storage APIs. The implementation lives in the toolkit/components/extensions/storage folder

Ideally you would already know about Rust and XPCOM - see this doc for more details

At a very high-level, the system looks like:

        graph LR
    A[Extensions API]
    A --> B[Storage JS API]
    B --> C{magic}
    C --> D[app-services component]

Where “magic” is actually the most interesting part and the primary focus of this document.

Note: The general mechanism described below is also used for other Rust components from the app-services team - for example, “dogear” uses a similar mechanism, and the sync engines too (but with even more complexity) to manage the threads. Unfortunately, at time of writing, no code is shared and it’s not clear how we would, but this might change as more Rust lands.

The app-services component lives on github. There are docs that describe how to update/vendor this (and all) external rust code you might be interested in.

To set the scene, let’s look at the parts exposed to WebExtensions first; there are lots of moving part there too.

WebExtension API

The WebExtension API is owned by the addons team. The implementation of this API is quite complex as it involves multiple processes, but for the sake of this document, we can consider the entry-point into the WebExtension Storage API as being parent/ext-storage.js

This entry-point ends up using the implementation in the ExtensionStorageSync JS class. This class/module has complexity for things like migration from the earlier Kinto-based backend, but importantly, code to adapt a callback API into a promise based one.

Overview of the API

At a high level, this API is quite simple - there are methods to “get/set/remove” extension storage data. Note that the “external” API exposed to the addon has subtly changed the parameters for this “internal” API, so there’s an extension ID parameter and the JSON data has already been converted to a string. The semantics of the API are beyond this doc but are documented on MDN.

As you will see in those docs, the API is promise-based, but the rust implementation is fully synchronous and Rust knows nothing about Javascript promises - so this system converts the callback-based API to a promise-based one.

xpcom as the interface to Rust

xpcom is old Mozilla technology that uses C++ “vtables” to implement “interfaces”, which are described in IDL files. While this traditionally was used to interface C++ and Javascript, we are leveraging existing support for Rust. The interface we are exposing is described in mozIExtensionStorageArea.idl

The main interface of interest in this IDL file is mozIExtensionStorageArea. This interface defines the functionality - and is the first layer in the sync to async model. For example, this interface defines the following method:

interface mozIExtensionStorageArea : nsISupports {
    ...
    // Sets one or more key-value pairs specified in `json` for the
    // `extensionId`...
    void set(in AUTF8String extensionId,
            in AUTF8String json,
            in mozIExtensionStorageCallback callback);

As you will notice, the 3rd arg is another interface, mozIExtensionStorageCallback, also defined in that IDL file. This is a small, generic interface defined as:

interface mozIExtensionStorageCallback : nsISupports {
    // Called when the operation completes. Operations that return a result,
    // like `get`, will pass a `UTF8String` variant. Those that don't return
    // anything, like `set` or `remove`, will pass a `null` variant.
    void handleSuccess(in nsIVariant result);

    // Called when the operation fails.
    void handleError(in nsresult code, in AUTF8String message);
};

Note that this delivers all results and errors, so must be capable of handling every result type, which for some APIs may be problematic - but we are very lucky with this API that this simple XPCOM callback interface is capable of reasonably representing the return types from every function in the mozIExtensionStorageArea interface.

(There’s another interface, mozIExtensionStorageListener which is typically also implemented by the actual callback to notify the extension about changes, but that’s beyond the scope of this doc.)

Note the thread model here is async - the set call will return immediately, and later, on the main thread, we will call the callback param with the result of the operation.

So under the hood, what happens is something like:

        sequenceDiagram
    Extension->>ExtensionStorageSync: call `set` and give me a promise
    ExtensionStorageSync->>xpcom: call `set`, supplying new data and a callback
    ExtensionStorageSync-->>Extension: your promise
    xpcom->>xpcom: thread magic in the "bridge"
    xpcom-->>ExtensionStorageSync: callback!
    ExtensionStorageSync-->>Extension: promise resolved

So onto the thread magic in the bridge!

webext_storage_bridge

The webext_storage_bridge is a Rust crate which, as implied by the name, is a “bridge” between this Javascript/XPCOM world to the actual webext-storage crate.

lib.rs

Is the entry-point - it defines the xpcom “factory function” - an extern “C” function which is called by xpcom to create the Rust object implementing mozIExtensionStorageArea using existing gecko support.

area.rs

This module defines the interface itself. For example, inside that file you will find:

impl StorageSyncArea {
    ...

    xpcom_method!(
        set => Set(
            ext_id: *const ::nsstring::nsACString,
            json: *const ::nsstring::nsACString,
            callback: *const mozIExtensionStorageCallback
        )
    );
    /// Sets one or more key-value pairs.
    fn set(
        &self,
        ext_id: &nsACString,
        json: &nsACString,
        callback: &mozIExtensionStorageCallback,
    ) -> Result<()> {
        self.dispatch(
            Punt::Set {
                ext_id: str::from_utf8(&*ext_id)?.into(),
                value: serde_json::from_str(str::from_utf8(&*json)?)?,
            },
            callback,
        )?;
        Ok(())
    }

Of interest here:

xpcom_method is a Rust macro, and part of the existing xpcom integration which already exists in gecko. It declares the xpcom vtable method described in the IDL.
The set function is the implementation - it does string conversions and the JSON parsing on the main thread, then does the work via the supplied callback param, self.dispatch and a Punt.
The dispatch method dispatches to another thread, leveraging existing in-tree moz_task support, shifting the Punt to another thread and making the callback when done.

Punt

Punt is a whimsical name somewhat related to a “bridge” - it carries things across and back.

It is a fairly simple enum in punt.rs. It’s really just a restatement of the API we expose suitable for moving across threads. In short, the Punt is created on the main thread, then sent to the background thread where the actual operation runs via a PuntTask and returns a PuntResult.

There’s a few dances that go on, but the end result is that inner_run() gets executed on the background thread - so for Set:

Punt::Set { ext_id, value } => {
    PuntResult::with_change(&ext_id, self.store()?.get()?.set(&ext_id, value)?)?
}

Here, self.store() is a wrapper around the actual Rust implementation from app-services with various initialization and mutex dances involved - see store.rs. ie, this function is calling our Rust implementation and stashing the result in a PuntResult

The PuntResult is private to that file but is a simple struct that encapsulates both the actual result of the function (also a set of changes to send to observers, but that’s beyond this doc).

Ultimately, the PuntResult ends up back on the main thread once the call is complete and arranges to callback the JS implementation, which in turn resolves the promise created in ExtensionStorageSync.sys.mjs

End result:

        sequenceDiagram
    Extension->>ExtensionStorageSync: call `set` and give me a promise
    ExtensionStorageSync->>xpcom - bridge main thread: call `set`, supplying new data and a callback
    ExtensionStorageSync-->>Extension: your promise
    xpcom - bridge main thread->>moz_task worker thread: Punt this
    moz_task worker thread->>webext-storage: write this data to the database
    webext-storage->>webext-storage: done: result/error and observers
    webext-storage-->>moz_task worker thread: ...
    moz_task worker thread-->>xpcom - bridge main thread: PuntResult
    xpcom - bridge main thread-->>ExtensionStorageSync: callback!
    ExtensionStorageSync-->>Extension: promise resolved