======================== How webext storage works ======================== This document describes the implementation of the the `storage.sync` part of the `WebExtensions Storage APIs `_. The implementation lives in the `toolkit/components/extensions/storage folder `_ Ideally you would already know about Rust and XPCOM - `see this doc for more details <../../../../writing-rust-code/index.html>`_ At a very high-level, the system looks like: .. mermaid:: graph LR A[Extensions API] A --> B[Storage JS API] B --> C{magic} C --> D[app-services component] Where "magic" is actually the most interesting part and the primary focus of this document. Note: The general mechanism described below is also used for other Rust components from the app-services team - for example, "dogear" uses a similar mechanism, and the sync engines too (but with even more complexity) to manage the threads. Unfortunately, at time of writing, no code is shared and it's not clear how we would, but this might change as more Rust lands. The app-services component `lives on github `_. There are docs that describe `how to update/vendor this (and all) external rust code <../../../../build/buildsystem/rust.html>`_ you might be interested in. To set the scene, let's look at the parts exposed to WebExtensions first; there are lots of moving part there too. WebExtension API ################ The WebExtension API is owned by the addons team. The implementation of this API is quite complex as it involves multiple processes, but for the sake of this document, we can consider the entry-point into the WebExtension Storage API as being `parent/ext-storage.js `_ This entry-point ends up using the implementation in the `ExtensionStorageSync JS class `_. This class/module has complexity for things like migration from the earlier Kinto-based backend, but importantly, code to adapt a callback API into a promise based one. Overview of the API ################### At a high level, this API is quite simple - there are methods to "get/set/remove" extension storage data. Note that the "external" API exposed to the addon has subtly changed the parameters for this "internal" API, so there's an extension ID parameter and the JSON data has already been converted to a string. The semantics of the API are beyond this doc but are `documented on MDN `_. As you will see in those docs, the API is promise-based, but the rust implementation is fully synchronous and Rust knows nothing about Javascript promises - so this system converts the callback-based API to a promise-based one. xpcom as the interface to Rust ############################## xpcom is old Mozilla technology that uses C++ "vtables" to implement "interfaces", which are described in IDL files. While this traditionally was used to interface C++ and Javascript, we are leveraging existing support for Rust. The interface we are exposing is described in `mozIExtensionStorageArea.idl `_ The main interface of interest in this IDL file is `mozIExtensionStorageArea`. This interface defines the functionality - and is the first layer in the sync to async model. For example, this interface defines the following method: .. code-block:: rust interface mozIExtensionStorageArea : nsISupports { ... // Sets one or more key-value pairs specified in `json` for the // `extensionId`... void set(in AUTF8String extensionId, in AUTF8String json, in mozIExtensionStorageCallback callback); As you will notice, the 3rd arg is another interface, `mozIExtensionStorageCallback`, also defined in that IDL file. This is a small, generic interface defined as: .. code-block:: cpp interface mozIExtensionStorageCallback : nsISupports { // Called when the operation completes. Operations that return a result, // like `get`, will pass a `UTF8String` variant. Those that don't return // anything, like `set` or `remove`, will pass a `null` variant. void handleSuccess(in nsIVariant result); // Called when the operation fails. void handleError(in nsresult code, in AUTF8String message); }; Note that this delivers all results and errors, so must be capable of handling every result type, which for some APIs may be problematic - but we are very lucky with this API that this simple XPCOM callback interface is capable of reasonably representing the return types from every function in the `mozIExtensionStorageArea` interface. (There's another interface, `mozIExtensionStorageListener` which is typically also implemented by the actual callback to notify the extension about changes, but that's beyond the scope of this doc.) *Note the thread model here is async* - the `set` call will return immediately, and later, on the main thread, we will call the callback param with the result of the operation. So under the hood, what happens is something like: .. mermaid:: sequenceDiagram Extension->>ExtensionStorageSync: call `set` and give me a promise ExtensionStorageSync->>xpcom: call `set`, supplying new data and a callback ExtensionStorageSync-->>Extension: your promise xpcom->>xpcom: thread magic in the "bridge" xpcom-->>ExtensionStorageSync: callback! ExtensionStorageSync-->>Extension: promise resolved So onto the thread magic in the bridge! webext_storage_bridge ##################### The `webext_storage_bridge `_ is a Rust crate which, as implied by the name, is a "bridge" between this Javascript/XPCOM world to the actual `webext-storage `_ crate. lib.rs ------ Is the entry-point - it defines the xpcom "factory function" - an `extern "C"` function which is called by xpcom to create the Rust object implementing `mozIExtensionStorageArea` using existing gecko support. area.rs ------- This module defines the interface itself. For example, inside that file you will find: .. code-block:: rust impl StorageSyncArea { ... xpcom_method!( set => Set( ext_id: *const ::nsstring::nsACString, json: *const ::nsstring::nsACString, callback: *const mozIExtensionStorageCallback ) ); /// Sets one or more key-value pairs. fn set( &self, ext_id: &nsACString, json: &nsACString, callback: &mozIExtensionStorageCallback, ) -> Result<()> { self.dispatch( Punt::Set { ext_id: str::from_utf8(&*ext_id)?.into(), value: serde_json::from_str(str::from_utf8(&*json)?)?, }, callback, )?; Ok(()) } Of interest here: * `xpcom_method` is a Rust macro, and part of the existing xpcom integration which already exists in gecko. It declares the xpcom vtable method described in the IDL. * The `set` function is the implementation - it does string conversions and the JSON parsing on the main thread, then does the work via the supplied callback param, `self.dispatch` and a `Punt`. * The `dispatch` method dispatches to another thread, leveraging existing in-tree `moz_task `_ support, shifting the `Punt` to another thread and making the callback when done. Punt ---- `Punt` is a whimsical name somewhat related to a "bridge" - it carries things across and back. It is a fairly simple enum in `punt.rs `_. It's really just a restatement of the API we expose suitable for moving across threads. In short, the `Punt` is created on the main thread, then sent to the background thread where the actual operation runs via a `PuntTask` and returns a `PuntResult`. There's a few dances that go on, but the end result is that `inner_run() `_ gets executed on the background thread - so for `Set`: .. code-block:: rust Punt::Set { ext_id, value } => { PuntResult::with_change(&ext_id, self.store()?.get()?.set(&ext_id, value)?)? } Here, `self.store()` is a wrapper around the actual Rust implementation from app-services with various initialization and mutex dances involved - see `store.rs`. ie, this function is calling our Rust implementation and stashing the result in a `PuntResult` The `PuntResult` is private to that file but is a simple struct that encapsulates both the actual result of the function (also a set of changes to send to observers, but that's beyond this doc). Ultimately, the `PuntResult` ends up back on the main thread once the call is complete and arranges to callback the JS implementation, which in turn resolves the promise created in `ExtensionStorageSync.sys.mjs` End result: ----------- .. mermaid:: sequenceDiagram Extension->>ExtensionStorageSync: call `set` and give me a promise ExtensionStorageSync->>xpcom - bridge main thread: call `set`, supplying new data and a callback ExtensionStorageSync-->>Extension: your promise xpcom - bridge main thread->>moz_task worker thread: Punt this moz_task worker thread->>webext-storage: write this data to the database webext-storage->>webext-storage: done: result/error and observers webext-storage-->>moz_task worker thread: ... moz_task worker thread-->>xpcom - bridge main thread: PuntResult xpcom - bridge main thread-->>ExtensionStorageSync: callback! ExtensionStorageSync-->>Extension: promise resolved