.. -*- Mode: rst; fill-column: 80; -*- ==================== Junit Test Framework ==================== GeckoView has `a lot `_ of `custom `_ code that is used to run junit tests. This document is an overview of what this code does and how it works. .. contents:: Table of Contents :depth: 2 :local: Introduction ============ `GeckoView `_ is an Android Library that can be used to embed Gecko, the Web Engine behind Firefox, in applications. It is the foundation for Firefox on Android, and it is intended to be used to build Web Browsers, but can also be used to build other types of apps that need to display Web content. GeckoView itself has no UI elements besides the Web View and uses Java interfaces called "delegates" to let embedders (i.e. apps that use GeckoView) implement UI behavior. For example, when a Web page's JavaScript code calls ``alert('Hello')`` the embedder will receive a call to the `onAlertPrompt `_ method of the `PromptDelegate `_ interface with all the information needed to display the prompt. As most delegate methods deal with UI elements, GeckoView will execute them on the UI thread for the embedder's convenience. GeckoResult ----------- One thing that is important to understand for what follows is `GeckoResult `_. ``GeckoResult`` is a promise-like object that is used throughout the GeckoView API, it allows embedders to asynchronously respond to delegate calls and GeckoView to return results asynchronously. This is especially important for GeckoView as it never provides synchronous access to Gecko as a design principle. For example, when installing a WebExtension in GeckoView, the resulting `WebExtension `_ object is returned in a ``GeckoResult``, which is completed when the extension is fully installed: .. code:: java public GeckoResult install(...) To simplify memory safety, ``GeckoResult`` will always `execute callbacks `_ in the same thread where it was created, turning asynchronous code into single-threaded javascript-style code. This is currently `implemented `_ using the Android Looper for the thread, which restricts ``GeckoResult`` to threads that have a looper, like the Android UI thread. Testing overview ---------------- Given that GeckoView is effectively a translation layer between Gecko and the embedder, it's mostly tested through integration tests. The vast majority of the GeckoView tests are of the form: - Load simple test web page - Interact with the web page through a privileged JavaScript test API - Verify that the right delegates are called with the right inputs and most of the test framework is built around making sure that these interactions are easy to write and verify. Tests in GeckoView can be run using the ``mach`` interface, which is used by most Gecko tests. E.g. to run the `loadUnknownHost `_ test in ``NavigationDelegateTest`` you would type on your terminal: .. code:: shell ./mach geckoview-junit org.mozilla.geckoview.test.NavigationDelegateTest#loadUnknownHost Another way to run GeckoView tests is through the `Android Studio IDE `_. By running tests this way, however, some parts of the test framework are not initialized, and thus some tests behave differently or fail, as will be explained later. Testing envelope ---------------- Being a library, GeckoView has a natural, stable, testing envelope, namely the GeckoView API. The vast majority of GeckoView tests only use publicly-accessible APIs to verify the behavior of the API. Whenever the API is not enough to properly test behavior, the testing framework offers targeted "privileged" testing APIs. Using a restricted, stable testing envelope has proven over the years to be an effective way of writing consistent tests that don't break upon refactoring. Testing Environment ------------------- When run through ``mach``, the GeckoView junit tests run in a similar environment as mochitests (a type of Web regression tests used in Gecko). They have access to the mochitest web server at `example.com`, and inherit most of the testing prefs and profile. Note the environment will not be the same as mochitests when the test is run through Android Studio, the prefs will be inherited from the default GeckoView prefs (i.e. the same prefs that would be enabled in a consumer's build of GeckoView) and the mochitest web server will not be available. Tests account for this using the `isAutomation `_ check, which essentially checks whether the test is running under ``mach`` or via Android Studio. Unlike most other junit tests in the wild, GeckoView tests run in the UI thread. This is done so that the GeckoResult objects are created on the right thread. Without this, every test would most likely include a lot of blocks that run code in the UI thread, adding significant boilerplate. Running tests on the UI thread is achieved by registering a custom ``TestRule`` called `GeckoSessionTestRule `_, which, among other things, `overrides the evaluate `_ method and wraps everything into a ``instrumentation.runOnMainSync`` call. Verifying delegates =================== As mentioned earlier, verifying that a delegate call happens is one of the most common assertions that a GeckoView test makes. To facilitate that, ``GeckoSessionTestRule`` offers several ``delegate*`` utilities like: .. code:: java sessionRule.delegateUntilTestEnd(...) sessionRule.delegateDuringNextWait(...) sessionRule.waitUntilCalled(...) sessionRule.forCallbacksDuringWait(...) These all take an arbitrary delegate object (which may include multiple delegate implementations) and handle installing and cleaning up the delegate as needed. Another set of facilities that ``GeckoSessionTestRule`` offers allow tests to synchronously ``wait*`` for events, e.g. .. code:: java sessionRule.waitForJS(...) sessionRule.waitForResult(...) sessionRule.waitForPageStop(...) These facilities work together with the ``delegate*`` facilities by marking the ``NextWait`` or the ``DuringWait`` events. As an example, a test could load a page using ``session.loadUri``, wait until the page has finished loading using ``waitForPageStop`` and then verify that the expected delegate was called using ``forCallbacksDuringWait``. Note that the ``DuringWait`` here always refers to the last time a ``wait*`` method was called and finished executing. The next sections will go into how this works and how it's implemented. Tracking delegate calls ----------------------- One thing you might have noticed in the above section is that ``forCallbacksDuringWait`` moves "backward" in time by replaying the delegates called that happened while the wait was being executed. ``GeckoSessionTestRule`` achieves this by `injecting a proxy object `_ into every delegate, and `proxying every call `_ to the current delegate according to the ``delegate`` test calls. The proxy delegate `is built `_ using the Java reflection's ``Proxy.newProxyInstance`` method and receives `a callback `_ every time a method on the delegate is being executed. ``GeckoSessionTestRule`` maintains a list of `"default" delegates `_ used in GeckoView, and will `use reflection `_ to match the object passed into the ``delegate*`` calls to the proxy delegates. For example, when calling .. code:: java sessionRule.delegateUntilTestEnd(object : NavigationDelegate, ProgressDelegate {}) ``GeckoSessionTestRule`` will know to redirect all ``NavigationDelegate`` and ``ProgressDelegate`` calls to the object passed in ``delegateUntilTestEnd``. Replaying delegate calls ------------------------ Some delegate methods require output data to be passed in by the embedder, and this requires extra care when going "backward in time" by replaying the delegate's call. For example, whenever a page loads, GeckoView will call ``GeckoResult onLoadRequest(...)`` to know if the load can continue or not. When replaying delegates, however, we don't know what the value of ``onLoadRequest`` will be (or if the test is going to install a delegate for it, either!). What ``GeckoSessionTestRule`` does, instead, is to `return the default value `_ for the delegate method, and ignore the replayed delegate method return value. This can be a little confusing for test writers, for example this code `will not` stop the page from loading: .. code:: java session.loadUri("https://www.mozilla.org") sessionRule.waitForPageStop() sessionRule.forCallbacksDuringWait(object : NavigationDelegate { override fun onLoadRequest(session: GeckoSession, request: LoadRequest) : GeckoResult? { // this value is ignored return GeckoResult.deny() } }) as the page has already loaded by the time the ``forCallbacksDuringWait`` call is executed. Tracking Waits -------------- To track when a ``wait`` occurs and to know when to replay delegate calls, ``GeckoSessionTestRule`` `stores `_ the list of delegate calls in a ``List`` object, where ``CallRecord`` is a class that has enough information to replay a delegate call. The test rule will track the `start and end index `_ of the last wait's delegate calls and `replay it `_ when ``forCallbacksDuringWait`` is called. To wait until a delegate call happens, the test rule will first `examine `_ the already executed delegate calls using the call record list described above. If none of the calls match, then it will `wait for new calls `_ to happen, using ``UiThreadUtils.waitForCondition``. ``waitForCondition`` is also used to implement other type of ``wait*`` methods like ``waitForResult``, which waits until a ``GeckoResult`` is executed. ``waitForCondition`` runs on the UI thread, and it synchronously waits for an event to occur. The events it waits for normally execute on the UI thread as well, so it `injects itself `_ in the Android event loop, checking for the condition after every event has executed. If no more events remain in the queue, `it posts a delayed 100ms `_ task to avoid clogging the event loop. Executing Javascript ==================== As you might have noticed from an earlier section, the test rule allows tests to run arbitrary JavaScript code using ``waitForJS``. The GeckoView API, however, doesn't offer such an API. The way ``waitForJS`` and ``evaluateJS`` are implemented will be the focus of this section. How embedders run javascript ---------------------------- The only supported way of accessing a web page for embedders is to `write a built-in WebExtension `_ and install it. This was done intentionally to avoid having to rewrite a lot of the Web-Content-related APIs that the WebExtension API offers. GeckoView extends the WebExtension API to allow embedders to communicate to the extension by `overloading `_ the native messaging API (which is not normally implemented on mobile). Embedders can register themselves as a `native app `_ and the built-in extension will be able to `exchange messages `_ and `open ports `_ with the embedder. This is still a controversial topic among smaller embedders, especially solo developers, and we have discussed internally the possibility to expose a simpler API to run one-off javascript snippets, similar to what Chromium's WebView offers, but nothing has been developed so far. The test runner extension ------------------------- To run arbitrary javascript in GeckoView, the test runner installs a `support extension `_. The test framework then `establishes `_ a port for the background script, used to run code in the main process, and a port for every window, to be able to run javascript on test web pages. When ``evaluateJS`` is called, the test framework will send `a message `_ to the extension which then `calls eval `_ on it and returns the `JSON`-stringified version of the result `back `_ to the test framework. The test framework also supports promises with `evaluatePromiseJS `_. It works similarly to ``evaluateJS`` but instead of returning the stringified value, it `sets `_ the return value of the ``eval`` call into the ``this`` object, keyed by a randomly-generated UUID. .. code:: java this[uuid] = eval(...) ``evaluatePromiseJS`` then returns an ``ExtensionPromise`` Java object which has a ``getValue`` method on it, which will essentially execute `await this[uuid] `_ to get the value from the promise when needed. Beyond executing javascript --------------------------- A natural way of breaking the boundaries of the GeckoView API is to run a so-called "experiment extension". Experiment extensions have access to the full Gecko front-end, which is written in JavaScript, and don't have limits on what they can do. Experiment extensions are essentially what old add-ons used to be in Firefox, very powerful and very dangerous. The test runner uses experiments to offer `privileged APIs `_ to tests like ``setPref`` or ``getLinkColor`` (which is not normally available to websites for privacy concerns). Each privileged API is exposed as an `ordinary Java API `_ and the test framework doesn't offer a way to run arbitrary chrome code to discourage developers from relying too much on implementation-dependent privileged code.