Testing with one-click loaners¶
Taskcluster is the task execution framework that supports Mozilla’s continuous integration and release processes.
Build and test jobs (like Marionette) are executed across all supported platforms, and job results are pushed to Treeherder for observation.
The best way to debug issues for intermittent test failures of Marionette tests for Firefox and Fennec (Android) is to use a one-click loaner as provided by Taskcluster. Such a loaner creates an interactive task you can interact with via a shell and VNC.
To create an interactive task for a Marionette job which is shown
as failed on Treeherder, select the job, click the ellipse in the lower
left pane, and choose
Create Interactive Task.
Please note that you need special permissions to actually request such a loaner.
When the task has been created you will receive an email with the connection details. Open the referenced shell and you will be connected via a WebSocket. Once that has been done a wizard will automatically launch and provide some options. Best here is to choose the second option, which will run all the setup steps, installs the Firefox or Fennec binary, and then exits.
Setting up the Marionette environment¶
Best here is to use a virtual environment, which has all the necessary packages installed. If no modifications to any Python package will be done, the already created environment by the wizard can be used:
% cd /builds/worker/workspace/build % source venv/bin/activate
Otherwise a new virtual environment needs to be created and populated with the mozbase and marionette packages installed:
% cd /builds/worker/workspace/build && rm -r venv % virtualenv venv && source venv/bin/activate % cd tests/mozbase && ./setup_development.py % cd ../marionette/client && python setup.py develop % cd ../harness && python setup.py develop % cd ../../../
Running Marionette tests¶
To run the Marionette tests execute the
runtests.py script. For all
the required options as best search in the log file of the failing job
the interactive task has been created from. Then copy the complete
command and run it inside the already sourced virtual environment:
% /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py --gecko-log=- -vv --binary=/builds/worker/workspace/build/application/firefox/firefox --address=127.0.0.1:2828 --symbols-path=https://queue.taskcluster.net/v1/task/GSuwee61Qyibujtxq4UV3A/artifacts/public/build/target.crashreporter-symbols.zip /builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini
The Marionette tests for Fennec are executed by using an Android emulator which runs on the host platform. As such some extra setup steps compared to Firefox on desktop are required.
The following lines set necessary environment variables before starting the emulator in the background, and to let Marionette know of various Android SDK tools.
% export ADB_PATH=/builds/worker/workspace/build/android-sdk-linux/platform-tools/adb % export ANDROID_AVD_HOME=/builds/worker/workspace/build/.android/avd/ % /builds/worker/workspace/build/android-sdk-linux/tools/emulator -avd test-1 -show-kernel -debug init,console,gles,memcheck,adbserver,adbclient,adb,avd_config,socket &
The actual call to
runtests.py is different per test job because
those are using chunks on Android. As best search for the command
and its options in the log file of the failing job the interactive
task has been created from. Then copy the complete command and run
it inside the already sourced virtual environment.
Here an example for chunk 1 which runs all the tests in the current chunk with some options for logs removed:
% /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py --emulator --app=fennec --package=org.mozilla.fennec_aurora --address=127.0.0.1:2828 /builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini --gecko-log=- --symbols-path=/builds/worker/workspace/build/symbols --startup-timeout=300 --this-chunk 1 --total-chunks 10
To execute a specific test only simply replace
with its name.