Crash Monitoring

Important

  • The main goal here is not to file an issue for every single distinct crash report, but to find regressions of new problems that need to be addressed.

  • Once you’re familiar with the process this should not take more than 10 mins in the morning.

  • Quick links that you should check everyday Sentry Query and Socorro Query

Things to note before starting:

  • Since we are focused on Java crashes, Sentry currently is the better choice.

  • Ignore level:info issues (blue labels in Sentry). These issues are informational only.

What to do when monitoring crashes:

  • Look at Sentry Fenix-nightly overview. Go though trending issues. Sentry Dashboard.

  • Look at Sentry custom search Sentry Query.

  • Sign up for https://groups.google.com/a/mozilla.org/g/stability to get a daily email on the stability of Fenix.

How to determine a crash requires actions:

  • Crashes that have level either Fatal or Error.

  • Crashes that are occurring on latest Fenix and A-C versions.

  • Crashes that are spiking.

  • Crashes that are new.

  • Crashes that happen repeatedly and often.

  • Crashes that are happening to multiple users.

When a crash report requires actions:

  • Is this a crash due to a recent change? If so, contact the developer.

    • The histogram on the right side can help determine this along with checking the Firefox-Beta and Firefox Sentry products.

  • Triage the crash to determine if the issue is real and requires a Bugzilla issue to track it.

    • When filing an issue add a link to it as a comment in the Sentry crash for the products (nightly, beta, release) where the crash appears.

  • Notify the relevant teams on Slack/Matrix that there’s a new crash in Nightly that needs urgent attention, e.g. #synced-client-integrations for all things involving application services (A-S), #nimbus-rust-sdk for Nimbus, and GeckoView on Matrix.

What can you do to help when not monitoring crashes

  • If you recently landed a new module / change that is significant, contact the crash monitor so they are aware of it.

Crash monitoring with Socorro

  • Look at Top Crashers for Fenix Nightly for reports on Nightly builds.

    • This will return zero results if GV build ID is greater than 3 days old. Change to the 7 day view and ask #releaseduty-mobile in Slack about the GV upgrade task being broken.

  • Use Sentry and Bugzilla to determine if the crash has already been reported. If a Bugzilla bug has been filed for a crash, a link to the bug should be listed in Socorro’s “Bugzilla IDs” column.

  • If the crash is new and the volume is high, then consider filing an issue using the crash-stats Bugzilla tab from a crash ID.

How to file bugs in Bugzilla from Socorro

  • Go to the crash report.

  • “Reports” tab.

  • Select a report.

  • Select the “Bugzilla” tab.

  • Just above the header “Related Bugs”, you will see a “Create a bug” option. Select the product that matches the crash report.