Symbolicating TreeHerder stacks locally¶
When using tools like the Dark Matter Detector (DMD) or refcount logging to investigate issues occurring on TreeHerder that you can’t reproduce locally, you can often end up with unsymbolicated stacks. Fortunately, there is a way to symbolicate these stacks on your own machine.
These instructions are for a Linux TreeHerder build for MacOS, so they might require some modifications for other combinations of platforms.
target.crashreporter-symbols.zip from the
Build job. Note that these files are very large so you’ll want to delete
them and the extracted files when you are done.
These files each contain a large number of files, so I’d recommend creating
a directory for each of them. Call these
and move the prior two files into these two directories, respectively.
<TARGET_DIR> and run something like
tar xf target.tar.bz2
then go to
<SYMB_DIR> and run something like
You should be able to delete the two original files now.
Next we need to ensure that the locations of binaries are rewritten from
where they are on TreeHerder to where we have them locally. We’ll do this by
fix_stacks.py. This file is located in the
tools/rb/ directory of
the Firefox source directory. You need to add these two lines to the function
line_str is defined and before it is written to
fix_stacks.stdin. I’ve done this right before the definition of
line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/firefox", "<TARGET_DIR>/firefox/firefox-bin") line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/libxul.so", "<TARGET_DIR>/firefox/libxul.so")
The initial locations should appear verbatim in the stack you are trying to
symbolicate, so double check that they match. Also,
<TARGET_DIR> of course
needs to be replaced with the actual local directories where those files are
located. Note that the
firefox executable is changed to
I don’t know why that is necessary, but only the latter existed for me.
Finally, we need to make it so that the stack fixer can find the location of
the breakpad symbols we downloaded. If you are running
dmd.py or directly (in a recent version), you can do this by running with the
BREAKPAD_SYMBOLS_PATH set to the
<SYMB_DIR> from above.
If that doesn’t work, you’ll have to edit
With all of that done, you should now be able to run
to fix the stacks. Note that the stack fixing process can take a minute or two.