In order for Socorro to produce useful stack traces for crash reports, it needs debug symbols matching the libraries that are loaded in each report. Currently these symbols are all in the Breakpad text symbol format, but there is work in progress (bug 607831, symboltests, minidump-stackwalk, binary-symbols branch) to make them use the new binary format that is implemented in upstream Breakpad.

How Socorro locates symbols

Currently, Socorro uses the stock minidump_stackwalk tool from upstream Breakpad to process minidumps into usable crash reports. minidump_stackwalk uses SimpleSymbolSupplier to locate symbols. SimpleSymbolSupplier simply looks for symbol files in a known directory structure given a list of symbol directories. Each library in a crash report has a "debug filename", which is simply the library filename everywhere except Windows, where it is the PDB filename, and a "debug identifier", which is a GUID+1 extra character that uniquely identifies a compiled version of a library. On Windows this is a GUID+age field that is baked into the binary by the linker. On other platforms this is a hash of certain parts of the file. The SimpleSymbolSupplier looks for symbol files in file paths of the form:

 <debug filename>/<debug identifier>/<debug filename>.sym

for example:

 firefox.pdb/C48C38C9E37E4A689D57C7509EEB017A2/firefox.sym

How symbols get to the symbol server

Mozilla nightly/release builds

Mozilla nightly and release builds run two makefile targets after the build is complete to generate a symbol package and upload it: buildsymbols and uploadsymbols.

The buildsymbols target runs the platform-specific dump_syms tool (Linux, Mac, Windows) via the symbolstore.py script in order to generate a symbol package with the correct directory structure. In addition to Breakpad-formatted symbols, symbolstore.py includes platform-native debug symbols (PDB files on Windows, .dSYM files on Mac, stripped debug sections on Linux) so that users can use the symbol server to debug nightly and release builds.

The uploadsymbols target calls the upload_symbols.sh script, which scp's a symbol package in zip format to the symbol server, unpacks it, and runs a post-symbol-upload command.

Windows system symbols

PDB symbols containing function names and stack walking information are available for Windows system libraries from Microsoft's symbol server. The Breakpad source contains a utility library that can fetch these PDB files and convert them to the Breakpad symbol format. There is a small utility binary built using this library which we use in a Python script. The Python script gets its input from a Socorro cron job which runs a map/reduce job to gather a list of modules from Windows crash reports.

The script is somewhat complex, because the symbol fetching and conversion needs to happen on a Windows machine, and in Mozilla's production environment, this machine cannot write directly to the symbol store, so it has a read-only mount of the symbol store, and uploads the symbols using SCP.

Note: This is not currently being run in production, but only on an ad-hoc basis. It is expected to move to production in the Socorro 1.7.7 release See bug 598908.

OS X system symbols

There is no central source for OS X debug symbols, so I wrote a Python script that runs dump_syms on system libraries to get the exported function names and stack walking information, and uploads them to a small RESTful symbol server. This symbol server lives on a separate machine from the actual production symbol server, so there's also another script involved that uploads symbols from there to production.

The script itself is intended to crowdsource the gathering of symbols, but at present it's likely that I'm the only one running it on a regular basis.

Android system symbols

Because of the large amount of Android crashes happening in system libraries, I wrote an extension to gather symbols from Android system libraries: https://addons.mozilla.org/en-US/mobile/addon/android-symbol-sender/

The extension code is here: http://hg.mozilla.org/users/tmielczarek_mozilla.com/androidsymbols/

but it also uses a bunch of the Breakpad code for symbol dumping, compiled into a shared library loaded via js-ctypes. It also relies on a few Breakpad patches in my patch queue: http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/file/3d8e40b12dfe

most notably one that uses some libunwind code to parse the ARM unwind tables: http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/file/3d8e40b12dfe/arm-unwind

The extension submits symbols to the same REST symbol server used for OS X system symbols.

Linux system symbols

We currently have no plans for collecting Linux system symbols. The huge variety of distributions and library versions makes this a difficult problem.

Flash Player symbols

Currently Adobe sends us symbols for Windows and Mac Flash Player releases. More information is available on the intranet wiki.

Symbol Cleanup

There is a symbol cleanup script that runs as a cron job on the #Symbol upload server. Currently it never removes symbols from alpha/beta/release builds, and tries to keep 30 days worth of nightly symbols per product-branch-os grouping. The cleanup script works by parsing the #Symbol Index Files, so symbols not listed in index files will not be cleaned up.

Extra Details

Symbol upload server

All symbols are currently uploaded via SSH to symbolpush.mozilla.org, which is a server that exists for this express purpose. The actual symbol storage is a NFS mount that is exposed to that server. This hardware is located in Mozilla's Phoenix data center.

Symbol Index Files

Every symbol upload includes an index file which lists all the symbols that are contained within this upload. The format is:

 ${appname}-${version}-${os}-${YYYYMMDDHHMMSS}[-${extra}]-symbols.txt

for example:

 firefox-4.0-WINNT-20110318052756-symbols.txt

The post-symbol-upload script

All upload processes run a post-symbol-upload script after uploading symbols. This script is currently a no-op, but it may change in the future if we decide to change the format in which symbols are stored.