X
    Categories Performance Monitoring

C++ Crash Debugging without GDB

AppNeta no longer blogs on DevOps topics like this one.

Feel free to enjoy it, and check out what we can do for monitoring end user experience of the apps you use to drive your business at www.appneta.com.

In many cases, you can debug using GDB and core files directly. But what about logging without the external tools? In this post I’ll talk about how to log stack traces from within your application.

Logging Stack Traces

Ideally most issues such as a segmentation fault are caught during development.  However, it can be impossible to simulate all the environments your application will run in, and sometimes issues occur in the field that you are not able to easily replicate.  Normally, a production release is not compiled with debug symbols, and strip may have been run on it to make it smaller.  Also, a crash in production probably won’t generate a core file unless core file generation was explicitly enabled.  How do you get a stack trace?

Add a Signal Handler

There are two steps to getting your application to log a stack trace when it crashes.  The first is to set up a signal handler, and the second is to make a system call to get the stack trace.  When a program terminates abnormally a signal is generated.  You can create a signal handler to catch the signal:

<br />#include &lt;stdlib.h&gt;<br />#include &lt;stdio.h&gt;<br />#include &lt;signal.h&gt;<br /><br />extern "C" void<br />exitHandler (int signal, siginfo_t *info, void *sigcontext)<br />{<br />        if (signal != SIGINT &amp;&amp; signal != SIGTERM) {<br />                // Unexpected termination - log stack trace<br />                printf ("Unexpected termination - signal %d\n", signal);<br />                exit (1);<br />        }<br />        // Normal termination<br />        exit (0);<br />}<br /><br />void causeAnError (char&amp; a)<br />{<br />        char* p = NULL;<br />        a = *p;<br />}<br /><br />int<br />main (int argc, char *argv[])<br />{<br />        struct sigaction sa;<br /><br />        sa.sa_sigaction = exitHandler;<br />        sigemptyset (&amp;sa.sa_mask);<br />        sa.sa_flags = SA_RESTART|SA_SIGINFO;<br /><br />        sigaction (SIGINT, &amp;sa, NULL);<br />        sigaction (SIGTERM, &amp;sa, NULL);<br />        sigaction (SIGILL, &amp;sa, NULL);<br />        sigaction (SIGSEGV, &amp;sa, NULL);<br />        sigaction (SIGABRT, &amp;sa, NULL);<br /><br />        char ch;<br />        causeAnError (ch);<br />        exit (0);<br />}<br /><br />

Log the Stack Trace

The second step is to print the stack trace to a file.  The backtrace_symbols_fd() function writes strings to a file descriptor without calling malloc, so is safe to use in a signal handler.  To keep this example simple, I’ve opened a file in main() and then write to this file from the signal handler.  Another solution would be to have a separate watchdog process that listens to one end of a pipe, and then have backtrace_symbols_fd() write to that pipe.

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <signal.h>
#include <execinfo.h>
#include <sys/types.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

static int StackFd;

extern "C" void
exitHandler (int signal, siginfo_t *info, void *sigcontext)
{
        if (signal != SIGINT && signal != SIGTERM) {
                void *array[50];
                size_t size = backtrace (array, 50);

                // Unexpected termination - write the stack trace to a file
                backtrace_symbols_fd (array, size, StackFd);

                _exit (1);
        }
        // Normal termination
        _exit (0);
}

void causeAnError (char& a)
{
        char* p = NULL;
        a = *p;
}

int
main (int argc, char *argv[])
{
        struct sigaction sa;
        StackFd = open ("stack_trace.log", O_WRONLY|O_CREAT, S_IRWXU);
        sa.sa_sigaction = exitHandler;
        sigemptyset (&sa.sa_mask);
        sa.sa_flags = SA_RESTART|SA_SIGINFO;

        sigaction (SIGINT, &sa, NULL);
        sigaction (SIGTERM, &sa, NULL);
        sigaction (SIGILL, &sa, NULL);
        sigaction (SIGSEGV, &sa, NULL);
        sigaction (SIGABRT, &sa, NULL);

        char ch;
        causeAnError (ch);
        exit (0);
}

The output from the printed stack trace is not as nice as the gdb stack trace:

Unexpected termination - signal 11
[bt] ./debugTest() [0x400831]
[bt] /lib/libc.so.6(+0x33af0) [0x7f6c4739aaf0]
[bt] ./debugTest() [0x4008dc]
[bt] ./debugTest() [0x4009ad]
[bt] /lib/libc.so.6(__libc_start_main+0xfd) [0x7f6c47385c4d]
[bt] ./debugTest() [0x400729]

Getting More Information From Your Stack Trace

To track down the issue you will need a mapfile to translate the addresses in the stack trace to function names.  The linker option, -Map=<file name>, can be used to generate a mapfile.

If you don’t mind your program being a little larger, you can use the -rdynamic compiler option.  This option tells the linker to add all symbols to the dynamic symbol table.  Compiling with “g++ -rdynamic -o debugTest main.cpp”, the output is now:

<br />Unexpected termination - signal 11<br />[bt] ./debugTest(exitHandler+0x4d) [0x400ac1]<br />[bt] /lib/libc.so.6(+0x33af0) [0x7fb3fd362af0]<br />[bt] ./debugTest(_Z12causeAnErrorRc+0x14) [0x400b6c]<br />[bt] ./debugTest(main+0xc6) [0x400c3d]<br />[bt] /lib/libc.so.6(__libc_start_main+0xfd) [0x7fb3fd34dc4d]<br />[bt] ./debugTest() [0x4009b9]<br />

That still doesn’t give you a line number though.  If you compile your program with -g then the addr2line utility will give you the line number.  Even if your stack trace came from an executable that was not compiled with -g, recompiling with that option added will not alter the addresses.  For example, compiling with “g++ -rdynamic -g -o debugTest main.cpp” results in the same stack trace given above.  Then addr2line will give the line number of the error:

<br />$ addr2line -e debugTest 0x400b6c<br />main.cpp:32<br />

Whether a program crashes during development, QA, or in the worst case in production, it is beneficial to have logging in place that will capture a stack trace.  It is not always possible or convenient to reproduce an issue running from a debugger, so the stack trace log may be the quickest way to track down the root cause.

Team AppNeta: