This post is from a tip that Michael Ballantyne passed along to me.

Occasionally we encounter tests that fail infrequently. Debugging these can be very frustrating. Here is one technique that can help.

Debug build

Add
[code]
-DCMAKE_CXX_FLAGS=”-g -O3″
[/code]
to your cmake configure line to include debug symbols but still do optimization. Inlining means that backtraces may not show exactly where errors occur, but it’s close enough and more likely to correctly reproduce than a build without optimization.

Running gdb

Per this StackOverflow answer, put the following in a file (here named commandfile):

[code .sh]
set $_exitcode = -999
run
if $_exitcode != -999
quit
end
[/code]

Then loop until gdb stalls on an error. In bash:
[code]
while gdb ./heateqn -x commandfile; do :; done
[/code]

Finally, once gdb breaks on the segfault give it the command
backtrace to see details of the error.

Testing

If you’d like to make sure the script works, make a program that
always segfaults; put the following in segfault.c:
[code]
main;
[/code]

Compile and run as:
[code]
while gdb ./a.out -x commandfile; do :; done
[/code]

It should immediately halt in gdb. Type backtrace to see the trace,
or quit 1 to exit gdb with a non-zero exit code so that the while
loop terminates.