Today, we are going to dig into a rabbit hole: searching an exception with our mate gdb!
source: https://pixabay.com/photos/message-in-a-bottle-bottle-sea-3437294/ [pixabay licence]]

I recently faced a complicated situation to debug. Unfortunately, my logs were not detailed enough to localize it. The only thing I had was a log with the backtrace of where my exception was caught, and it was a generic handler, catching all exceptions before displaying an error to the user.

So I started-up gdb, put a few breakpoints, run the application and found the faulty part of my code. This post describes what I did.

Prerequisite

I consider that the reader knows how to start gdb, attach it to another process (either by starting this other process within gdb, or by attaching gdb to the PID of the other process). If the reader wants to do more investigation within gdb, basic navigation (continue, step, next, finish, …) is also assumed.

Detailed process

First of all I started gdb. Since the application I was debugging is a server, I attached gdb to the PID of this already running application.

The first thing I did was to create a catchpoint that will stop the execution of the program as soon as an exception is thrown.

catch throw

Since my application uses a lot of exceptions, I had to find a way to enable this catchpoint at the right moment (otherwise gdb would stop immediately). I disabled the breapoint 1 (you can check the state of any breakpoint with info breakpoints).

disable 1

So then, I looked in the logs of my application and created a breakpoint on the last known line that my code was executing before throwing the exception. That line had to be in a place where the code was executed only once in this process, otherwise the catchpoint would have been enabled way too early. The syntax that I prefer to create breapoint is break file:line_number. Like most gdb command, you can use an abbreviated form (ex: b foo.cpp:42).

break some_file.cpp:42

With info b I was able to know witch number was associated with each breakpoint. The catchpoint on the exception thrown was the breakpoint 1 and my second breakpoint was 2.

(gdb) info breakpoints
Num     Type           Disp Enb Address            What
1       breakpoint     keep n   <PENDING>          exception throw
2       breakpoint     keep y   0x0000000000001191 in some_file.cpp:13

What I wanted was that when I was going to resume the execution of my software, gdb would go through breakpoint 2, then stop on the breakpoint 1 waiting for inputs. Given that my application was multithreaded, I had to spend the least amount of time between the two. Otherwise, an exception in any other thread could be thrown in the meantime, and therefore gdb would bring me to this totally unrelated part of the code. So I entered a list of gdb commands that was going to be executed automatically as soon as the breakpoint was going to be hit.

First of all, on the breakpoint 2, I told gdb to enable the catchpoint 1, then immediately continue the execution of the program.

commands 2
  enable 1
  continue
end

Then, and since I wanted to be able to run gdb multiple times in a row, I also automatically disabled the catchpoint 1 as soon at it was reached.

commands 1
  disable 1
end

Everything was ready, I just had to continue the execution of my software:

continue

I run a scenario that was going to trigger the bug on another terminal, then I waited for the breakpoint 2 to be hit. Gdb run the associated commands: enabling the catchpoint 1, and continuing the execution. Finally ,the breakpoint 1 was hit, gdb disabled it, and eventually waited for inputs.

At this point I was able to start my investigation. First of all, I prefer to use the TUI to be able to see my source code by typing ctrl+x then a.

ctrl+x, a

I used a mix of those commands to navigate in the backtrace:

 - backtrace
 - up
 - down

And I displayed the value of any expression with print (abbreviation p):

 - print myVariable
 - print foo(myVariable)
 - print *myPointer

Sum up

  • Put a breakpoint: b file:line_number
  • Put a breakpoint on exception thrown catch throw
  • Display information about all breakpoint info break
  • Bind a list of command to a specific breakpoint commands breakpoint_number
  • Enable and disable breakpoints: enable breakpoint_number/disable breakpoint_number
  • Continue the execution of the program: continue
  • Print an expression: print expression
  • Toggle TUI: ctrl+x, a
  • Display the backtrace: backtrace
  • Navigate in the backtrace: up/down

Creative Commons License