GDB - How does it know about function calling stack?

Debuggers like GDB have two primary means of walking the stack in order to print a backtrace. They either assume the value in the frame pointer register (RBP) is a pointer to start of a linked list of stack frames, or they use special unwind info stored in the executable that describes how to walk (unwind) the stack.

Using the frame pointer

When using the frame pointer, the assumption is that it points to where the current function saved the value of its caller's frame pointer. It also assumes that just before that saved frame pointer is where the return address for the current function is stored. So that's how it knows both what the RBP value of the calling function was, and what function called the current function, which it can easily determine from the return address. It can then find all the previous stack frames and functions on the stack by walking the linked RBP values.

However, this assumes that functions use the frame pointer this way, and generally there's no guarantee that they will. Basically it's assuming that the function prologue and epilogue looks something like this:

func:
    push %rbp         # save previous frame pointer
    mov  %rsp, %rbp   # new frame pointer points to previous value
    sub  $24, %rsp    # allocate stack space for this funciton

    ...

    pop %rbp          # restore previous frame pointer
    ret

But when optimizing many compilers won't do this, because they rarely need to use a frame pointer and instead will treat RBP like any other general purpose register and use it for something else.

Using unwind info

So to generate a backtrace across functions that don't use RBP as a frame pointer, a debugger can potentially use unwind information. This is special data stored in executables (and dynamic libraries) that describes exactly how to virtually undo all the stack operations performed by a function at any point in the execution of that function. The format and location of the unwind info varies based on the executable format and CPU type. For ELF x86-64 executables the unwind info is stored in the .eh_frame section in a format based on the DWARF debugging format's unwind info. This format is too complex to describe here, but you can read the System V AMD64 ABI for more details.