Using gdb

← Back to the Lab 1 Guide

gdb stops the kernel at any function, shows the chain of calls that led there, and prints the fields of every process. It helps most with a panic or a hang, where ./grade shows no log. For a wrong log, start with Oops! What's Wrong?.

Two Terminals

You need two terminals in the same container: one for xv6, one for gdb. How to open the second one depends on how you started the container:

You started the container withOpen a second terminal with
docker run … --name xv6 …a new terminal on your computer, then docker exec -it -w /xv6 xv6 bash
VS Code, "Reopen in Container"Terminal → Split Terminal
./lab shelltmux inside the container (below)

tmux splits one terminal into panes. Run tmux, then press Ctrl-b and then % to split the window into a left and a right pane. Ctrl-b and an arrow key moves between panes. Ctrl-b and [ lets you scroll back with the arrow keys, and q stops scrolling. Type exit in a pane to close it.

A First Session

  1. In the first terminal, run make qemu-gdb. QEMU starts xv6, stops it before its first instruction, and waits for gdb.
  2. In the second terminal, in the same folder, run gdb-multiarch, not gdb, although make qemu-gdb says "run 'gdb'". Plain gdb in the container cannot read the RISC-V kernel. gdb-multiarch reads the file .gdbinit that make qemu-gdb wrote, loads the kernel symbols and connects to QEMU.
  3. Type continue in gdb. xv6 boots in the first terminal.
  4. Run a test in the first terminal, for example schedcheck late. Press Ctrl-c in gdb to stop xv6 at any moment, and continue to go on.

Here is part of a session on q0, with addresses and line numbers replaced by …:

(gdb) break sched_timer
Breakpoint 1 at …: file kernel/sched.c, line ….
(gdb) continue
Continuing.

Breakpoint 1, sched_timer () at kernel/sched.c:…
(gdb) bt                                  # the chain of calls, newest first
#0  sched_timer () at kernel/sched.c:…
#1  … in kerneltrap () at kernel/trap.c:…
#2  … in kernelvec () at kernel/kernelvec.S:…
Backtrace stopped: frame did not save the PC
(gdb) p p
$1 = <optimized out>
(gdb) p cpus[0].proc->name                # the running process
$2 = "schedcheck\000\000\000\000\000"
(gdb) p cpus[0].proc->slice_used
$3 = 1
(gdb) delete                              # remove all breakpoints
(gdb) continue

Three things in this output are normal:

  • <optimized out>. The kernel is compiled with optimization, so gdb cannot show some local variables, such as p here. Use cpus[0].proc instead: with one CPU it is always the running process, or 0 when none runs.
  • Backtrace stopped at kernelvec. That is where the trap entered the kernel. This interrupt came through kerneltrap(), because the job was inside the system call cputicks().
  • ?? () after usertrap(). When the interrupt came from the job's own code, the frame below usertrap() shows as #1 … in ?? (). The trap entered through code that gdb has no name for.

Catching a Panic

Set a breakpoint on panic before you type continue. When the kernel panics, gdb stops there, and bt shows the code that caused it. Here should_preempt() still holds p->lock when it calls pick_rr():

(gdb) break panic
Breakpoint 1 at …: file kernel/printk.c, line ….
(gdb) continue
Continuing.

Breakpoint 1, panic (s=… "acquire") at kernel/printk.c:…
(gdb) bt
#0  panic (s=… "acquire") at kernel/printk.c:…
#1  … in acquire (lk=… <proc+1248>) at kernel/spinlock.c:…
#2  … in pick_by (level=0, before=<optimized out>) at kernel/sched.c:…
#3  pick_rr (level=level@entry=0) at kernel/sched.c:…
#4  … in should_preempt (p=… <proc+1248>) at kernel/sched.c:…
#5  sched_timer () at kernel/sched.c:…
#6  … in kerneltrap () at kernel/trap.c:…
#7  … in kernelvec () at kernel/kernelvec.S:…

Read it from the bottom up. should_preempt(p) called pick_rr(), whose pick_by() locks every process in turn. The lock it tried to take, <proc+1248>, is the lock of the same p, which was already held.

Finding a Hang

Press Ctrl-c in gdb and look at where the CPU is, at ticks, and at the processes (procs, below). Then continue, stop again, and compare. If ticks grows but every job stays SLEEPING and the CPU waits in scheduler(), nothing wakes the jobs: sched_tick() is not called.

Listing the processes

Without gdb, press Ctrl-p in the xv6 console. It prints each process's pid, state and name:

1 sleep  init
2 sleep  sh

For the scheduling fields, save these lines as procs.gdb in the repository folder. In gdb, type source procs.gdb once. After that, procs prints every process in use. Add other fields, such as priority or allot_used, the same way.

# procs: print every process in use, with the scheduling fields.
define procs
  set $i = 0
  while $i < 64
    if proc[$i].state != UNUSED
      printf "%3d %-10s ", proc[$i].pid, proc[$i].name
      output proc[$i].state
      printf "  level=%d enter_tick=%d slice_used=%d wait_ticks=%d\n", proc[$i].level, proc[$i].enter_tick, proc[$i].slice_used, proc[$i].wait_ticks
    end
    set $i = $i + 1
  end
end

Stopped during schedcheck late, it printed:

(gdb) procs
  1 init       SLEEPING  level=0 enter_tick=1 slice_used=0 wait_ticks=0
  2 sh         SLEEPING  level=0 enter_tick=78 slice_used=0 wait_ticks=0
  3 schedcheck SLEEPING  level=0 enter_tick=78 slice_used=0 wait_ticks=0
  4 schedcheck RUNNING  level=0 enter_tick=83 slice_used=2 wait_ticks=0
  5 schedcheck RUNNABLE  level=0 enter_tick=83 slice_used=0 wait_ticks=2
  6 schedcheck SLEEPING  level=0 enter_tick=78 slice_used=0 wait_ticks=0

All the jobs of a test are named schedcheck (mlfqcheck in q4), and their pids follow the order A, B, C. Here pid 3 is the test program itself, so pid 4 is A, 5 is B and 6 is C, which has not arrived yet. The ticks are counted from boot, not from the start of the test.

Commands for This Lab

CommandWhat it does
continue (c)Run until the next breakpoint. Ctrl-c stops xv6 at once.
break f (b f)Stop when function f starts. break f if cond stops only when cond is true, for example break sched_wakeup if p->pid == 6. tbreak f stops once.
info breakpoints, delete [n]List the breakpoints, or remove one or all of them.
btShow the chain of calls. frame n moves to one frame, and up and down move by one.
next (n), step (s), finishRun one line, run one line and enter calls, or run to the end of the function and print what it returns.
p exprPrint a value: p ticks, p proc[3], p *cpus[0].proc, p/x expr in hex.
display exprPrint expr at every stop.
watch -l exprStop when the memory of expr changes, and show the old and new value.
listShow the source around the current line.
  • A breakpoint in a function that runs at every tick, such as sched_tick(), stops at every tick. Add a condition, or delete it when you are done.
  • To stop, type quit in gdb, and xv6 keeps running. Press Ctrl-a and then x in the first terminal to stop QEMU.
  • Run ./grade without gdb. It starts its own QEMU.