Using gdb
gdb stops the kernel at any function, shows the chain of calls that led there, and prints the fields of every process. It helps most with a panic or a hang, where ./grade shows no log. For a wrong log, start with Oops! What's Wrong?.
Two Terminals
You need two terminals in the same container: one for xv6, one for gdb. How to open the second one depends on how you started the container:
| You started the container with | Open a second terminal with |
|---|---|
docker run … --name xv6 … | a new terminal on your computer, then docker exec -it -w /xv6 xv6 bash |
| VS Code, "Reopen in Container" | Terminal → Split Terminal |
./lab shell | tmux inside the container (below) |
tmux splits one terminal into panes. Run tmux, then press Ctrl-b and then % to split the window into a left and a right pane. Ctrl-b and an arrow key moves between panes. Ctrl-b and [ lets you scroll back with the arrow keys, and q stops scrolling. Type exit in a pane to close it.
A First Session
- In the first terminal, run
make qemu-gdb. QEMU starts xv6, stops it before its first instruction, and waits for gdb. - In the second terminal, in the same folder, run
gdb-multiarch, notgdb, althoughmake qemu-gdbsays "run 'gdb'". Plaingdbin the container cannot read the RISC-V kernel.gdb-multiarchreads the file.gdbinitthatmake qemu-gdbwrote, loads the kernel symbols and connects to QEMU. - Type
continuein gdb. xv6 boots in the first terminal. - Run a test in the first terminal, for example
schedcheck late. PressCtrl-cin gdb to stop xv6 at any moment, andcontinueto go on.
Here is part of a session on q0, with addresses and line numbers replaced by …:
(gdb) break sched_timer
Breakpoint 1 at …: file kernel/sched.c, line ….
(gdb) continue
Continuing.
Breakpoint 1, sched_timer () at kernel/sched.c:…
(gdb) bt # the chain of calls, newest first
#0 sched_timer () at kernel/sched.c:…
#1 … in kerneltrap () at kernel/trap.c:…
#2 … in kernelvec () at kernel/kernelvec.S:…
Backtrace stopped: frame did not save the PC
(gdb) p p
$1 = <optimized out>
(gdb) p cpus[0].proc->name # the running process
$2 = "schedcheck\000\000\000\000\000"
(gdb) p cpus[0].proc->slice_used
$3 = 1
(gdb) delete # remove all breakpoints
(gdb) continue
Three things in this output are normal:
<optimized out>. The kernel is compiled with optimization, so gdb cannot show some local variables, such asphere. Usecpus[0].procinstead: with one CPU it is always the running process, or 0 when none runs.Backtrace stoppedatkernelvec. That is where the trap entered the kernel. This interrupt came throughkerneltrap(), because the job was inside the system callcputicks().?? ()afterusertrap(). When the interrupt came from the job's own code, the frame belowusertrap()shows as#1 … in ?? (). The trap entered through code that gdb has no name for.
Catching a Panic
Set a breakpoint on panic before you type continue. When the kernel panics, gdb stops there, and bt shows the code that caused it. Here should_preempt() still holds p->lock when it calls pick_rr():
(gdb) break panic
Breakpoint 1 at …: file kernel/printk.c, line ….
(gdb) continue
Continuing.
Breakpoint 1, panic (s=… "acquire") at kernel/printk.c:…
(gdb) bt
#0 panic (s=… "acquire") at kernel/printk.c:…
#1 … in acquire (lk=… <proc+1248>) at kernel/spinlock.c:…
#2 … in pick_by (level=0, before=<optimized out>) at kernel/sched.c:…
#3 pick_rr (level=level@entry=0) at kernel/sched.c:…
#4 … in should_preempt (p=… <proc+1248>) at kernel/sched.c:…
#5 sched_timer () at kernel/sched.c:…
#6 … in kerneltrap () at kernel/trap.c:…
#7 … in kernelvec () at kernel/kernelvec.S:…
Read it from the bottom up. should_preempt(p) called pick_rr(), whose pick_by() locks every process in turn. The lock it tried to take, <proc+1248>, is the lock of the same p, which was already held.
Finding a Hang
Press Ctrl-c in gdb and look at where the CPU is, at ticks, and at the processes (procs, below). Then continue, stop again, and compare. If ticks grows but every job stays SLEEPING and the CPU waits in scheduler(), nothing wakes the jobs: sched_tick() is not called.
Listing the processes
Without gdb, press Ctrl-p in the xv6 console. It prints each process's pid, state and name:
1 sleep init
2 sleep sh
For the scheduling fields, save these lines as procs.gdb in the repository folder. In gdb, type source procs.gdb once. After that, procs prints every process in use. Add other fields, such as priority or allot_used, the same way.
# procs: print every process in use, with the scheduling fields.
define procs
set $i = 0
while $i < 64
if proc[$i].state != UNUSED
printf "%3d %-10s ", proc[$i].pid, proc[$i].name
output proc[$i].state
printf " level=%d enter_tick=%d slice_used=%d wait_ticks=%d\n", proc[$i].level, proc[$i].enter_tick, proc[$i].slice_used, proc[$i].wait_ticks
end
set $i = $i + 1
end
end
Stopped during schedcheck late, it printed:
(gdb) procs
1 init SLEEPING level=0 enter_tick=1 slice_used=0 wait_ticks=0
2 sh SLEEPING level=0 enter_tick=78 slice_used=0 wait_ticks=0
3 schedcheck SLEEPING level=0 enter_tick=78 slice_used=0 wait_ticks=0
4 schedcheck RUNNING level=0 enter_tick=83 slice_used=2 wait_ticks=0
5 schedcheck RUNNABLE level=0 enter_tick=83 slice_used=0 wait_ticks=2
6 schedcheck SLEEPING level=0 enter_tick=78 slice_used=0 wait_ticks=0
All the jobs of a test are named schedcheck (mlfqcheck in q4), and their pids follow the order A, B, C. Here pid 3 is the test program itself, so pid 4 is A, 5 is B and 6 is C, which has not arrived yet. The ticks are counted from boot, not from the start of the test.
Commands for This Lab
| Command | What it does |
|---|---|
continue (c) | Run until the next breakpoint. Ctrl-c stops xv6 at once. |
break f (b f) | Stop when function f starts. break f if cond stops only when cond is true, for example break sched_wakeup if p->pid == 6. tbreak f stops once. |
info breakpoints, delete [n] | List the breakpoints, or remove one or all of them. |
bt | Show the chain of calls. frame n moves to one frame, and up and down move by one. |
next (n), step (s), finish | Run one line, run one line and enter calls, or run to the end of the function and print what it returns. |
p expr | Print a value: p ticks, p proc[3], p *cpus[0].proc, p/x expr in hex. |
display expr | Print expr at every stop. |
watch -l expr | Stop when the memory of expr changes, and show the old and new value. |
list | Show the source around the current line. |
- A breakpoint in a function that runs at every tick, such as
sched_tick(), stops at every tick. Add a condition, or delete it when you are done. - To stop, type
quitin gdb, and xv6 keeps running. PressCtrl-aand thenxin the first terminal to stop QEMU. - Run
./gradewithout gdb. It starts its own QEMU.