HOWTO debug a kernel with virt-manager

I’ve got a bunch of virtual machines that I create and manage with virt-manager. I do Linux kernel work. Once in awhile a stubborn bug creeps in that defies all attempts to find it by reading code or using printk. When that happens, QEMU’s gdb support is very useful. Here is how to get started.

Build a kernel with CONFIG_KGDB

There are several related options that are worth looking into as well. See this page for more information. Since I am typically debugging a modified RHEL kernel, I just use their kernel-debug package, which includes all of the necessary configuration.

Put vmlinux onto the box used for debugging

After building the kernel RPM, the vmlinux file with all of the necessary debug information already in it can be found at a path like rpmbuild/BUILD/kernel-2.6.32-358.18.1.el6/linux-2.6.32-358.18.1.el6.x86_64/vmlinux. Put that file somewhere on the machine where the debugger will be executed.

Use virsh to prepare to debug

Run virsh and edit the domain for the target virtual machine. Change the first line from:

<domain type='kvm'>

to:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

Then in the body of the domain tag, add these lines:

<qemu:commandline>
  <qemu:arg value='-s'/>
  <qemu:arg value='-S'/>
</qemu:commandline>

Launch the virtual machine

Use virt-manager to launch the virtual machine to be debugged. When the boot menu comes up, choose the debug kernel.

Launch gdb

On the machine used for debugging, run gdb vmlinux, then enter “target remote localhost:1234”. You may also want to execute the directory command to tell gdb where the source files are located.

Debug a kernel module

If the bug you are chasing is in a kernel module, we need to do a little more work. Assuming you can load the module without killing the kernel (!), do the following in the guest:

  1. modprobe _module_
  2. cd /sys/module/_module_/sections
  3. cat .text
  4. cat .bss
  5. cat .data
  6. cat .rodata

Now on the host, break into the debugger and run this command:

add-symbol-file _path/to/module.o text_address_
-s .bss _bss_address_ -s .data _data_address_
-s .rodata _rodata_address_

where the addresses are those printed in steps 3, 4, 5, and 6 above, respectively.

This script is designed to be an aid in constructing the add-symbol-file command. Run it in the guest after loading the module, then save its output on the host somewhere for later use in gdb. The script assumes that debug information for the guest module is stored on the host in /usr/lib/debug. It should work without modification on RHEL, CentOS, and Fedora systems, but may require some tweaking for other distributions.