Running a QEMU VM for kernel hacking
Motivation
Among the reasons why to want a virtual machine for doing kernel hacking are
- faster development cycles
- The time it takes to reboot and test a modified kernel is greatly reduced.
- lack of dedicated testing hardware
- If you need to do your
testing on the same machine you're developing on, you'll
quickly realize that this simply isn't practicable:
- You'll lose your session each time you do reboot.
- Your data is in real danger.
- You won't have a serial console where all this kmsg stuff is logged. This can't get underestimated: saving the kernel log into a file allows you to use all those fine tools for analyzing it.
- You can't use kgdb since you need to have a remote system for it. The opinions about kgdb's usefulness diverge, but not being able to try it out at all is quite frustrating.
Of course testing on a VM can't be done for drivers of real hardware as long as the device in question can't get passed through from the host to the guest. In theory, this should be possible, but I do not have any experience with it.
Create a root image
Before we start, we'll need a root image in a format that QEMU
understands, i.e. qcow2
. To create one, simply install your
favourite distro into a suitable image using the method that deems
easiest, virt-install
for example.
Run the VM
Fire up qemu
I do all my kernel development under /mnt/scratch/nic
and my qcow2
image created in the previous step is named fedora-rawhide.qcow2
.
So let's fire up qemu
with that image attached:
#!/bin/sh qemu-system-x86_64 \ -no-user-config \ -no-shutdown \ -vga std \ -display gtk \ -name fedora-rawhide \ -uuid e4c743c1-b160-4087-8841-1fdfc068ad20 \ -boot strict=on \ -realtime mlock=off \ -machine pc-i440fx-2.3,accel=kvm,usb=off \ -cpu Haswell-noTSX \ -m 4096 \ -smp 2,sockets=2,cores=1,threads=1 \ -rtc base=utc,driftfix=slew \ -no-hpet \ -global kvm-pit.lost_tick_policy=discard \ -global PIIX4_PM.disable_s3=1 \ -global PIIX4_PM.disable_s4=1 \ \ -drive file=/mnt/scratch/nic/fedora-rawhide.qcow2,if=virtio,id=drive-virtio-disk0,format=qcow2 \ \ -chardev stdio,id=chardevmon,mux=on,signal=off \ -mon chardev=chardevmon \ \ -chardev socket,id=chardev0,port=1234,host=localhost,server,nodelay,telnet \ -device isa-serial,chardev=chardev0,id=serial0 \ \ -chardev pty,id=chardev1 \ -device isa-serial,chardev=chardev1,id=serial1 \ \ -fsdev local,security_model=passthrough,id=fsdev-fs0,path=/mnt/scratch/nic \ -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=scratch-nic,bus=pci.0,addr=0xa \ \ -netdev user,id=netdev0 \ -device virtio-net-pci,netdev=netdev0,bus=pci.0,addr=0xb \ \ -S
Explanation:
- General boilerplate stuff in the first block.
- Attachment of my
qcow2
formatted root filesystem to the VM instance in the second block. - Tell
qemu
to run a (text based) monitor on its standard IO. The monitor will be used to control the VM instance later on. - The fourth block installs a serial connection between the host
and the guest. The interface towards the host emulates a telnet
server at port
1234
. This serial connection will be used for the guest's console. The reason for chosing a telnet socket should become clear in Magic Sysrq below. - The fifth block installs another serial connection between the
host and the guest. The host-side interface is exposed as a pty.
This serial connection will be used to attach a
gdb
running on the host to the guest'skgdb
. - The sixth block exports the host's
/mnt/scratch/nic
directory to the host via the 9P filesystem protocol. This allows us to access that directory from the guest without the tedious procedure of setting up NFS. - The seventh block attaches a very simple network interface to
the guest. This basically only allows for outgoing connections
but suffices to fetch packages via http etc. If you need
something more sophisticated, check the
qemu
docs for "-netdev
". - Finally, the
-S
flag tells us not to boot the VM instance, but wait for us to type a 'c
' at its monitor made accessible in step 3. This gives us time to attach a telnet client to the serial console as well as a gdb, if wanted.
Now, after having started the qemu
process, due to the -S
flag,
your VM will not be instantly started but it will be waiting for you
to attach a telnet client to the port proxying to the serial
connection:
QEMU waiting for connection on: disconnected:telnet:localhost:1234,server
Connect to it by starting a telnet client on your host. For example,
screen //telnet localhost 1234
or
telnet localhost 1234
After the telnet connection has been made, it is time to get your VM
instance running: type a 'c
' and hit enter
at the QEMU monitor,
i.e. at the prompt presented to you at the stdio of the qemu
process.
Because we have attached a VGA display to the guest, a window will appear on your host's desktop, allowing you to interface with the quest.
First time setup
Setting up the serial console within the guest
The first time you boot your VM, you'll need to tell your guest to actually use the serial console attached.
- GRUB2
Letting GRUB2 use the serial console is not strictly necessary. However, if you do, it will be more comfortable to cut&paste kernel command line arguments from your host to your guest. Also it avoids some window switching.
Enabling GRUB2 to use a serial console is uniform among distributions and consists of adding
GRUB_TERMINAL=serial GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
to your
/etc/default/grub
. Making your changes take effect is distribution specific though and usually consists of runninggrub2-mkconfig
orupdate-grub2
. Google is your friend here ;). - Linux guest
For Linux, the first serial interface attached to the quest by
qemu
will be known as "ttyS0
". To make full use of it, you'll need to- tell the kernel to use it as its console to log messages to
- as well as to tell yout
init
to run agetty
on it in order to be able to login through the serial connection.
- Console
To make your guest's kernel use the serial interface as its console, you need to add a
console
parameter to its command line. Again, this is done by modifying/etc/default/grub
: the parameterGRUB_CMDLINE_LINUX
must be changed to include the argumentconsole=ttyS0,115200n8
. Also, make sure to remove any "quiet
" parameter. After having made this change, make them take effect method suitable for your distribution, i.e.grub2-mkconfig
. getty
Getting a
getty
listening on your serial interface is highly distribution specific. It might involve editing/etc/inittab
or whatever. However, on my Fedora Rawhide guest, it suffices to tellsystemd
to do the right thing by creating a symlink:ln -s /usr/lib/systemd/system/getty@.service /etc/systemd/system/getty.target.wants/getty@tty1.service
On other
systemd
based distribution, it might very well be as easy as this, too.
- Reboot
Now that you've put everything in place in your guest's rootfs, reboot and cross fingers. Your guest should come up with GRUB2 displaying its boot menu within your telnet client attached to the host side of the serial connection now. The Linux kernel should display its messages there as well and eventually, you should get a login promt.
Using the VM
Now that we've configured the guest to make use of the serial connection, let's see how we can interact with it from the host.
QEMU monitor
The QEMU monitor allows you to control the VM instance's virtual
hardware by commands like power_down
, system_reset
, etc.
Magic Sysrq
Magic Sysrqs are special key combinations intercepted and specially interpreted by the kernel. Upon encountering such a Sysrq, the kernel carries out a defined action such as syncing filesystems, trapping into kgdb or whatever.
Sysrqs need to be enabled explicitly in your guest's kernel:
CONFIG_MAGIC_SYSRQ=y CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
Sending a Sysrq through a serial connection is encoded as sending
an out-of-band signal called "BREAK
", followed by the character
assigned to the desired action.
Fortunately, the telnet protocol supports encoding a BREAK
by
means of escape sequences and qemu
nicely translates it into a
"real" BREAK
on the virtual serial connection.
Actually, this support for BREAK
is the reason why we told qemu
to
present the first serial connection as a telnet socket to the
host. PTY's for example, do not support encoding BREAK~s and we
had no way of transferring one to ~qemu
.
Now, how to send a BREAK
over the telnet connection depends on
your telnet client. With GNU Screen, it is C-a C-b
. Thus,
triggring a filesystem sync on the guest would be initiated by
C-a C-b s
, for example.
kgdb
Connecting a gdb
running on the host to a kgdb
interface in
the guest requires several things. Detailed usage information can
be found in Documentation/DocBook/kgdb.tmpl
.
First of all, support in the guest's kernel is required:
CONFIG_KGDB=y CONFIG_KGDB_SERIAL_CONSOLE=y
- Enable the kgdb serial interface in the guest's kernel
You'll need to tell your guest kernel to actually expose the kgdb interface through its second serial interface by editing its command line from the GRUB2 menu:
kgdboc=ttyS1,115200 kgdbwait
The
kgdbwait
flag tells the kernel to just wait for a connection from your host'sgdb
at an early stage. - Connecting
gdb
through the serial interface tokgdb
Right after you've connected your telnet client to the port offered by
qemu
,qemu
will print on its standard output which PTY has been allocated on the host for the second serial connection:char device redirected to /dev/pts/3 (label chardev1)
Remember that device and start up
gdb
with the Linux kernel binary as the image to debug and tell it to debug a remote target accessible through this PTY:gdb ./vmlinux (gdb) set remotebaud 115200 (gdb) target remote /dev/pts/3
After having connected, set breakpoints or whatever you want to do at this early stage and make the kernel continue normal operation by typing a '
c
' for *c*ontinue at thegdb
prompt.If you ever want to make it trap into your
gdb
, send it a 'g
'-Sysrq.
Access to a directory on the host via the 9p filesystem
Remember that we told qemu
to expose a directory on the host via
9p to the guest. To actually mount it therein, the guest kernel
must have support for it:
CONFIG_NET_9P=m CONFIG_NET_9P_VIRTIO=m CONFIG_9P_FS=m
Now, here's how to mount it from the guest:
mount -t 9p -o trans=virtio,ro scratch-nic /mnt/wherever
where "scratch-nic
" is the so called mount_tag
assigned to the
9p export on the qemu
command line.
Now, have fun compiling your kernels and modules on your host and installing them directly from within your guest.