Running a QEMU VM for kernel hacking

Motivation

Among the reasons why to want a virtual machine for doing kernel hacking are

faster development cycles

The time it takes to reboot and test a modified kernel is greatly reduced.

lack of dedicated testing hardware

If you need to do your testing on the same machine you're developing on, you'll quickly realize that this simply isn't practicable:

You'll lose your session each time you do reboot.
Your data is in real danger.
You won't have a serial console where all this kmsg stuff is logged. This can't get underestimated: saving the kernel log into a file allows you to use all those fine tools for analyzing it.
You can't use kgdb since you need to have a remote system for it. The opinions about kgdb's usefulness diverge, but not being able to try it out at all is quite frustrating.

Of course testing on a VM can't be done for drivers of real hardware as long as the device in question can't get passed through from the host to the guest. In theory, this should be possible, but I do not have any experience with it.

Create a root image

Before we start, we'll need a root image in a format that QEMU understands, i.e. qcow2. To create one, simply install your favourite distro into a suitable image using the method that deems easiest, virt-install for example.

Run the VM

Fire up qemu

I do all my kernel development under /mnt/scratch/nic and my qcow2 image created in the previous step is named fedora-rawhide.qcow2.

So let's fire up qemu with that image attached:

#!/bin/sh
qemu-system-x86_64 \
	-no-user-config \
	-no-shutdown \
	-vga std \
	-display gtk \
	-name fedora-rawhide \
	-uuid e4c743c1-b160-4087-8841-1fdfc068ad20 \
	-boot strict=on \
	-realtime mlock=off \
	-machine pc-i440fx-2.3,accel=kvm,usb=off \
	-cpu Haswell-noTSX \
	-m 4096 \
	-smp 2,sockets=2,cores=1,threads=1 \
	-rtc base=utc,driftfix=slew \
	-no-hpet \
	-global kvm-pit.lost_tick_policy=discard \
	-global PIIX4_PM.disable_s3=1 \
	-global PIIX4_PM.disable_s4=1 \
	\
	-drive file=/mnt/scratch/nic/fedora-rawhide.qcow2,if=virtio,id=drive-virtio-disk0,format=qcow2 \
	\
	-chardev stdio,id=chardevmon,mux=on,signal=off \
	-mon chardev=chardevmon \
	\
	-chardev socket,id=chardev0,port=1234,host=localhost,server,nodelay,telnet \
	-device isa-serial,chardev=chardev0,id=serial0 \
	\
	-chardev pty,id=chardev1 \
	-device isa-serial,chardev=chardev1,id=serial1 \
	\
	-fsdev local,security_model=passthrough,id=fsdev-fs0,path=/mnt/scratch/nic \
	-device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=scratch-nic,bus=pci.0,addr=0xa \
	\
	-netdev user,id=netdev0 \
	-device virtio-net-pci,netdev=netdev0,bus=pci.0,addr=0xb \
	\
	-S

Explanation:

General boilerplate stuff in the first block.
Attachment of my qcow2 formatted root filesystem to the VM instance in the second block.
Tell qemu to run a (text based) monitor on its standard IO. The monitor will be used to control the VM instance later on.
The fourth block installs a serial connection between the host and the guest. The interface towards the host emulates a telnet server at port 1234. This serial connection will be used for the guest's console. The reason for chosing a telnet socket should become clear in Magic Sysrq below.
The fifth block installs another serial connection between the host and the guest. The host-side interface is exposed as a pty. This serial connection will be used to attach a gdb running on the host to the guest's kgdb.
The sixth block exports the host's /mnt/scratch/nic directory to the host via the 9P filesystem protocol. This allows us to access that directory from the guest without the tedious procedure of setting up NFS.
The seventh block attaches a very simple network interface to the guest. This basically only allows for outgoing connections but suffices to fetch packages via http etc. If you need something more sophisticated, check the qemu docs for "-netdev".
Finally, the -S flag tells us not to boot the VM instance, but wait for us to type a 'c' at its monitor made accessible in step 3. This gives us time to attach a telnet client to the serial console as well as a gdb, if wanted.

Now, after having started the qemu process, due to the -S flag, your VM will not be instantly started but it will be waiting for you to attach a telnet client to the port proxying to the serial connection:

QEMU waiting for connection on: disconnected:telnet:localhost:1234,server

Connect to it by starting a telnet client on your host. For example,

screen //telnet localhost 1234

telnet localhost 1234

After the telnet connection has been made, it is time to get your VM instance running: type a 'c' and hit enter at the QEMU monitor, i.e. at the prompt presented to you at the stdio of the qemu process.

Because we have attached a VGA display to the guest, a window will appear on your host's desktop, allowing you to interface with the quest.

First time setup

Setting up the serial console within the guest

The first time you boot your VM, you'll need to tell your guest to actually use the serial console attached.

GRUB2
Letting GRUB2 use the serial console is not strictly necessary. However, if you do, it will be more comfortable to cut&paste kernel command line arguments from your host to your guest. Also it avoids some window switching.

Enabling GRUB2 to use a serial console is uniform among distributions and consists of adding
```
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
```
to your /etc/default/grub. Making your changes take effect is distribution specific though and usually consists of running grub2-mkconfig or update-grub2. Google is your friend here ;).
Linux guest
For Linux, the first serial interface attached to the quest by qemu will be known as "ttyS0". To make full use of it, you'll need to
- tell the kernel to use it as its console to log messages to
- as well as to tell yout init to run a getty on it in order to be able to login through the serial connection.
- Console
  
  To make your guest's kernel use the serial interface as its console, you need to add a console parameter to its command line. Again, this is done by modifying /etc/default/grub: the parameter GRUB_CMDLINE_LINUX must be changed to include the argument console=ttyS0,115200n8. Also, make sure to remove any "quiet" parameter. After having made this change, make them take effect method suitable for your distribution, i.e. grub2-mkconfig.
- getty
  Getting a getty listening on your serial interface is highly distribution specific. It might involve editing /etc/inittab or whatever. However, on my Fedora Rawhide guest, it suffices to tell systemd to do the right thing by creating a symlink:
```
ln -s /usr/lib/systemd/system/getty@.service /etc/systemd/system/getty.target.wants/getty@tty1.service
```
  On other systemd based distribution, it might very well be as easy as this, too.
Reboot

Now that you've put everything in place in your guest's rootfs, reboot and cross fingers. Your guest should come up with GRUB2 displaying its boot menu within your telnet client attached to the host side of the serial connection now. The Linux kernel should display its messages there as well and eventually, you should get a login promt.

Using the VM

Now that we've configured the guest to make use of the serial connection, let's see how we can interact with it from the host.

QEMU monitor

The QEMU monitor allows you to control the VM instance's virtual hardware by commands like power_down, system_reset, etc.

Magic Sysrq

Magic Sysrqs are special key combinations intercepted and specially interpreted by the kernel. Upon encountering such a Sysrq, the kernel carries out a defined action such as syncing filesystems, trapping into kgdb or whatever.

Sysrqs need to be enabled explicitly in your guest's kernel:

CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1

Sending a Sysrq through a serial connection is encoded as sending an out-of-band signal called "BREAK", followed by the character assigned to the desired action.

Fortunately, the telnet protocol supports encoding a BREAK by means of escape sequences and qemu nicely translates it into a "real" BREAK on the virtual serial connection.

Actually, this support for BREAK is the reason why we told qemu to present the first serial connection as a telnet socket to the host. PTY's for example, do not support encoding BREAK~s and we had no way of transferring one to ~qemu.

Now, how to send a BREAK over the telnet connection depends on your telnet client. With GNU Screen, it is C-a C-b. Thus, triggring a filesystem sync on the guest would be initiated by C-a C-b s, for example.

kgdb

Connecting a gdb running on the host to a kgdb interface in the guest requires several things. Detailed usage information can be found in Documentation/DocBook/kgdb.tmpl.

First of all, support in the guest's kernel is required:

CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y

Enable the kgdb serial interface in the guest's kernel
You'll need to tell your guest kernel to actually expose the kgdb interface through its second serial interface by editing its command line from the GRUB2 menu:
```
kgdboc=ttyS1,115200 kgdbwait
```
The kgdbwait flag tells the kernel to just wait for a connection from your host's gdb at an early stage.
Connecting gdb through the serial interface to kgdb
Right after you've connected your telnet client to the port offered by qemu, qemu will print on its standard output which PTY has been allocated on the host for the second serial connection:
```
char device redirected to /dev/pts/3 (label chardev1)
```
Remember that device and start up gdb with the Linux kernel binary as the image to debug and tell it to debug a remote target accessible through this PTY:
```
 gdb ./vmlinux
(gdb) set remotebaud 115200
(gdb) target remote /dev/pts/3
```
After having connected, set breakpoints or whatever you want to do at this early stage and make the kernel continue normal operation by typing a 'c' for *c*ontinue at the gdb prompt.

If you ever want to make it trap into your gdb, send it a 'g'-Sysrq.

Access to a directory on the host via the 9p filesystem

Remember that we told qemu to expose a directory on the host via 9p to the guest. To actually mount it therein, the guest kernel must have support for it:

CONFIG_NET_9P=m
CONFIG_NET_9P_VIRTIO=m
CONFIG_9P_FS=m

Now, here's how to mount it from the guest:

mount -t 9p -o trans=virtio,ro scratch-nic /mnt/wherever

where "scratch-nic" is the so called mount_tag assigned to the 9p export on the qemu command line.

Now, have fun compiling your kernels and modules on your host and installing them directly from within your guest.