Running a QEMU VM for kernel hacking
Among the reasons why to want a virtual machine for doing kernel hacking are
- faster development cycles
- The time it takes to reboot and test a modified kernel is greatly reduced.
- lack of dedicated testing hardware
- If you need to do your
testing on the same machine you're developing on, you'll
quickly realize that this simply isn't practicable:
- You'll lose your session each time you do reboot.
- Your data is in real danger.
- You won't have a serial console where all this kmsg stuff is logged. This can't get underestimated: saving the kernel log into a file allows you to use all those fine tools for analyzing it.
- You can't use kgdb since you need to have a remote system for it. The opinions about kgdb's usefulness diverge, but not being able to try it out at all is quite frustrating.
Of course testing on a VM can't be done for drivers of real hardware as long as the device in question can't get passed through from the host to the guest. In theory, this should be possible, but I do not have any experience with it.
Create a root image
Before we start, we'll need a root image in a format that QEMU
qcow2. To create one, simply install your
favourite distro into a suitable image using the method that deems
virt-install for example.
Run the VM
Fire up qemu
I do all my kernel development under
/mnt/scratch/nic and my
image created in the previous step is named
So let's fire up
qemu with that image attached:
#!/bin/sh qemu-system-x86_64 \ -no-user-config \ -no-shutdown \ -vga std \ -display gtk \ -name fedora-rawhide \ -uuid e4c743c1-b160-4087-8841-1fdfc068ad20 \ -boot strict=on \ -realtime mlock=off \ -machine pc-i440fx-2.3,accel=kvm,usb=off \ -cpu Haswell-noTSX \ -m 4096 \ -smp 2,sockets=2,cores=1,threads=1 \ -rtc base=utc,driftfix=slew \ -no-hpet \ -global kvm-pit.lost_tick_policy=discard \ -global PIIX4_PM.disable_s3=1 \ -global PIIX4_PM.disable_s4=1 \ \ -drive file=/mnt/scratch/nic/fedora-rawhide.qcow2,if=virtio,id=drive-virtio-disk0,format=qcow2 \ \ -chardev stdio,id=chardevmon,mux=on,signal=off \ -mon chardev=chardevmon \ \ -chardev socket,id=chardev0,port=1234,host=localhost,server,nodelay,telnet \ -device isa-serial,chardev=chardev0,id=serial0 \ \ -chardev pty,id=chardev1 \ -device isa-serial,chardev=chardev1,id=serial1 \ \ -fsdev local,security_model=passthrough,id=fsdev-fs0,path=/mnt/scratch/nic \ -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=scratch-nic,bus=pci.0,addr=0xa \ \ -netdev user,id=netdev0 \ -device virtio-net-pci,netdev=netdev0,bus=pci.0,addr=0xb \ \ -S
- General boilerplate stuff in the first block.
- Attachment of my
qcow2formatted root filesystem to the VM instance in the second block.
qemuto run a (text based) monitor on its standard IO. The monitor will be used to control the VM instance later on.
- The fourth block installs a serial connection between the host
and the guest. The interface towards the host emulates a telnet
server at port
1234. This serial connection will be used for the guest's console. The reason for chosing a telnet socket should become clear in Magic Sysrq below.
- The fifth block installs another serial connection between the
host and the guest. The host-side interface is exposed as a pty.
This serial connection will be used to attach a
gdbrunning on the host to the guest's
- The sixth block exports the host's
/mnt/scratch/nicdirectory to the host via the 9P filesystem protocol. This allows us to access that directory from the guest without the tedious procedure of setting up NFS.
- The seventh block attaches a very simple network interface to
the guest. This basically only allows for outgoing connections
but suffices to fetch packages via http etc. If you need
something more sophisticated, check the
qemudocs for "
- Finally, the
-Sflag tells us not to boot the VM instance, but wait for us to type a '
c' at its monitor made accessible in step 3. This gives us time to attach a telnet client to the serial console as well as a gdb, if wanted.
Now, after having started the
qemu process, due to the
your VM will not be instantly started but it will be waiting for you
to attach a telnet client to the port proxying to the serial
QEMU waiting for connection on: disconnected:telnet:localhost:1234,server
Connect to it by starting a telnet client on your host. For example,
screen //telnet localhost 1234
telnet localhost 1234
After the telnet connection has been made, it is time to get your VM
instance running: type a '
c' and hit
enter at the QEMU monitor,
i.e. at the prompt presented to you at the stdio of the
Because we have attached a VGA display to the guest, a window will appear on your host's desktop, allowing you to interface with the quest.
First time setup
Setting up the serial console within the guest
The first time you boot your VM, you'll need to tell your guest to actually use the serial console attached.
Letting GRUB2 use the serial console is not strictly necessary. However, if you do, it will be more comfortable to cut&paste kernel command line arguments from your host to your guest. Also it avoids some window switching.
Enabling GRUB2 to use a serial console is uniform among distributions and consists of adding
GRUB_TERMINAL=serial GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
/etc/default/grub. Making your changes take effect is distribution specific though and usually consists of running
update-grub2. Google is your friend here ;).
- Linux guest
For Linux, the first serial interface attached to the quest by
qemuwill be known as "
ttyS0". To make full use of it, you'll need to
- tell the kernel to use it as its console to log messages to
- as well as to tell yout
initto run a
gettyon it in order to be able to login through the serial connection.
To make your guest's kernel use the serial interface as its console, you need to add a
consoleparameter to its command line. Again, this is done by modifying
/etc/default/grub: the parameter
GRUB_CMDLINE_LINUXmust be changed to include the argument
console=ttyS0,115200n8. Also, make sure to remove any "
quiet" parameter. After having made this change, make them take effect method suitable for your distribution, i.e.
gettylistening on your serial interface is highly distribution specific. It might involve editing
/etc/inittabor whatever. However, on my Fedora Rawhide guest, it suffices to tell
systemdto do the right thing by creating a symlink:
ln -s /usr/lib/systemd/system/getty@.service /firstname.lastname@example.org
systemdbased distribution, it might very well be as easy as this, too.
Now that you've put everything in place in your guest's rootfs, reboot and cross fingers. Your guest should come up with GRUB2 displaying its boot menu within your telnet client attached to the host side of the serial connection now. The Linux kernel should display its messages there as well and eventually, you should get a login promt.
Using the VM
Now that we've configured the guest to make use of the serial connection, let's see how we can interact with it from the host.
The QEMU monitor allows you to control the VM instance's virtual
hardware by commands like
Magic Sysrqs are special key combinations intercepted and specially interpreted by the kernel. Upon encountering such a Sysrq, the kernel carries out a defined action such as syncing filesystems, trapping into kgdb or whatever.
Sysrqs need to be enabled explicitly in your guest's kernel:
Sending a Sysrq through a serial connection is encoded as sending
an out-of-band signal called "
BREAK", followed by the character
assigned to the desired action.
Fortunately, the telnet protocol supports encoding a
means of escape sequences and
qemu nicely translates it into a
BREAK on the virtual serial connection.
Actually, this support for
BREAK is the reason why we told
present the first serial connection as a telnet socket to the
host. PTY's for example, do not support encoding
BREAK~s and we
had no way of transferring one to ~qemu.
Now, how to send a
BREAK over the telnet connection depends on
your telnet client. With GNU Screen, it is
C-a C-b. Thus,
triggring a filesystem sync on the guest would be initiated by
C-a C-b s, for example.
gdb running on the host to a
kgdb interface in
the guest requires several things. Detailed usage information can
be found in
First of all, support in the guest's kernel is required:
- Enable the kgdb serial interface in the guest's kernel
You'll need to tell your guest kernel to actually expose the kgdb interface through its second serial interface by editing its command line from the GRUB2 menu:
kgdbwaitflag tells the kernel to just wait for a connection from your host's
gdbat an early stage.
gdbthrough the serial interface to
Right after you've connected your telnet client to the port offered by
qemuwill print on its standard output which PTY has been allocated on the host for the second serial connection:
char device redirected to /dev/pts/3 (label chardev1)
Remember that device and start up
gdbwith the Linux kernel binary as the image to debug and tell it to debug a remote target accessible through this PTY:
gdb ./vmlinux (gdb) set remotebaud 115200 (gdb) target remote /dev/pts/3
After having connected, set breakpoints or whatever you want to do at this early stage and make the kernel continue normal operation by typing a '
c' for *c*ontinue at the
If you ever want to make it trap into your
gdb, send it a '
Access to a directory on the host via the 9p filesystem
CONFIG_NET_9P=m CONFIG_NET_9P_VIRTIO=m CONFIG_9P_FS=m
Now, here's how to mount it from the guest:
mount -t 9p -o trans=virtio,ro scratch-nic /mnt/wherever
scratch-nic" is the so called
mount_tag assigned to the
9p export on the
qemu command line.
Now, have fun compiling your kernels and modules on your host and installing them directly from within your guest.