Quasi una fantasia (Mac's Blog)

Intro: In this post, I’m going to document something particularly arduous (for me) that I accomplished over the last few days.

Goal: Create an ArchLinux guest image that can be booted in qemu-kvm using a custom kernel specified to qemu-kvm with the -kernel option, and which doesn’t use an initrd. (Such a host can be used for testing kernel work.)

Part 1: Installing the guest. This wasn’t too bad; there are instructions out there. The problem I ran into was trying to do it without using a GUI, since I was actually doing the install on a remote machine I was connected to using SSH. Maybe there’s a way to get around the problem I had using X forwarding and/or VNC, but I used the –curses option on the qemu installation command line. (I tried the same thing with a Debian image and could not get it to work, though.)

Part 2: Compiling the kernel. In the “normal” boot process, the kernel starts; passes control to an initrd, which loads the filesystem drivers and then passes control back to the kernel; and /sbin/init is started, which sets everything up for the OS and spawns a login shell. We want to skip having an initrd. So, we need to compile our filesystem drivers directly into the kernel (i.e. not as modules). I used ext4 when I installed Arch Linux, so I had to add that to my kernel config. Everything else I needed was already in my config. I had a config designed to work with x86-64 qemu-kvm “hardware,” which I got from a colleague. Without that, the process of getting a working kernel config may have been much worse.

Part 3: Booting the guest. I used a command like the following:

qemu-system-x86_64 -smp $NR_CPU -cpu core2duo -hda $IMAGE -m 2000 -net nic,model=e1000 -net user -k en-us -kernel $VMLINUZ -append "console=ttyS0 root=/dev/hda3" ro -nographic -redir tcp:3333::22

First, note the root=/dev/hda3. For some reason, my root partition always becomes /dev/hda3 instead of /dev/hda1. You may have to play around with this. It’s OK to guess different numbers until you get it to work.

The -nographic is what lets this work over an SSH terminal; the console=ttyS0 tells the kernel to send output to serial console 0, which is what qemu interfaces with in -nographic mode. If you run this command without modifying the guest OS at all (just doing a basic install), the guest will appear to hang right when a login: prompt should be presented. It turns out, that’s because init has to be told to present an instance of agetty on ttyS0 (normally it’s not presented there). (NOTE: the following instructions may be slightly off for other distros.) To make this happen, you need to modify /etc/inittab in the guest (should be fairly obvious how to do so). You will now get a login prompt, but you will not be able to log in as root. To allow root to login from ttyS0, you need to modify /etc/securetty in the guest (again, pretty self-explanatory once you look at that file).

By the way, the -redir tcp makes the host port 3333 tunnel to the guest port 22, so if you run an SSH server on the guest at port 22, you will be able to connect to it from the host if you ssh to localhost port 3333.

Additional tweaks: I found that my Arch Linux guest was not sending printk messages to the console (as I wanted), or at least, not the ones I was looking for. You can configure what “level” of printk message get sent to the console by writing to /proc/sys/kernel/printk or by putting kernel.printk = <# # # #> in /etc/sysctrl.conf. I got it right by using 7 4 1 7.

Special thanks: #archlinux IRC channel, and particularly the user falconindy, who gave good advice and kept a good sense of humor despite my obvious frustration.