Why all this ?
Today, nearly every modern computing-related company uses directly or indirectly a virtualization software. As seen with the recent Cloud computing trend, it is a sector that recently exploded in popularity and it is now accessible to everyone.
There are now a LOT of alternatives to virtualize an OS :
- Isolation : OpenVZ, VServer, LXC, UML, BSD jails, Solaris zones…
- Paravirtualization : Xen
- Full virtualization : VMWare Player/Workstation/ESX/ESXi, KVM/Qemu, Virtualbox, Bochs…
When having to deal with a large pack of machines for building packages, testing software on multiple OSes or optimally use a powerful server with lots of RAM, you rapidly should, if not have to, realize that Virtual Machines can make your life significantly easier. It was the case for us.
Here at Normation, we love free software (as in free speech, but a free beer is fine too, thanks), so we decided to use something that was FOSS, multi OS compatible (ruling out all isolation softwares), and convenient to use. We chose to use the libvirt/KVM and LVM mix. (ESX/VirtualBox users, do not leave yet. You will be much pleased to learn that libvirt is compatible with your software too ! You will loose the graphical virt-manager, but virsh might come in handy for you if you have scripting needs by example)
Overview
First, let’s deepen a bit the details on these two softs :
- KVM : KVM is a port of the Qemu fullvirt software to the standardized Linux virtualization layer. It consists of two parts : the qemu/kvm binary and the KVM kernel module that handles every CPU/Memory/device operation depending of the CPU support.
- libvirt : libvirt is a RedHat sponsored project that tries to be a “meta” hypervisor, offering a standardized set of instructions to manage VMs independently of the underlying virtualization software. It currently supports Xen, QEmu, KVM, LXC, OpenVZ, VirtualBox and VMware ESX.
- LVM : The Linux Logical Volume Manager, which enables a much more flexible storage volume management compared to regular MBR partitions, and permits to create “pools” of volumes if needed.
Now let’s see how to create a powerful hypervisor in no time !
Installation of the solution
Shopping list
To install this solution, you will need the following prerequisites :
- A debian distribution, preferably squeeze (or wheezy/testing if you want the latest features) has been used in this post. However, any Linux distribution that bundles libvirt and KVM will do !
- A free partition on the disk, with lots of space (As much as what your VMs will eat, 40GB is a bare minimum)
- A PC that supports the hardware virtualization CPU instructions (AMD SVM or Intel VT-X, this isn’t mandatory but you WILL have performances issues if you do not have it), a minimum of 2GB of RAM, and a fast disk drive. (You might want to have a hardware RAID on a production server, or anything having decent I/O performances)
- A basic comprehension of how GNU/Linux and a PC works
Installation of the packages
You only have to fire up this one-liner, and you will have every tool at hand:
- Debian users:
[code lang=”shell”]
aptitude update
aptitude install libvirt-bin kvm qemu lvm
[/code]
- RedHat users:
[code lang=”shell”]
yum install libvirt qemu-kvm lvm2
[/code]
- SuSE/OpenSuSE users:
[code lang=”shell”]
zypper refresh
zypper install libvirt kvm qemu lvm2
[/code]
I assume that your hypervisor is also your client (the one that will manage your VM), and that you have a X server/a Desktop environment already installed. If not, or if you just want to use the CLI tools, do not try to install this on the server. Note that you can also install this on a remote machine and access the hypervisor using an SSH tunnel (The applications creates it automatically if needed. There are other ways but I will let you the pleasure to read the manual if you are interested)
[code lang=”shell”]
aptitude install virt-manager
# or
yum install virt-manager
# or
zypper install virt-manager
[/code]
LVM pool creation
This part is optional, you can also use plain files to store the VMs data, but please bear in mind that I personally had some real performance issues with some hardware/filesystem combinations, which were resolved using this. Also, we seek to create a production-ready environment so you really should do it.
Remember when we talked about a big free partition on the drive ? Now let’s use it for real. Here I assume it is /dev/sda4, but your mileage may vary.
[code lang=”shell”]
fdisk /dev/sda
# Type t, 4, and 8e to switch the partition type to Linux LVM. I will help the kernel and LVM tools to locate the partition and use it.
pvcreate /dev/sda4
vgcreate (vgname) /dev/sda4 # Change to a meaningful name for you
[/code]
Now we have a (vgname) LVM Volume Group ready to store all our VMs.
Virt manager usage
At this point, the hypervisor machine is ready. Now we just need to set up in libvirt the storage pools for the VMs and the network parameters. You can use virt manager or virsh (the libvirt shell) to do this.
First, you need to add whichever user who will have access to the system to the libvirt system group, to grant them access to the libvirt control socket.
[code lang=”shell”]
for i in user1 user2 user3
do addgroup $i libvirt
done
[/code]
Virsh uses XML templates to do that, like this one for the network:
[code lang=”xml”]
default
65f40d0d-219a-2dd9-a6ac-3a612c26e36b
[/code]
Please note that there is no clean and user friendly wizard to do this kind of operation using virsh, you either must have a template at hand and edit it or use the virt manager GUI which is way more user friendly.
Virt Manager is a Graphical User Interface to the libvirt software. It enables the user to get back his clean interface to see which VMs are launched, edit them or create them at will, much like Virtualbox or VMWare Player.
Start it from your desktop menu, and it will at first try to connect to a local hypervisor. If you have one, bingo, no more steps to connect are required, you are in, however, if your hypervisor is a remote machine, you have to click File -> Add a connection and use the QEMU/KVM hypervisor with a SSH remote connection method.
Now that you are connected, right click the name of your connection (like “localhost (QEMU)”) and choose “Details”: This is where you can tune some of libvirt’s access to the machine.
Network parameters
Here you can tune how libvirt will set up connections to the outside world or create internal networks. If a simple NAT is all you need, nothing more to do here than enable the “default” network to start on boot. You might want to create an internal network or a second outside one, if so click the “plus” sign below the network names and follow the guide.
Storage parameters
Here you will have to create storage pools to store your VMs. A default pool has already be created for those who want to use plain files, and stores them in /var/lib/libvirt/images. You will probably need two more pools : A LVM one and somewhere to store your OS installation ISOs if you use a remote hypervisor.
Here the same routine applies: Click the “plus” sign and follow the guide, and do not forget to select an appropriate pool type.
As of my installation, I like having my ISOs in /home/ISO, so if you want to do this, here is the procedure:
[code lang=”shell”]
mkdir -p /home/ISO
chown -R libvirt-qemu:libvirt /home/ISO
[/code]
and add this directory as a pool. Please note that nothing forbids you to use the VMs Volume Group to create a logical volume to store your ISOs and mount it on /home/ISO !
[code lang=”shell”]
lvcreate –name ISO –size 16G (VGName)
mke2fs -j /dev/(VGName)/ISO
# And append to your fstab the following entry :
/dev/(VGName)/ISO /home/ISO ext3 defaults 0 2
[/code]
Now we have everything set up and ready for use !
Everyday usage
Useful procedures
You will likely create VMs using virt manager’s interface, and there is a quick and easy wizard to do it: you just have to click the first icon in the main interface.
To backup a VM, you have two options :
- Use a dedicated software inside the VM, like backup-manager or Bacula/Amanda and store the backups using FTP/NFS/…
- Backup the VM raw data directly
The first solution is much more economic in terms of used space, but if you have lots of backup-dedicated space you can use the second solution, which offers a easier and quicker recovery alternative. Remember that you can not backup a VM raw data while it is running of you WILL have serious data corruption in your backup. You can use dd for that, or virt-clone.
Note that if you chose to store your VMs using plain QCOW2 or RAW images, it is easier: a simple copy and it is all done.
Which brings me to the LVM method first advantage, after the performance : snapshots !
LVM magic
You can use snapshots to reduce your disk memory usage, using a single base image for several VMs, or have a minimal downtime during backups : Stop the VM, snapshot, immediately backup and restart it.
Simply put, here is the magic command to create a snapshot :
[code lang=”shell”]
lvcreate –snapshot /dev/(VGName)/(VMName) –name (SnapshotName) –size (VM Logival Volume Size)
# After you have backuped the snapshot (if you do not consider the snapshot to be a backup itself), delete it.
lvremove /dev/(VGName)/(SnapshotName)
[/code]
Further than that, LVM enabled us to make a simple failover configuration and get a quick and efficient fault tolerance, using DRBD, every VM is replicated over the network in a raid1-like fashion between two hypervisors and in case the primary machine went to fail, we only need to switch DRBD from secondary to primary on the secondary host and reboot the VM’s and here, incident bypassed !
Usage tips
Here is a list of things that may prove useful to you then :
- IMPORTANT: If you boot a Linux VM, you might want to add “elevator=noop” to your Linux boot command line to force the disk scheduler to let the host machine handle the disk writes reorganisations (like tunnelling tcp over tcp, it is bad to have two schedulers trying to do each other’s job). For example, on GRUB2 on Debian, you have to append “elevator=noop” after GRUB_CMDLINE_LINUX in /etc/default/grub.
- IMPORTANT: Remember that the remote display will always be slower than a real dedicated connection directly to the VM. In other terms, you will be much more comfortable using an SSH / NX / RDP client instead of the console, if you have this choice.
- Do not forget to set the networks to “autostart”, or you will get an error after a reboot if you try to launch a VM
- The connection to the VMs screen is done via VNC, and sometimes if you close one VM window and try to reopen it, the VNC client will get stuck trying to connect. If it happens, restart virt manager
- If you use a recent version of libvirt and virt manager (like the ones in Debian testing, I did not test squeeze ones), you might prefer to switch your VM display to the SPICE protocol, much more efficient than VNC but younger. I had great performance and nearly no bug with it, so feel free to test
- Do not expect hype features like 3D acceleration yet. As of now, libvirt is primarily aimed towards data serving machines accessed using SSH/NX/RDP. However, a lot of advance has been made recently with SPICE and the QXL video emulation.
- You have the choice to install your OS using either your host CD-Rom drive, or ISO images. I advise you to prefer ISO images, they are much more faster to install.
- If you do not need sound devices, delete them from the VM configuration. No real reason for that, it is just a bit cleaner. LibVirt will not try to add one on a remote hypervisor though.
