About 6-7 years ago I recall my first brief foray into virtualization; it was done purely out of curiosity. The host system was my college computer, a 1.8 GHz AMD Athlon CPU running Windows XP with maybe a gig of RAM… I thought, “Surely this beast was up to the task, it could run Quake 3 after all!”
After installing VMWare Workstation and getting familiar with the interface, I had installed another Windows XP system… running on my Windows XP system. For some reason at the time, I felt there was something strange about this. While my hardware struggled along, I waited patiently to install a few applications and convince myself that I was indeed working with another separate system.
At that point, I just didn’t get it. Why cripple good hardware to run another operating system? You could run one operating system with good performance, or many at a fraction of the performance (depending on settings and configuration and such, but you get the point). Sure it was a fun gimmick, you could close an instance of Windows with a button click or try to install sketchy programs and viruses in the virtual machine to see how much it could take before it croaked. Though, running a virtual machine as a server in this fashion (full OS running on full OS) somehow seemed like a bad idea…
Over the next couple of years, while “virtualization” became the “value-add” corporate-buzzword (before “cloud computing” hit the scene), the industry had a chance to fully harness this new enigmatic concept. Now, many innovative uses for virtualization have appeared, and many are still just taking shape. I’ll provide you with a small list of them that I lacked back during my first trials…
- Able to run various operating systems to test cross-platform applications during development
- Run software that is not compatible with native OS
- Isolate single server roles to separate virtual machines
- Create flexible backup/restore system that does not have any strict hardware limitations
- Migrate virtual machines between physical hosts based on resource consumption
- Testing virus propagation through isolated networks of virtual machines
- Running simulations of various network structures
I have just skimmed the surface here, and for the purpose of this article I will be focusing mainly on the “Server” uses.
About a year ago, I was asked to look into various virtualization solutions. We had some powerful, slightly-aged, Ubuntu Linux systems running along just fine; however, a few hard lessons in crash recovery and redundancy were enough to warrant a double-take at virtualization. I might add that while we run a tight ship here on our internal network, we host our main site www.logicsupply.com externally for additional reliability and performance (our hosting provider happens to run Xen as well!).
The main benefits we saw in virtualization focused around crash-recovery, organization, and manageability. We wanted the ability to better organize our internal servers, as well as develop an easy-to-manage, crash-proof configuration.
There were many options to be considered… Did we want to run a Windows or Linux host? Did we value the latest and greatest visual management suite with more knobs and buttons and add-on software than you can shake a stick at, or a quick and effective command line interface?
There are many ways to approach virtualization, and not all of them come with price tags. After testing offerings such as Microsoft’s Hyper-V, VMWare’s ESX server, and Xen, as well as taking a close look at many others, we came to a conclusion.
While all of these packages had their own sets of pros and cons, (for the sake of some brevity I will refer you to Google), we were able to come up with a clear decision that we felt was a good fit for our setup. We found that by sacrificing friendly visual interfaces and getting our elbows dirty in a bit of scripting and command line wizardry, it would pay off in the increased performance and low-overhead that could be provided by Xen. Plus, we are software developers after all!
Xen is a mature and widespread virtualization solution that has existed for quite some time. It was developed at the University of Cambridge (UK) and was first publicly released in 2003. Since then, it has snowballed into a powerful virtualization platform that has gained the support of companies such as Citrix, IBM, Intel, Hewlett-Packard, Novell, Red Hat, Sun Microsystems, and Oracle. Citrix actively markets a few of its own products that are powered by Xen technology, though the core project remains free and open to all who wish to use it (Licensed under the GPL2). Novell and Red Hat even provide some well-integrated Xen management interfaces with their Linux distributions.
Under the Hood
Before discussing the features of Xen, there are a few important concepts that anyone looking into virtualization should understand. Below, I have a list of relevant terms.
What follows is some terminology that I had to trudge my way through to get a stronger grasp on virtualization. To start this section off, I will describe a few terms and concepts that are important with respect to Xen (and virtualization in-general), and are also pretty fun to say, I might add!
The hypervisor is essentially the layer that sits between the virtual machines and the host hardware. It handles all the system calls made and manages access to the underlying hardware. Hypervisors are not specific to Xen; they are also referred to as “virtual machine monitors.”
Hypercalls are more relevant to a Xen configuration (see “Paravirtualization” in the next section for an additional description). The XenSource wiki puts it best: “…a hypercall is to a hypervisor what a syscall is to a kernel.”
This is the name, in Xen-speak, given to the host operating system that is installed on the raw hardware (the virtualization host).
This stands for “Unprivileged domain” or a guest operating system that is not able to directly access physical hardware (a virtualized guest or virtual machine).
There are many ways to create an abstraction of hardware resources to provide a virtualized environment to multiple guest operating systems (this is considered platform virtualization, versus application virtualization). The following are three fairly common approaches that all have their pros and cons.
This approach enables a host system to run virtualized guest operating systems that are fully unaware of their virtualized state. The host takes care of handling all of the system calls these guests make. The benefit is that you are able to run a wide range of operating systems in this manner as they don’t require modifications to run as a guest. The downside is that this incurs a bit of additional overhead on the host system. It wouldn’t be safe or stable to give a VM direct access to the hardware, so it has to be orchestrated by the host hypervisor.
- Hardware-assisted virtualization
This is basically slight variation of full-virtualization, often providing better performance through reduced software overhead. Virtual machines running on the host system are able to make requests to the hardware that need very little, if any, modification before hitting the actual CPU. This is possible, provided you have a processor that supports this technique. I will add that many current desktop processors do provide this functionality (Intel VT, AMD-V).
Not only is it a great buzzword, it is also the bread-and-butter of our Xen setup. Paravirtualization is kind of a middle-ground with respect to the above methods. At the cost of slightly modifying an operating system’s kernel, on both host and virtual machine, we are able to achieve a performance gain without requiring a hardware upgrade. By altering a guest operating system’s kernel, it is able to make system calls that are already virtualization friendly and impose less “translation” overhead on the hypervisor. This approach works great with Linux systems; however, it is not well suited for Windows. (Windows can be run in Xen, but requires a hardware-assisted setup.)
So by now you have had a brief overview of some of the virtualization methods available, their general pros and cons, and perhaps some insight into our decision process when it came to choosing Xen. Let’s take a step into the details of our current setup and how we are using Xen here at Logic Supply.
For our host’s operating system, we wanted to stick with Ubuntu, as it is the distribution we are most familiar with. Unfortunately no 9.04 or 9.10 for us, Ubuntu 8.04 is the last LTS distribution that can be run as a virtual machine host or “Domain-0” with Xen. This is mainly because the Xen hypervisor requires integration into the Ubuntu kernel. From what I gathered, this integration with the Ubuntu kernel incurred additional maintenance overhead that the Ubuntu team felt was not warranted. Ubuntu runs well as a server operating system, however with a broad goal to appeal to people with a wide range of needs, perhaps the integration overhead was best left to be tackled by the more server focused distributions such as SUSE and Red Hat. Ubuntu’s “official” virtualization solution is now KVM (Kernel Virtual Machine). KVM is an up-and-coming package that provides a lot of similar functionality; however, it requires hardware that supports virtualization and lacks the time-tested industry support of the aforementioned companies. KVM supports many of the same features as Xen, and it is possible to migrate Xen virtual machines to KVM. I should mention it is also still possible to run newer versions of Ubuntu as virtual machines on a Xen host. All this being said, Ubuntu 8.04 is rock solid and actively supported long enough for us to consider it a worthy platform to virtualize on.
As far as hardware is concerned, our old server hardware was completely up to this task. Why create more waste? Sure, there are much larger Xen configurations, running machines with tens of gigabytes of ram and terabytes of hard drive space. But our twin systems sporting Intel P4’s with their 500 GB RAID 5 storage are more than sufficient for providing us with all we need. In practice, we have not seen a noticeable decrease in performance of our internal systems.
Many of you may be asking yourselves, “Why does a company that sells computer hardware, have an issue with upgrading their own?” Well, the explanation is rather straightforward: the systems we planned to virtualize, our local development server and our local intranet server, didn’t require many resources to properly perform their tasks. Additionally, Xen allows us to create virtual machines that take up about 300 MB initially and can run side by side with another one or two systems with an acceptable level of latency.
To give you a better idea of how we have set up our virtual machines, please refer to the following diagram:
As you can see here we have isolated our web server and database server. Isolating servers on a per-role basis is a nice feature that is cheaper to implement with VMs, as and additional role does not equal an additional set of hardware. We are also able to modify our backup policy based on how frequently data changes. For a database virtual machine, backups can be done more frequently. For a web server, we only need to take a full backup when we install new web applications.
You may recall I mentioned us having two systems with the same hardware. Now that we are virtualized, hardware doesn’t really matter to the virtual machines. Installing a copy of Ubuntu without the graphical component, and “apt-get”-ing 5 or 6 packages is all it takes to have another Xen host at our disposal. Our second set of hardware acts as a staging area, where we are able to create new virtual machines for testing purposes. Additionally, in the event that our main host fails, we are able to boot the backed-up virtual machine images in a snap and be back to normal in a matter of minutes.
Command Line Love
One of the features of Xen that I briefly mentioned earlier, is that it can be easily driven entirely from the command line. While this was initially a slight worry for someone coming from the friendly VMWare/VirtualBox user interfaces, it was quickly found to be a powerful tool.
Now I don’t want to get into the whole process for setting up and working with Xen, that information is easily available all over the Internet (feel free to ask specifics in the comments as well). I will, however, give you a taste of the power at your fingertips of using the command line to manage virtual machines!
One process you may be familiar with is creating a virtual machine… Set up the allocated drive space and image file, choose RAM size, set up networking… where was the window for the networking again… oh man I need to go back a few screens… d’oh!
Behold! The creation of a new Xen virtual machine in a single command (additional whitespace + indentation added for cleanliness):
xen-create-image --hostname=sandbox01 --size=2GB --swap=512MB --dir=/home/xen --memory=512MB --arch=i386 --kernel=/boot/vmlinuz-2.6.24-24-xen --initrd=/boot/initrd.img-2.6.24-24-xen --install-method=debootstrap --ip 192.168.1.101 --passwd --force
I mentioned paravirtualization earlier, here it is in action: we are configuring this machine to use a modified Xen kernel. The host is also running the same kernel! For the host, the Xen kernel is acquired and set up at the time of installing the Ubuntu Xen packages.
This is typically the command we issue when we need to create a new system. You can see that we are passing some basic arguments that describe a new Ubuntu-based virtual machine to the tool “xen-create-image” (part of the “xen-tools” Ubuntu package). Through an additional global configuration file for xen-tools we have simply said that any machine created with that command should be 8.04 (hardy) and should be given a static IP address. The new machine image (located at /home/xen/domains/sandbox01) will have its image pre-populated with a minimal install via “debootstrap”. (“debootstrap” is a method for populating a partition/directory-path with a bootable Linux installation, pulled from the aptitude repositories for the desired distribution.) There is also the possibility to create new machines without using all the above parameters, given they are specified in the configuration file for the xen-tools package.
With the creation of a virtual machine complete (approximately a 5-minute process for the above system command), we must now start it up…
xm create /etc/xen/sandbox01.cfg
Using the above command, all we need to do is specify where the machine’s configuration file sits. With our creation method (xen-create-image), we have these files all stored in “/etc/xen/”.
From this point on, we are able to manage our virtual machines by running a fairly self-descriptive set of commands, all prefixed with “xm” (Xen management user interface):
Common 'xm' commands: console Attach to 's console. vncviewer Attach to 's VNC server. create Create a domain based on . new Adds a domain to Xend domain management delete Remove a domain from Xend domain management. destroy Terminate a domain immediately. dump-core Dump core for a specific domain. help Display this message. list List information about all/some domains. mem-set Set the current memory usage for a domain. migrate Migrate a domain to another machine. pause Pause execution of a domain. reboot Reboot a domain. reset Reset a domain. restore Restore a domain from a saved state. resume Resume a Xend managed domain save Save a domain state to restore later. shell Launch an interactive shell. shutdown Shutdown a domain. start Start a Xend managed domain suspend Suspend a Xend managed domain top Monitor a host and the domains in real time. unpause Unpause a paused domain. uptime Print uptime for all/some domains. vcpu-set Set the number of active VCPUs for allowed for the domain.
The above commands can be used to work with the virtual machines in the same way that any typical graphical interface would allow. While you could easily run many of these commands on their own, the ability to incorporate them into standard scripts as well adds a bit of power.
Performing any console work on the actual virtual machine through the command line is pretty easy:
xm console sandbox01
That command will get us into the specified virtual machines console. Simply hitting “ctrl + ]” pops you back to the host’s terminal. You could also install something like “openssh-server” and give yourself remote access. SSH leaves us with an access method that is standard for all our servers. While working remotely we can tell we are working on a virtual machine by checking our kernel:
matthew.gagne@sandbox01:~$ uname -r 2.6.24-24-xen
…ahh yes that’s right!
I should also add that while the console is great, it may not be for everyone. It is entirely possible to run a full graphical environment on a virtual machine in Xen. VNC is actually the standard method used to connect to VMs running X, though other remote desktop applications can be installed as well.
As I have just shown, Xen can be quickly and efficiently managed from any remote system over an SSH connection. There are lots of short commands that can be run to provide all the same functionality that a large management interface would expose to the user. I should mention that there are also many graphical programs available that can work with Xen and provide an interface similar to VirtualBox or VMWare.
Now that I have covered some of the basics of Xen, and given a snapshot of our local configuration, I will get into a couple of the more juicy details of why it’s so great.
If you’re coming from a Linux background, scripting with Xen doesn’t require any additional knowledge of new language, unfamiliar procedures, etc. We simply write down what we would like to do using the “xm” commands.
To take a simple backup using the save/restore method we issue a few simple commands:
xm save sandbox01 /home/xen/domains/sandbox01/checkpoints/temp_check.ckpt cp /home/xen/domains/sandbox01/disk.img /mnt/backup/sandbox01/disk.img xm restore /home/xen/domains/sandbox01/checkpoints/temp_check.ckpt
This is similar to hibernating a Windows system, copying its hard drive, and then booting it back up.
You can see from this example that writing a script to back up a running Xen system would be pretty straight forward. There are many more advanced ways to work with Xen in this fashion, including using a multitude of different storage methods as opposed to plain .img files, but I won’t get into that here.
There are many more advanced and powerful features that Xen provides… For example, while researching Xen I came across this paper. It explains an experiment that was conducted to evaluate Xen’s ability to move a live virtual machine from one physical host to another. The test involved moving a virtual machine, running a Quake 3 server from one system to another, with only 60ms of latency imposed by the process. This idea is called “live migration” and really unlocks a whole new world for virtualization. Imagine being able to dynamically manage a bunch of virtual machines, moving them between hosts as resources allow.
While the Xen hypervisor is not unique in its ability to provide features such as live migration, it has a few other tricks up its sleeve. One of these is its API, a Xen system that can be programmatically controlled using languages like Python. This has some rather powerful implications. It opens many possibilities such as building an application that could be used to dynamically spin out virtual machines as needed, perhaps even for public use (VPS hosting for example).
Lastly, I would like to point out a project that caught my eye recently: Remus. This project is a great example of what is possible with a well-managed open-source project. It’s out there, and people can adapt it to suit their needs and even improve upon it freely. In this case the UBC Distributed Systems Group has come up with an addition to Xen that provides some pretty amazing functionality. To sum it up, Remus enables a Xen virtual machine to be duplicated on a separate host system running Xen as well. Over time, as changes are made, both systems are kept in sync; the duplicate system is completely identical, though not active. At the moment where the main virtual machine fails or the host system explodes, the duplicate is enabled. The change-over is instant and everything resumes. This has a benefit over live migration in that when host A explodes, you aren’t able to move it’s virtual machines over to host B at that point. Pretty cool, no?
Hopefully, I have been able to provide a somewhat comprehensive snapshot of our journey into virtualization. Through our experience with Xen we have learned a lot, and discovered some very beneficial features. Be aware that I have intentionally left out many details as virtualization is a very dense topic and my aim was to boil it down for the purpose of this general post. Feel free to pick my brain in the comments section, I can’t guarantee an answer but I will try my darnedest!