Chapter 6: domU Management: Tools and Frontends

From PrgmrWiki
Fishtank.jpg

Most of the material in this book focuses on fairly low-level administrative tasks. We’ve got a couple of reasons for this focus: first, because we feel that it’s better to understand what the GUI tools are doing before trusting them with your data,1 and second, because the add-on tools are not fully developed.

However, the true benefit of Xen is that it allows you to do things with a virtual machine that you can’t do—or can’t do easily—with a simple collection of physical machines. The main advantage of the more advanced management tools is that they exploit Xen virtualization to improve flexibility.

Besides, it gets kind of tedious to do everything from base principles all the time. In this chapter, we’ll take an excursion from our usual fixation on doing things in the most laborious way possible and look at some of the labor-saving innovations available for Xen.

Broadly, we can categorize the various frontend packages by their intended audience; some tools are for the dom0 administrator and some are for the domU administrator (that is, the customer in Xen’s computing-service model). The first group tends to focus on provisioning and destroying VMs, and the second group allows users who most likely don’t have access to the dom0 to control their own VM at a higher level so they can, for example, give the domain a hard reboot or recover when the domU won’t boot at all.

Despite this neat and theoretically useful division of labor, we’re going to ignore the second category almost completely. There are two reasons for this: First, most end users won’t want to do anything especially complex to their Xen instance. In our opinion, most of the Xen control panels are solutions in search of a problem. Second, almost none of the tools that we’ve tried in this category seem to have stabilized as of this writing.2 Instead, we’ll focus on the first category: software to simplify your life as a service provider, ranging from the simple to the elaborate. We’ll end by briefly discussing the Xen-shell, which is a useful minimal customer-facing tool.

Tools for the VM Provider

When looking for a management tool, as with any piece of software, the first question to ask yourself is, What features do I need? Xen management tools run the gamut from simple provisioning scripts, like Xen-tools, to elaborate data-center-oriented packages, like OpenQRM.

The biggest factor influencing your choice of frontend, assuming that multiple ones provide the necessary functionality, is probably the dom0 operating system. Some frontends, such as Xen-tools, are designed and built with Debian in mind. Some work best with Red Hat. Slackware users, you’re still on your own. Although you can install, say, virt-manager on Debian, it would be a difficult process, contrary to the dictates of nature.3 In this chapter, we’re going to focus on each tool in its native environment, beginning with Xen-tools for Debian.

Xen-tools

Xen-tools, at heart, consists of a cross-platform set of Perl scripts for automated installs, so it’s fairly distro agnostic. Even though the authors develop on Debian, distribute .deb packages, and have an Apt repository, Xen-tools is relatively easy to install on other systems, so we encourage you to try it regardless of which distro you’re running. Download a tarball at http://xen-tools.org/.

Installing Xen-tools

In the interest of keeping everything flowing smoothly, we installed Xen-tools on a Debian machine using Debian’s Apt system. Because, like everything Xen-related, Xen-tools is under heavy development, we opted to get the package from the author’s own repository to avoid getting an old version.

To do this, add his repo to your /etc/apt/sources.list. For Etch, we appended:

#
# Steve Kemp's repository: Etch
#
deb     http://apt.steve.org.uk/etch etch main non-free contrib
deb-src http://apt.steve.org.uk/etch etch main non-free contrib

NOTES: Sometimes even the version in Apt is not as current as the one on the website. If all else fails, download the tar package, unpack it, and run make install to install it.

Then run, as usual:

# apt-get update
# apt-get install xen-tools

Apt will then work its customary magic, installing the Xen-tools scripts and creating a configuration directory, /etc/xen-tools.

For usage information, if you have perldoc, you can access any of the programs’ embedded manual pages by running them with the --manual option. For example:

# xen-create-image --manual

will print out a long and intimidating man page. Don’t be discouraged; it’s just exposing the bewildering array of options Xen itself makes available. You can simplify things by specifying most of these options ahead of time in the Xen-tools config file rather than by command-line options.

Configuring Xen-tools

So let’s make a config file. Trust us, it’s much more pleasant to spend a bit of time setting some defaults rather than specifying global options every time you use the command.4 Put your preferred options in /etc/xen-tools/xen-tools.conf. We would use something like this:

lvm = verona
size = 2Gb
image = full
memory = 128Mb
swap = 128Mb
fs = ext3
dist = sarge

initrd = /boot/initrd.img-2.6.16-2-xen-686
kernel = /boot/vmlinuz-2.6.16-2-xen-686

install-method = debootstrap

Fill in appropriate values, as always, and feel free to add from the liberally commented sample config anything that strikes your fancy. Some of these options, like initrd and kernel, specify literal directives that’ll wind up in the final domU config file. Of the remaining options, most are self-explanatory; size specifies the filesystem size, swap is the amount of swap the domain will have, and so forth.

Because we’ve specified an LVM group, domains will be created with LVM volumes as backing store. You can also use filesystem images by specifying dir = /path/ rather than an LVM group. If you do that, make sure that the directory exists, otherwise the image creation step will fail silently and xen-create-image will populate the directory where the filesystem would have been mounted. This is almost certainly not what you want.

Also note the dist= line; this specifies which set of postinstall hook scripts xen-create-image will run to configure the new domain. If there isn’t a directory under /usr/lib/xen-tools corresponding to the dist value, xen-create-image will exit with an instructive error message. If you don’t want to configure the domain at creation time, you can create an empty directory— say, /usr/lib/xen-tools/plan9—and pass the name of the distribution (plan9 in this case) as the dist value.

When you have the config file populated, actually creating domains is so easy as to be almost anticlimactic. Just specify a hostname, preferably fully qualified so that the postinstall scripts can configure the image correctly, on the command line, and the tool will do the rest. For example:

# xen-create-image mercutio.prgmr.com

NOTES: Although setting a fully qualified domain name allows the postinstall scripts to handle

domain configuration, it can cause trouble with the xendomains script on certain

Red Hat derivatives, which assumes a domU name no longer than 18 characters.

With the config file previously shown, this creates two logical volumes, /dev/verona/mercutio.prgmr.com-disk and /dev/verona/mercutio.prgmr.com-swap. It then mounts the disk volume and uses debootstrap to install sarge (Debian 3.1).

Easy.

Xen-tools and RPM-based DomU Images

The first versions of Xen-tools were developed with debootstrap installs of Debian in mind. However, the package has come a long way, and it’s been generalized to support virtually every system out there. RPM-based distros are covered via a debootstrap-like tool. Other systems—even non-Linux systems— can be installed by copying a pristine filesystem image or extracting tarballs.

Although older versions of Xen-tools used RPMstrap, which we’ve used with some success in the past, the author of RPMstrap has ceased to develop it. Accordingly, the Xen-tools author has been working on a replacement called rinse. It’s the recommended way of installing CentOS and Fedora with Xen-tools, and it’s a fairly neat package by itself.

rinse’s home page is at http://xen-tools.org/software/rinse/. Download it either from the download page at that site or by adding his apt repository and downloading via your package manager.

A full discussion of rinse’s configuration options is probably out of place here. We enjoin you to read the fine manual. However, it works out of the box with an install method for xen-create-image, with a simple command line like the following:

# xen-create-image --hostname tybalt.prgmr.com --install-method=rinse
dist=centos-5

No problem.

Xen-tools Postinstall

After the image is installed, but before it’s started for the first time, xen-create-image does some postinstall work. First it runs some scripts in the mounted domU filesystem to perform setup tasks, like setting the hostname and disabling unneeded gettys. Finally it creates a config file so that you can start the domain.

At this stage you can also have the machine configure itself with a role— specify the --role <script> command-line option to run the corresponding script located in /etc/xen-tools/role.d at the end of the install, taking the mount point of the domU root filesystem as an argument. The roles that you’ll want will depend on your needs. For example, you may want roles that differentiate between web, mail, and dns servers. The Xen-tools distribution comes with some samples that you can build upon.

After populating the domU image, the xen-create-image script will create a configuration file based on a template at /etc/xen-tools/xm.tmpl. You can also specify a template on the command line using the --template option.

Extending the Config File Template

You can, as you might suppose, edit the template freely. Xen-tools passes options to the setup scripts by reading environment variables, which makes it easy to extend the template by passing in new variables and adding code to interpret them.

For example, because we like to use PyGRUB, we might edit the template by adding a bootloader option right beneath the kernel and initrd sections:

{ if ( $bootloader )
   {
      $OUT.= "bootloader = '$bootloader'";
   }
}

Now we can create the image with the additional value specified as an environment variable:

# bootloader=/usr/bin/pygrub xen-create-image --hostname tybalt.prgmr.com
--install-method=rinse --dist=centos-5

and, as if by magic, the config file will have a bootloader entry.

We could also update the script that parses the config file and have it pass through the value as an environment variable, just as with the other options.

xen-list-images

Having created some domains, whether with xen-create-image or otherwise, it stands to reason that you might want to view a summary of domains that exist on the machine. The Xen-tools suite therefore also includes a tool to list existing domains, xen-list-images. Running it on our test system shows:

# xen-list-images
Name: mercutio.prgmr.com
Memory: 128
DHCP

Name: benvolio.prgmr.com
Memory: 96
IP: 192.168.1.93

Name: tybalt.prgmr.com
Memory: 128
DHCP

The tool parses Xen config files—both those created by Xen-tools and otherwise—and prints some information.

xen-delete-image

Finally we have xen-delete-image, which does exactly what the name suggests. It’ll only work on images that follow the naming conventions used by xen-create-image, that is, it doesn’t have the intelligence necessary to parse arbitrary domain definitions. Nonetheless, if you’ve standardized on Xen-tools or name disks in the format used by Xen-tools, it can be handy.

WARNING: When run as root, this command will destroy data with no confirmation, even if you’ve specified the --test option.

Run xen-delete-image like this, but very carefully:

# xen-delete-image --lvm verona mercutio.prgmr.com

This will delete mercutio.prgmr.com-disk and mercutio.prgmr.com-swap from the verona VG and delete the config file. If the data source option isn’t specified, it defaults to the value in /etc/xen-tools/xen-tools.conf.

libvirt, virsh, and virt-manager

On the RPM side, including SUSE, CentOS, and Fedora, we have a suite of tools based on libvirt. Although packages are available for Debian-based distros and Solaris, libvirt is primarily developed on, for, and largely by Red Hat, and it shows in the project’s focus.

libvirt isn’t a management tool per se, but it’s worth mentioning here as a Xen frontend, part of a framework that sits in front of Xen and makes management tools easier to develop. The stated goal of the libvirt project (at http://libvirt.org/) is to “provide a long-term stable C API for virtualization.” In other words, libvirt aims to allow a single tool to control any type of virtualization that libvirt supports, including Xen. This is wonderful, as far as it goes. However, the libvirt-based tools still aren’t complete. In particular, their focus on Red Hat is sometimes inconvenient.

libvirt’s main advantages are that it integrates very closely with Red Hat’s tools and that the management tool, virt-manager, is excellent for interacting with live virtual machines.

The basic libvirt-based tool—or at least the first-generation, proof-ofconcept version of this tool—is virsh, or virtualization shell. Right now it can perform most of the same operations as xm on Xen, QEMU, KVM, or OpenVZ domains.

This isn’t the place to give a complete how-to on virsh; we’re writing about Xen, and we’ve therefore focused on xm. However, virsh is the frontend of choice for Red Hat, so we’ll provide some discussion and examples, but we’re going to stick to Xen and virsh’s capabilities as an xm replacement rather than emphasizing the new features virsh introduces.

The first thing to mention about virsh is that it uses a flagrantly different syntax from xm for domain definitions. virsh has a create command, just as xm does, but it expects an XML file as an argument.

Fortunately, virsh allows you to create the XML definition from a Xen domain. For example, to get an XML definition for the running domain ophelia:

# virsh dumpxml ophelia
<domain type='xen' id='8'>
<name>ophelia</name>
<uuid>162910c82a0c03332349049e8e32ba90</uuid>
<bootloader>/usr/bin/pygrub</bootloader>
<os>
   <type>linux</type>
   <kernel>/var/lib/xen/vmlinuz.8gmQDM</kernel>
   <initrd>/var/lib/xen/initrd.H3wHj2</initrd>
   <cmdline>ro root=/dev/VolGroup00/LogVol00 rhgb quiet</cmdline>
</os>
<memory>105472</memory>
<vcpu>1</vcpu>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
   <interface type='bridge'>
      <source bridge='xenbr0'/>
      <mac address='00:16:3e:4b:af:c2'/>
      <script path='vif-bridge'/>
   </interface>
   <graphics type='vnc' port='5900'/>
   <disk type='file' device='disk'>
      <driver name='tap' type='aio'/>
      <source file='/opt/xen/images/ophelia.img'/>
      <target dev='xvda'/>
   </disk>
   <console tty='/dev/pts/6'/>
</devices>
</domain>

You can see the correspondences between this XML definition and the domain config file; it defines the same basic resources in a different but still recognizable format. We can redirect this to a file, say ophelia.xml, shut down the original ophelia, and create a domain:

# virsh dumpxml ophelia > ophelia.xml
# virsh shutdown ophelia
# virsh create ophelia.xml

virsh can also list domains, just like xm:

# virsh list
  Id Name            State
----------------------------------
    0 Domain-0       running
    4 ophelia        blocked

Finally, just as with xm, virsh can shutdown, restart, or destroy a domain, using the obvious command for each.

virt-manager

Apart from virt-install, which we discussed in Chapter 3, the most useful tool in the suite is probably virt-manager. It’s pretty slick, and it’s great in an area that’s not covered by the Xen-tools scripts: interacting with live virtual machines. As Figure 6-1 shows, virt-manager provides a centralized location from which to view performance data, virtual framebuffers, and consoles. It also provides a quick overview of resource allocation. Like most GUI tools, virt-manager probably requires a bit more manual effort and clicking through dialog boxes than you’d like to use for everyday domU creation. Nonetheless, it’s got support for Xen’s basic life cycle: create and destroy. It also integrates with Red Hat’s Kickstart deployment method for convenient semiautomated installs; you can just specify a .ks file when asked during the install dialog.

Getting Started with virt-manager

Because virt-manager comes with the operating system on Red Hat–based distros, assuming you selected the virtualization target during the install, you can invoke it without setting a finger to the keyboard. From the default GNOME desktop, click the Applications menu at the upper left, then System Tools->Virtual Machine Manager. Enter your root password when prompted, and select Local Hypervisor to connect to the local instance of xend.5 You’ll be presented with something that looks vaguely like Figure 6-1.

Figure6-1.jpg

Figure 6-1: Here you can see the virt-manager UI. It has a main screen that lists virtual machines; a detail view that shows VM resources, performance, and statistics; and a framebuffer console as well as a text console interface.

To create an image using virt-manager, click the File menu, and select New Machine. virt-manager will then escort you through a set of friendly dialog boxes.

The first prompt asks for the name of the machine. Enter something appropriate, but make sure that it includes only letters, numerals, and underscores—fully qualified domain names won’t work.

Choose whether the new domain should be paravirtualized or fully virtualized. If you don’t have HVM, the dialog will chide you for your inadequate hardware and gray out the Fully Virtualized option. For the purposes of this example, we’re picking paravirtualization.

Then virt-manager asks for install media. Because it’s calling virt-install behind the scenes, the same constraints apply; enter a network-accessible Red Hat–style directory tree. For example, http://archive.fedoraproject.org/pub/archive/fedora/linux/releases/7/Fedora/i386/os/ will install Fedora 7 from an HTTP Fedora mirror.

At this stage you can also specify a Kickstart file using the same syntax. This is a handy way to automate the install process. For example, to duplicate your dom0’s install configuration, upload your /root/anaconda-ks.cfg file somewhere convenient and specify it.

Select a backing store for the new virtual machine. The two choices map to phy: and tap:aio: devices. The GUI also gives you the option of creating a sparse file, but we don’t recommend that for the reasons we described in some detail in Chapter 3.

Select a networking option. The two choices correspond to a network-nat work-alike and the standard network-bridge.

Finally, select memory and CPU allocation.

At the end, virt-manager will list your configuration and give you a chance to back out. Make sure everything looks right and click Finish. It’ll validate your selections and then start creating the domain using virt-install.

The domain creation itself is probably the coolest feature of the libvirt suite. Rather than populating a filesystem from the dom0 and then booting a Xen instance, it downloads a Xen-aware net install image, immediately boots the domain from the installation kernel, and then uses that kernel to download the packages and populate the system. The install looks just like a normal install, using the framebuffer console to provide a completely ordinary Red Hat install experience. Because virt-manager integrates a VNC viewer, you can watch the install as it progresses from within the management application.

When the domain is running, you can pause it, shut it down, or examine its configuration from the main virt-manager window. Right-click the domain name to get a context menu with a list of operations, or select the domain by clicking it and use the button bar at the top.

NOTE: Early versions of virt-manager, including the version shipped with Red Hat Enterprise

Linux 5.0, suffered from the oversight that when a domain had stopped, it vanished from the GUI’s list and thus couldn’t be restarted without dropping to a command prompt. If you have an afflicted version, you can easily restart a domain configured

with virt-manager using xm in the normal way.

Still, though, virt-manager is too interactive and too limited to use for large installations. To address this, Red Hat’s Emerging Technologies group (http://et.redhat.com/) is also working on a tool called oVirt, which aims to scale libvirt-based management across the entire data center. Another tool, the Puppet Recipe Manager, emphasizes the software appliance aspect of virtualization. It enables an administrator to build software recipes and automatically install them on virtual machines. We also mentioned Cobbler, an automated tool that can provision virtual machines, in Chapter 3.

One last libvirt-based tool that you might want to look at is virt-clone which is capable of duplicating a domU image and its config file while altering any items that must be unique, such as the MAC address—a nice balance between convenience and control.

It’s simple to run, taking most of its input from command-line options. For example, to clone the machine sebastian as viola:

# virt-clone -o sebastian -n viola -f /opt/xen/viola-root.img -f /opt/xen/
viola-swap.img preserve-data

Administering the Virtualized Data Center

The chief benefit of frontends like these is in doing things that are too large or complex for simpler tools, though some would hold that no problem is too large for a well-written shell script. The largest and best of these tools allow Xen to truly flex its capabilities, leading to giant automated systems such as we see in Amazon’s EC2. In this category are several packages, including OpenQRM, ConVirt, and Enomalism. All three of these have their adherents. However, we’ve chosen to focus on lower-level, more commonly available tools while the more advanced management frontends stabilize.

Administration for the VM Customer

Although there are a number of packages aimed at allowing customers to manage their virtual machine, it’s not clear to us that control-plane operations are really necessary in this context. Furthermore, most of the customerfacing tools are still in flux. It’s difficult for us to recommend them at this point.

The best available solution, at least for now, seems to be to provide a reasonably simple menu through which customers can attach to the console, reboot their machines, shut down their machines, and re-create their domU image. There are many ways to do this. Our favorite is to allow customers to SSH to a console server, with sharply limited accounts.

Xen-shell

Although we like our home brew approach to management as described in Chapter 7, there are other options for those who believe that software can be perfectly adequate even if they didn’t personally write it. Of these, our favorite is the Xen-shell. It’s by the author of Xen-tools and is another example of his no-frills approach. We recommend it, not merely because we like Xen-tools or even because it’s got a good feature set, because it doesn’t have a giant list of dependencies. It’s a simple product that does its job well.

It’s available at http://xen-tools.org/software/xen-shell/. After you’ve downloaded it, use the standard unpack && make install process to install it. At this point there’s some configuration that needs to be done. Xen-shell works by taking user commands and running sudo xm in response to their input. You’ll need to put xm into their path or, conversely, alter their path to include xm. We took the former approach:

# ln -s /usr/sbin/xm /usr/bin

We also need to configure /etc/sudoers to make sure that the users are allowed to use sudo to run xm on their domain (and only their domain). This entails quite a number of additions to the file, one for each command we want to allow:

marlowe ALL=NOPASSWD:/usr/sbin/xm create goneril
marlowe ALL=NOPASSWD:/usr/sbin/xm create -c goneril
marlowe ALL=NOPASSWD:/usr/sbin/xm destroy goneril
marlowe ALL=NOPASSWD:/usr/sbin/xm shutdown goneril
marlowe ALL=NOPASSWD:/usr/sbin/xm list goneril
marlowe ALL=NOPASSWD:/usr/sbin/xm console goneril
marlowe ALL=NOPASSWD:/usr/sbin/xm reboot goneril

Then change the shell of the appropriate users to the Xen-shell. For example:

# chsh -s /usr/local/bin/xen-login-shell marlowe

To mark that a user is allowed to administer a domain, simply add the user to a line in the domain config file—an elegant and ingenious solution. We’ll use the domain goneril as an example:

name = 'goneril'
xen-shell = 'marlowe'

Now, when marlowe logs in, he’ll be presented with the Xen-shell interface from which he can execute various commands (get a list by typing Help).

NOTE: Although Xen-shell reads domain config files to find which domains can be administered

by a user, it doesn’t actually keep track of the config file’s name, as of this writing. Domain configuration filenames, to work with Xen-shell’s boot command, must be of

the form <domU name>.cfg. Thus, goneril’s config file must be /etc/xen/goneril.cfg.

To extend the example, let’s say that marlowe can administer multiple domains. Simply add the username to both domains, and use the control command in Xen-shell to switch between them. One of the niceties of Xenshell is that the command only shows up if it’s necessary.

xen-shell[goneril]> control regan
Controlling: regan
xen-shell[regan]>

Convenient, isn’t it?

Really, though, this is just the beginning. The client software for Xen is still in turmoil, with constant development by multiple factions.

You may have noticed that we’ve left out a couple of prominent frontends. For one, we haven’t even mentioned Citrix’s offering because we cover it in Chapter 11. We also haven’t addressed Amazon’s EC2, which is probably the nearest thing to utility computing at present. As always, there’s a big field of tools out there, and we’re just aiming to make them seem manageable and talk about what works for us.


Footnotes

1A position unlikely to occasion much disagreement among sysadmins.
2We blame Python’s scorched-earth policy toward compatibility.
3Of course there are packages, but they’re less closely integrated.
4Cdrecord, anyone?
5Well, if you’d been logged in as root, you wouldn’t have had to type anything. Not that we recommend that, of course.

Navigation

Previous Chapter | Next Chapter