Chapter 14: Tips

From PrgmrWiki
Fishtank.jpg

By now you’re some kind of Xen expert, we imagine.1 As such, now we’d like to devote a chapter to the more esoteric aspects of working with Xen. Here are some things that didn’t seem to fit anywhere else—stuff like the framebuffer, forwarding PCI devices, or building added functionality into the XenStore. Tips, in other words.

A number of the topics in Chapter 15 might also come in handy when you work through our examples here. Some of the software discussed here is even more bleeding edge than the rest of Xen, which is itself some kind of heavenly sword, ravening and incarnadine. What we’re trying to get at is that the material in this chapter might not work straight off.

Compiling Xen

Although we’ve relied, for the most part, on Xen packages provided by distro maintainers, we think it’s generally worthwhile to compile Xen from scratch. This will allow you to use a much more up-to-date version of Xen than versions that come with the distros. It also lets you enable options that the distro maintainers may have disabled.

If you’re feeling adventurous, it’s also nice to be able to play with the code—change it around a bit, perhaps, or add some printk messages to help with debugging.

The easiest way to compile is to check out the latest source from the Mercurial repository. Start by making sure you have Mercurial and a bunch of build dependencies. On CentOS 5, we installed these packages with yum:2

  • mercurial
  • zlib-devel
  • gcc
  • libX11-devel
  • openssl-devel
  • ncurses-devel
  • bridge-utils
  • python-devel
  • git
  • dev86
  • glibc-devel

If you want the docs to build successfully, you should also install the following packages:

  • texinfo
  • tetex-latex
  • pstoedit
  • transfig

Given the large amount of Xen documentation available online and from other sources, the included docs are fairly optional.

When these packages are installed, clone the development repository. We’re using xen-unstable here, but if you’d like to use a less unstable repo, you might want to try something like xen-3.3-testing.hg. As of early 2009, prgmr.com runs xen-3.3-testing.hg. It has been pretty stable.

# hg clone http://xenbits.xen.org/xen-unstable.hg
RPMFORGE REPOSITORY

If you want to install all the previously listed packages using yum, you must use the RPMForge repo. To do so safely, install yum-priorities. On CentOS 5:

# yum install yum-priorities

Edit each of the files under /etc/yum.repos.d/* and add the line priority=N, where N is a number from 1 to 99 (lower number is greater priority). You want base, addons, updates, and extras to be priority 1; centosplus and contrib to be priority 2; and everything else to be priority 10 or higher.

Now, install RPMforge:

wget
http://apt.sw.be/redhat/el5/en/i386/RPMS.dag/rpmforge-release-0.3.6-1
.el5.rf.i386.rpm

Of course, if your machine is x86_64, substitute the appropriate architecture:

rpm --import http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt

then install rpmforge with:

rpm -K rpmforge-release-0.3.6-1.el5.rf.*.rpm

Finally, edit /etc/yum.repos.d/rpmforge.repo' and add priority=10.

This will download the repo to a local directory (xen-unstable.hg in this case). Next, cd into that directory and run make world:

# cd xen-unstable.hg
# make world && make install

This will build and install the Xen hypervisor, its supporting tools, and a Linux kernel for dom0. DomUs can use it as well. Often, this will be all you need. However, if you want to change your kernel configuration, you can. To configure the Linux kernel, run:

# make linux-2.6-xen-config configmode=MENUCONFIG

This will open the standard Linux kernel configurator. Configure the kernel as usual.

NOTE: You probably want to leave the dom0 8250 serial driver disabled because it conflicts with the Xen serial console. As usual, don’t forget the drivers for your boot device.

Then run:

# make linux-2.6-build
# make linux-2.6-install

This builds and installs the kernel. Now, if you are on CentOS, you probably want to make an initrd:

# mkinitrd /boot/initrd-2.6.18.8-xen.img 2.6.18.8-xen

NOTE: There is a bug in early releases of RHEL 5.3 that causes problems with this. See

https://bugzilla.redhat.com/show_bug.cgi?id=488991 for details. The solution is to add --allow-missing to the mkinitrd command line, thus: # mkinitrd /boot/

initrd-2.6.18.8-xen.img 2.6.18.8-xen --allow-missing.

Now, you need to fix /boot/grub/menu.lst. Add a stanza like this, but remember to use appropriate devices, paths, and possibly filenames:

title Xen.org 2.6.18.8-xen-3.3
        root (hd0,0)
        kernel /boot/xen-3.3.gz
        module /boot/vmlinuz-2.6.18.8-xen ro root=/dev/md0
        module /boot/initrd-2.6.18.8-xen.img

Reboot and enjoy your new Xen installation.

Compile-Time Tuning

That’s the quick and easy way to build Xen, but the basic compilation with make world is just the beginning. Compilation represents the first opportunity we have to configure Xen, and there’s a lot more that we can do with it now that we’ve had some practice.

Most of the compile-time tuning can be done by twiddling variables in Config.mk, at the top level of the Xen source tree. This file is fairly extensively commented and amenable to editing—take a look. You’ll find that there’s a brief section where you can decide which optional Xen bits to build.

We usually turn on all of the optional components except for the Virtual Trusted Platform Module (VTPM) tools, leading to a section like this:

XENSTAT_XENTOP     ?= y
VTPM_TOOLS         ?= n
LIBXENAPI_BINDINGS ?= y
XENFB_TOOLS        ?= y
PYTHON_TOOLS       ?= y

NOTE: Xen’s VTPM tools are interesting. They’ve been a subject of heavy development, they

have some interesting implications for signed code, and there’s the looming specter of DRM, but we just haven’t gotten into them. If you decide to build them, you can add

virtual TPMs to domains via the vtpm= option in the domain configuration.

If you’re having trouble (trust us, you probably will at some point), it would be a good idea to make a debug build. To do that, set the DEBUG variable at the top of the file:

DEBUG ?= y

Don’t worry: Xen will not run in debug mode unless you specifically instruct it to do so at runtime.

These optional Xen components have a bunch of undocumented dependencies, some of which aren’t checked for by the Makefiles. In particular, the LIBXENAPI_BINDINGS demand libxml2 and curl or the -devel versions of these packages, if you’re using a Red Hat derivative.

Also, if something doesn’t work when you build the tools, it would probably be a good idea to avoid running make world again because that takes a while. Most likely, you can get by with just make tools.

Alternate Kernels (Dom0 and DomU)

The default Xen Makefile will build a single kernel that can be used in both the dom0 and domU. If saving memory is a high priority, you can build a separate kernel for each. These kernels will each have a reasonable set of configuration options: minimal for the domU, modular for the dom0. Specify the KERNELS variable on your make command line:

# make KERNELS="linux-2.6-dom0 linux-2.6-domU"

The primary reason to do this, of course, is so that you can strip all the non-Xen device drivers out of the domU kernel. This saves memory and—if you happen to be testing a lot of kernels—compile time.

paravirt_ops Dom0

To understand why paravirt_ops gets treated as a separate piece, we have to recall that a lot of the early Xen development took place before virtualization went mainstream. The Xen developers, to paravirtualize the Linux kernel, made sweeping changes that proved to be difficult to merge with mainline kernel development.

paravirt_ops is a generic solution to this problem. It’s a kernel-level framework for adding code to enable Linux to run under various hypervisors, including Xen. The idea is that, by making these interfaces part of the official kernel, we can make Xen less invasive and easier to maintain.

Xen has supported paravirt_ops domUs since version 3.1, and the official Linux kernel has had domU support since version 2.6.23 for i386 and since version 2.6.26 for x86_64. Unfortunately, the kernel.org kernel, as of this writing, only has guest support.

But there is light at the end of the tunnel. With the latest patches from Jeremy Fitzhardinge’s paravirt_ops dom0 work and a Xen 3.4 hypervisor, it is, in fact, possible to run a paravirt_ops dom0 based on Linux kernel version 2.6.30.

These directions represent a snapshot from a very long development process. They work for us today. URLs may change. The status of the software certainly will. With that in mind, though, here’s how we set up a functioning paravirt_ops dom0.

First, you’re going to need some development packages. This time we’re using the Debian package names:

  1. mercurial
  2. build-essential
  3. libncurses5-dev
  4. gawk
  5. openssl
  6. xorg-dev
  7. gettext
  8. python-dev
  9. gitk
  10. libcurl4-openssl-dev
  11. bcc
  12. libzlib-dev
  13. libxml2-dev

Next, check out Xen-unstable with Mercurial. We warned you that this stuff is still in development.

# hg clone http://xenbits.xensource.com/xen-unstable.hg
# cd xen-unstable.hg
# make xen
# make install-xen
# make tools
# make install-tools

Then check out the current Linux patches from Jeremy Fitzhardinge’s git repo:

# git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
# cd linux-2.6-xen
# git checkout origin/push2/xen/dom0/master -b push2/xen/dom0/master

Configure the kernel. We copied the Ubuntu configuration to .config and used that as a base.

# cp /boot/config-2.6.26-11-server .config
# make menuconfig

Since we’re building a paravirt_ops dom0, make sure to turn on the appropriate support:

Processor type and features
    -> Paravirtualized guest support
........-> Enable Xen privileged domain support

Be sure to turn on the Xen block device frontend support, under:

Device Drivers
    -> Block devices

Next, build the kernel.

# make
# make modules_install install
# depmod 2.6.30-tip
# mkinitramfs -o /boot/initrd-2.6.30-tip.img 2.6.30-tip

Add /proc/xen to fstab and mount it so that tools like xend will be able to communicate with the hypervisor:

none /proc/xen xenfs defaults 0 0

Create a GRUB entry to boot your new Xen paravirt_ops dom0:

title Xen 3.4 / Ubuntu 8.10, kernel 2.6.30-tip
kernel /boot/xen-3.4.gz
module /boot/vmlinuz-2.6.30-tip root=/dev/sdb2 ro console=tty0
module /boot/initrd-2.6.30-tip.img

Make sure that these are appropriate values for your setup, of course. That’s all there is to it.

paravirt_ops DomU

“But what,” you ask, “is all this about using a kernel.org kernel in a domU?” If you just want to make your own domU kernel, this is a much less involved process, supported without out-of-tree patches since version 2.6.23. All of these directions are presented from within a domU that boots using PV-GRUB or PyGRUB—no intervention from the dom0 administrator should be necessary.

First, download the kernel source you prefer:

# wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.29.3.tar.bz2

Next, install the packages normally required to build the kernel. This example is for Debian, but it should be easy enough to find out what packages your favorite distro needs to build the kernel.

# apt-get install build-essential libncurses5-dev

Untar and configure the kernel. Personally, we like menuconfig, but that’s just a matter of taste:

# tar -jxf linux-2.6.29.3.tar.bz2
# cd linux-2.6.29.3
# make menuconfig

Don’t forget to enable Xen support:

-> Processor type and features
  -> Paravirtualized guest support
    -> XEN

Don’t forget your network driver:

-> Device Drivers
  -> Network device support
    -> XEN_NETDEV_FRONTEND

or your disk driver:

-> Device Drivers
  -> Block devices
    -> XEN_BLKDEV_FRONTEND

Xenfs, which allows you to access the XenBus, is sometimes useful:

-> Device Drivers
  -> XENFS

Then customize to your heart’s content. Remember, you can remove support for just about all hardware now. We also leave out the balloon driver. RAM is cheap, and we like having definite memory allocations.

Now, make the kernel as usual:

make -j4 ; make install modules_install

Make your initrd as per the usual kernel build. Since we’re using Debian for this example, that means using mkinitramfs. If you compiled xenblk as a module, make sure to include it.

mkinitramfs -o /boot/initrd-2.6.29.3.img 2.6.29.3

Set up GRUB as you normally would:

title kernel.org paravirt DomU
  root (hd0,0)
  kernel /boot/vmlinuz-2.6.29.3 root=LABEL=DISK1 ro
  initrd /boot/initrd-2.6.29.3.img

One last thing before you reboot: Note that the device name for your console will be hvc0, for hypervisor console. This takes the place of the Xenspecific xvc0. If your distro doesn’t do so already, you probably want to set up the domain to start a getty on hvc0. Now, simply restart your domain (halt it and start it up if you are using PyGRUB) and enjoy your modern kernel.

# uname -a
Linux sebastian.xen.prgmr.com 2.6.29.3 #1 SMP Tue May 12 06:32:52 UTC 2009 x86_64 GNU/Linux 2009

The Xen API: The Way of the Future

The Xen API is an XML-RPC interface to Xen that replaces the old interface used for communication with the hypervisor. It promises to provide a standard, stable interface so that people can build Xen frontends without worrying about the interface changing out from under them. It also extends the previous Xen command set so that more of Xen’s functionality can be harnessed in a standardized tool.

In current versions of Xen, the API is an optional component, but that shouldn’t deter you from using it; the most recent Citrix Xen Server product, for example, relies on the API exclusively for communication between the administration frontend and the virtualization host.

The Xen API is enabled by setting the LIBXENAPI_BINDINGS flag at the top of Config.mk:

LIBXENAPI_BINDINGS ?= y

When you’ve built Xen with support for the Xen API, the use of the API is controlled by the (xen-api-server) directive in /etc/xen/xend-config.sxp.

(xen-api-server ((9363 none) (unix none)))

This directive turns on the API server and specifies how to connect to it. Each list in parentheses is a connection method. In this case, we’re using TCP port 9363 and a local Unix socket, each with no authentication whatsoever.

To specify that we want to authenticate using PAM, we can modify this configuration a bit:

(xen-api-server ((9363 pam '192.0.2.*'))

Ordinarily, even when developing a Xen client, you won’t need to interact with the Xen API at a low level. Bindings exist for most of the popular languages, including C and, of course, Python. The Xen.org API documentation, accessible from http://wiki.xensource.com/xenwiki/XenApi/, is the last word on the subject.

Managing Memory with the Balloon Driver

Moving on from compile time and installation issues to the serious day-to-day business of running Xen, we encounter the problem of memory. As we’ve mentioned, most Xen installations are limited in practice by physical memory.

Xen expends a great deal of effort on virtualizing memory; its approach is one of the defining features of paravirtualization, and it usually “just works,” on a level low enough to ignore completely. However, it sometimes can benefit from a bit of attention by the administrator.

We’ve been dancing around the subject of memory oversubscription for a long time, and we’d better come clean: It is possible to assign a dynamic amount of memory to a domU, but we don’t do it because it’s not suitable for our virtual private server model. Also, the developers have historically been leery about recommending it for production. However, it does exist, and there are some good reasons to use it, which we’re sure you can imagine.

Xen’s control over reallocating memory to virtual machines is somewhat indirect. First, the value in the domU config file for memory is a maximum— conceptually, the amount of memory that’s physically accessible to the virtual machine. The amount of memory in the config file is what the kernel sees at boot. Adding more would require a reboot. This is being worked on, mostly from the direction of Linux’s memory hotplug. We confidently expect someone to provide a patch soon.

Within this inflexible maximum, Xen can reduce this memory using the balloon driver, which is nothing but a module that sits in the domU and inflates to consume memory, which it then hands back to the hypervisor.

Because the dom0 is also a Xen domain, it also can have a memory balloon, which Xen uses to reclaim dom0 memory for allocation to domUs.

NOTE: We favor setting the dom0 memory directly with the dom0_mem boot option, which

actively hides memory from the dom0. By setting (dom0-min-mem 0) and (enable-dom0-ballooning no) in xend-config.sxp, we can ensure that dom0 doesn’t balloon out,

and thus has a consistent memory reservation.3

You can use xm to manually adjust the amount of memory used by the balloon:

# xm mem-set sebastian 112

The domain treats this as a target, giving memory to the balloon as it becomes free.4 Thus, it’s possible that a heavily loaded domain will take a while to give up memory to the balloon.

You can see the effect of the balloon in the list of VMs:

# xm list sebastian
Name                                ID Mem(MiB) VCPUs State   Time(s)
sebastian                           71      111     1 -b----     38.4

You can also see balloon-related information from within the domU via /proc/xen/balloon:

# cat /proc/xen/balloon
Current allocation:   114688 kB
Requested target:     114688 kB
Low-mem balloon:       24576 kB
High-mem balloon:          0 kB
Driver pages:            136 kB
Xen hard limit:          ??? kB

NOTE: The balloon is pretty aggressive; it can cause an out-of-memory condition in the domU. Use it with caution.

PCI Forwarding

You can allow a domU to access arbitrary PCI devices and use them with full privileges. Of course, there’s no such thing as a free lunch; Xen can’t miraculously duplicate PCI hardware. For a domU to use a PCI device, it has to be hidden from the dom0 and not forwarded to any other domUs.

Figure14-1.jpg

Figure 14-1: Xen PCI device forwarding

As Figure 14-1 shows, PCI forwarding uses a client/server model in which the pcifront driver runs in the domU and communicates directly with the pciback driver, which binds to the PCI device and hides it from the dom0.

First, consider the device that you want to forward to the domU. The test machine that we’re sitting in front of appears to have seven (!) USB controllers, so we’ll just take a couple of those.

Use lspci to determine bus IDs:

# lspci

00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB
UHCI #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB
UHCI #5 (rev 02)
00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2
EHCI #2 (rev 02)

(&c.)

We’ll forward 00:1a.1 and 00:1a.7, which are the second of the USB1 controllers listed and the USB2 controller. Your device names will most likely vary from those in this example.

If pciback is compiled into the kernel, you can boot the dom0 with a pciback.hide option on the kernel command line. For these two controllers, the option would look like this:

pciback.hide=(00:1a.1)(00:1a.7)

If pciback is a module, it’s a little more difficult. We need to detach the PCI device from its driver and attach it to the pciback passthrough.

# insmod pciback hide=(00:1a.1)(00:1a.7)
# echo -n 00:1a.1 > /sys/bus/pci/drivers/uhci_hcd/unbind
# echo -n 00:1a.1 > /sys/bus/pci/drivers/pciback/new_slot
# echo -n 00:1a.1 > /sys/bus/pci/drivers/pciback/bind

Now put these devices into the domU config file:

pci = [ '00:1a.1', '00:1a.7' ]

On the next domU boot, these USB controllers should appear and be available to the native drivers in the domU. Hardware devices on platforms without an IOMMU can DMA to arbitrary memory regions. This can be a security problem if you’re giving PCI access to arbitrary domains. The moral is to treat all domains with access to the PCI bus as privileged. Make sure you can trust them.

GRUB Configuration

Of course, we’ve dealt with GRUB in passing because it’s one of the basic prerequisites for Xen. However, there are a few more aspects of GRUB that are worth mentioning in depth. A fair number of Xen’s behavior knobs can be tweaked in GRUB at boot time by adjusting the command-line parameters passed to the hypervisor.

For example, the already-mentioned dom0_mem parameter adjusts the amount of memory that Xen allows the dom0 to see:

kernel /boot/xen.gz dom0_mem=131072

To keep the system from rebooting if you have a kernel panic, which happens more often than we would like, especially when trying to get machines initially set up, add noreboot to the kernel line:

kernel /boot/xen.gz dom0_mem=131072 noreboot

as well as panic=0 to the Linux module line:

module /boot/vmlinuz-2.6.18-53.1.21.el5xen panic=0

This is, of course, in addition to the plethora of options supported by the Linux kernel, which you can then add to vmlinuz’s module line as you see fit.

The Serial Console

One other important GRUB-related task is setting up your serial console. As mentioned, we consider the serial console to be the gold standard for console access to any sort of server. It’s much simpler than any sort of graphical interface, easy to access with a variety of devices, and is the output most likely to provide useful information when the machine is crashing. Furthermore, because of the client/server architecture inherent in the system, anything that a crashing machine manages to print goes to another, physically separate machine, where it can be analyzed at leisure.

Xen comes with miniterm, a minimal serial client for this sort of thing, in case you don’t have access to a serial client. This is unlikely, but the client is tiny, so why not?

Miniterm is in the tools/misc/miniterm subdirectory of the Xen source tree. If you’ve built all the tools with Xen, it’ll already be built and possibly even installed; if not, you can simply type make in that directory and run the resulting executable.

Enabling Serial Output

There are four components that need to have their output redirected to the serial port: GRUB, Xen, the Linux kernel, and Linux’s userland. Each of the first three is a simple matter of adding a directive to GRUB’s menu.lst.

First, near the top of the file, add these lines:

serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout 10 serial console

Edit the Xen kernel line to tell the hypervisor to use the first serial port for output:

kernel /boot/xen.gz-2.6.18-53.1.21.el5 console=com1 com1=115200,8n1

Tell the Linux kernel to print its messages on ttyS0:

module /boot/vmlinuz-2.6.18-53.1.21.el5xen ro root=/dev/md0
console=ttyS0,115200n8

Finally, edit /etc/inittab and add a line like the following:

7:2345:respawn:/sbin/agetty 115200 ttyS0

You may also want to add ttyS0 to /etc/securetty so that root will be able to log in, after the manner of a traditional console.

The Xen Hypervisor Console

Xen adds another layer to the serial console by using it to access extra hypervisor features. First, break into the hypervisor console by pressing CTRL-A three times on the serial console. This won’t work on the VGA console. You’ll get a (XEN) prompt.

When you’re in the hypervisor console, there are several useful (or at least interesting) commands you can give Xen. Try typing h for help or one of the informational commands, like m. You can also crash the machine, reboot it, or dump various pieces of information. Poke around and try it.

To exit the hypervisor console, type CTRL-A three more times.

Xen and LILO

This section only applies to the real dinosaurs out there, but we sympathize. In keeping with the feeling that you have vanished into the mysterious past, we will present this example using Xen 3.0.

If you’re dead set on using LILO, rather than GRUB, you will be pleased to learn that it is possible. Although it’s generally thought that LILO’s lack of an equivalent to GRUB’s module directive makes it impossible for it to boot Xen, it’s possible to get around that by combining the hypervisor, dom0 kernel, and initrd into one file using mbootpack.

Consider the following entry in grub.conf:

title slack-xen
        root (hd0,0)
        kernel /boot/xen.gz
        module /vmlinuz-2.6-xen ro root=/dev/hda1 ro
        module /initrd-2.6.18-xen.gz

It loads the hypervisor, xen-3.0.gz, as the kernel then unpacks vmlinuz-2.6- xen and initrd.gz into memory. To combine these files, first decompress:

# cd /boot
# gzcat xen-3.0.gz > xen-3.0
# gzcat vmlinuz-2.6-xen0 > vmlinux-2.6-xen0
# gzcat initrd.gz > initrd.img

Note the change from vmlinuz to vmlinux. It’s not important except that it keeps you from overwriting the kernel at the beginning of the gzcat process.

Then combine the three files using mbootpack:

# mbootpack -o vmlinux-2.6-xen.mpack -m vmlinux-2.6-xen0 -m initrd.gz
  -m initrd.img xen3.0

The grub.conf entry then becomes a lilo.conf entry:

image=/boot/vmlinux-2.6-xen.mpack
  label=xen
  root=/dev/ram0

Finally, run the lilo command.

# /sbin/lilo</pre.

==The Virtual Framebuffer==
For as much as purists would like to claim that all administration should be
done via serial port, there’s something to be said for all this newfangled
graphical technology that we’ve been using for, oh, around the last 25 years.
Xen makes a concession to these forward-thinking beliefs by including a
facility for a ''virtual framebuffer''.

You will need to edit your ''Config.mk'' file to build the VFB:

<pre>XENFB_TOOLS ?= y

At this point you’ll also need libvncserver and libsdl-dev. Install them in your chosen way. We installed CentOS’s SDL-devel package and installed libvncserver from source. Then we built Xen and installed it in the usual way.

To actually use the framebuffer within a domain, you’ll need to specify it in the config file. Recent versions of Xen have improved the syntax somewhat. The vfb= option controls all aspects of the virtual framebuffer, just as the vif= and disk= lines control virtual interfaces and virtual block devices. For example:

vfb = [ 'type=vnc, vncunused=1' ]

Here we specify a VNC VFB and tell the VNC server to listen on the first unused port that’s over the given number. (We go into more detail on the options available in Appendix B.) Or, if you’re feeling adventurous, there’s the SDL version:

vfb = [ 'type=sdl' ]

Simple.

Use of the XenStore for Fun and Profit

The XenStore is the configuration database in which Xen stores information on the running domUs. Although Xen uses the XenStore internally for vital matters like setting up virtual devices, you can also write arbitrary data to it from domUs as well as from dom0. Think of it as some sort of interdomain socket.

This opens up all sorts of possibilities. For example, domains could, in theory, negotiate among themselves for access to shared resources. Or you could have something like the talk system on the shared UNIX machines of yore—multiuser chat between people running on the same host. You could use it to propagate host-specific messages, for example, warning people of impending backups or migration. For the most part, though, such applications remain to be written.

It’s a little inconvenient to interact with the XenStore manually because no one’s gotten around to providing a handy shell-style interface. In the meantime, we have to make do with tools that interrogate single keys.

To look at the XenStore, you can use the xenstore-list command. Here’s a shell script from the Xen wiki that dumps keys from the xenstore recursively with xenstore-list:

#!/bin/sh
function dumpkey() {
   local param=${1}
   local key
   local result
   result=$(xenstore-list ${param})
   if [ "${result}" != "" ] ; then
     for key in ${result} ; do dumpkey ${param}/${key} ; done
   else
     echo -n ${param}'='
     xenstore-read ${param}
   fi
}

for key in /vm /local/domain /tool ; do dumpkey ${key} ; done

You’ll see that we have three hard-coded top-level keys: vm, local/domain, and tool. These each have a well-defined purpose to the hypervisor: vm stores domain information by UUID; local/domain stores domain information by ID (one might say that vm exports domain data in a form suitable for migration, and local/domain stores it for local use); and tool stores toolspecific information.

Poke around, look at the keys and how they map the information that you already know about the domain from other sources, like xm list --long.

For example, to get the memory usage target for the domain, run:

# xenstore-read /local/domain/15/memory/target
1048576

Many of the keys in the XenStore are also writable. Although we don’t recommend adjusting memory usage by writing to the XenStore, see the next section for an example of interdomain communication via writable XenStore keys.

Automatically Connecting to the VNC Console on Domain Boot

One neat feature of the Xen LiveCD is that Xen domains, when started, will automatically pop up a VNC window when they’ve finished booting. The infrastructure that makes this possible is a script in the domU, a listener in the dom0, and the XenBus between them.

The script in the domU, vnc-advertiser, fires off from the domU startup scripts and waits for an Xvnc session to start. When it finds one, it writes to the XenStore:

xenstore-write /tool/vncwatch/${domid} ${local_addr}${screen}

In the dom0, a corresponding script watches for writes to the XenStore. On the LiveCD, it’s named vnc-watcher.py. This script is a good example of general-purpose uses for the XenStore, so we’ve copied it wholesale here, with verbose annotations:

#!/usr/bin/env python
###
# VNC watch utility
# Copyright (C) 2005 XenSource Ltd
#
# This file is subject to the terms and conditions of the GNU General
# Public License. See the file "COPYING" in the main directory of
# this archive for more details.
###
# Watches for VNC appearing in guests and fires up a local VNC
# viewer to that guest.
###

# Import libraries necessary to interact with the xenstore. Xswatch
# watches a xenstore node and activates a script-defined function
# when the node changes, while xstransact supports standard read and
# write operations.

from xen.xend.xenstore import xswatch
from xen.xend.xenstore.xstransact import xstransact
from os import system

def main():
   # first make the node:
   xstransact.Mkdir("/tool/vncwatch")
   xstransact.SetPermissions("/tool/vncwatch",
                             { "dom" : 0,
                               "read" : True,
                               "write" : True })
   active_connections = {}

# The watchFired method does the actual work of the script. When the
# watcher notes changes to the path "/tool/vncwatch/", it calls
# watchFired with the path (and arguments, which are unused in this
# script).

   def watchFired(path, *args, **nargs):
       if path == "/tool/vncwatch":
           # not interested:
           return 1

# If we reach this point, something's changed under our path of
# interest. Let's read the value at the path.

       vncaddr = xstransact.Read(path)
       print vncaddr

# When the vnc-advertiser notices that Xvnc's shut down in the domU,
# it removes the value from the xenstore. If that happens, the
# watcher than removes the connection from its internal list (because
# presumably the VNC session no longer exists).

       if vncaddr == None:
           # server terminated, remove from connection list:
           if path in active_connections:
               active_connections.remove(path)
       else:
           # server started or changed, find out what happened:
           if (not active_connections.has_key(path)) or
              active_connections[path] != vncaddr:

# Recall that the vnc-advertiser script writes ${domid}
# ${local_addr}${screen} to the patch /tool/vncwatch/. The watcher
# takes that information and uses it to execute the vncviewer command
# with appropriate arguments.

        active_connections[path] = vncaddr system("vncviewer
-truecolour " + vncaddr + " &") return 1

# Associate the watchFired event with a watcher on the path
# "tool/vncwatch"

   mywatch = xswatch.xswatch("/tool/vncwatch", watchFired)
   xswatch.watchThread.join()

if __name__ == "__main__":
   main()

===

There are a couple of other sections that we would have loved to include here, but that aren’t ready as of this writing, for example, the ongoing open source efforts to build an Amazon EC2 clone or the high-availability work being done by Project Kemari.

Anyway, please visit our website (http://prgmr.com/xen/) for more on the cool yet frightfully everyday things that we do with Xen.

Also, if you’ve broken your system trying to upgrade Xen from source, there’s no better time than the present to take a look at the next chapter.

Footnotes

1Or at least, anyone who hasn’t thrown this book out the window must be extremely good at filling in vague directions.
2We’ve also included a more Debian-centric set of compilation instructions in “paravirt_ops Dom0” on page 203 and “paravirt_ops DomU” on page 205.
3 In newer versions of Xen, you really only need (enable-dom0-ballooning no), but that has no effect on older versions. Before the enable-dom0-ballooning option was enabled, setting dom0-min-mem to 0 would disable ballooning. Really, you could get away with just setting dom0-min-mem to 0; I tested it after embarrassing myself on the xen-devel list, and it works, but (enable-dom0-ballooning no) is nice and clear, and this sort of thing is important enough to specify twice.
4Although Linux does its best to keep memory in use at all times, it’ll give memory to the balloon rather than using it for buffers or cache.

Navigation

Previous Chapter | Next Chapter