XEN FAQ Site

[이누기]

http://wiki.xensource.com/xenwiki/XenFaq

1. General Information
   1. What is Xen?
   2. How is Xen licensed?
   3. Which OSes run on Xen?
   4. Does Xen support Microsoft Windows?
   5. Does Xen run on laptops?
   6. Which architectures does Xen support?
   7. 32bit and 64bit
2. Booting Xen
   1. I get an error from Xen... (console output starting "(XEN)")
   2. How do I hide a pci device from dom0?
   3. Error about root device still mounted when it's not mounted, zombie domU that can't be killed, domU hangs under heavy I/O (e.g disk) access
3. Networking Issues
   1. What is veth, vif or xenbr0?
   2. Why can't I ssh into or ping a newly created domain?
   3. Why does my new domain receive no network traffic until after it initiates an outgoing connection?
   4. How do I fix MTU issues resulting in "Received packet needs 8 bytes more headroom" in dmesg or /var/log/message output ?
   5. TCP and UDP checksum errors, ping but nothing else, ipsec tunnels don't form, DNAT translation doesn't work
   6. Xen and Shorewall
   7. Too many vethX and vif0.X
   8. I can't use more than 3 network interfaces in domU
4. Running Linux on Xen
   1. A large warning message is displayed when I boot Linux, saying something about '/lib/tls'. What should I do?
   2. Lots of programs crash with 'Segmentation fault' when I boot my system. What's going wrong?
   3. I can not export block device as /dev/sdaX in DomU
   4. When I start a 9th domain using a loop-back file as its virtual disk why do I get "Error: vbd: Segment not found: uname=file:/path/to/image"?
   5. I'm trying to get X working in domain 0 but the server seems to hang (or domain 0 panics). What's going on?
   6. I get "Badness in local_bh_enable at kernel/softirq.c" messages, why is this?
   7. How do I load and run an initial ram disk (initrd)?
   8. The initrd file that came with my distro doesn't work well with Xen. How do I edit it?
   9. When I use the balloon driver, why do tools like 'top' and 'free' not show a change in memory usage?
   10. I increased the memory (RAM) in a domain and rebooted it, but it still shows the old memory value?
   11. Booting a guest complains about trying to use an already-mounted block device, but it isn't mounted?!
5. Building Linux for Xen
   1. How do I configure a Linux domain 0 kernel for my hardware?
   2. Why does my build fail with "error: isa_virt_to_bus_is_UNSUPPORTED undeclared"?
   3. How do I build Xen from source with a more recent kernel?
6. Linux Distributions
   1. What Debian packages do I need for Xen?
   2. Why does Fedora Core 3 stop working after printing the line "Freeing unused kernel memory: ..."?
   3. Why can't I save VM instances or migrate with Fedora Core 3 or Debian Sarge? I get a connection refused error from twisted referring to port 111.
   4. How do I fix the 4gb seg fixup messages in my syslog?
7. Using XEN
   1. How do I connect to or detach from a console?
   2. In 3.0, how can I expose a console on a TCP port like in 2.0
   3. In DomU, how can I use 3D graphics.
8. Help With Xen
   1. Where do I find documentation on Xen?
   2. What support is available for Xen?
   3. Where and how can i report a bug?
   4. Where and how can I submit a patch?

1. General Information

1.1. What is Xen?

Xen is a virtual machine monitor (VMM) for x86-compatible computers. Xen can securely execute multiple virtual machines, each running its own OS, on a single physical system with close-to-native performance.

1.2. How is Xen licensed?

Xen is Open Source, and is released under terms of the GNU General Public License. Operating systems or other applications written to use Xen's hypercall interface are not derived works of Xen, hence may be licensed differently.

1.3. Which OSes run on Xen?

To achieve such high performance, Xen requires that OSes are ported to run on it. See the Current status of OS ports

1.4. Does Xen support Microsoft Windows?

The paravirtualized approach we use to get such high performance has not been usable directly for Windows to date. However Xen 3.0 added Intel VT-x support to enable the running of unmodified guest operating systems, including Windows XP & 2003 Server, using hardware virtualization technology. Xen 3.0.2 and later support AMD Pacifica technology as well. Check to see if your CPU is among the list of HVM Compatible Processors, and if your motherboard is among the list of HVM Compatible Motherboards.

(Note: This does not necessarily mean, that just about _any_ OS runs in HVM Mode! Reports vary about getting *BSD to work in HVM mode, on the Mailing List there's only 1 success for OpenBSD (http://www.openbsd-france.org/ml/archives/msg02494.html), no success message for the others, but some problem reports)

1.5. Does Xen run on laptops?

Xen will typically run on laptops, but there's currently no support for APM or ACPI, hence you'll experience reduced battery life and no suspend/resume. We hope to add ACPI support in the future, exploiting Linux's existing support.

1.6. Which architectures does Xen support?

Xen currently runs only on the x86 architecture, requiring a "P6" or newer processor (that's any Intel or AMD x86 CPU purchased in the last five years). Multiprocessor machines are supported, including basic support for Hyper-Threading (SMT). Support for x86/64 is available since Xen 3.0. In addition a port to the IA64 architecture is approaching completion. We hope to add other architectures such as PPC and ARM in due course.

1.7. 32bit and 64bit

The 32bit and 64bit questions are many and more complex than you might think. Things can be 32bit, 32bit PAE or 64bit. There is the hardware, the xen hypervisor, the dom0 kernel and userland and the domU kernel and userland.

	32bit Hypervisor	64bit Hypervisor
32bit PV DomU	Yes	Yes (starting from Xen 3.1), kernel must be PAE
64bit PV DomU	No	Yes
32bit HVM DomU	Yes	Yes
64bit HVM DomU	No	Yes

2. Booting Xen

2.1. I get an error from Xen... (console output starting "(XEN)")

(XEN) Initial guest OS requires too much space
(XEN) (8MB is greater than 0MB limit)

Check that you are correctly specifying the dom0_mem command line option for starting Xen. Note that: dom0_mem=512M should work in recent versions while older versions such as 2.0.5 require the memory specification to be in kilobytes, e.g. dom0_mem=524288.

(Note: you need to leave some memory for Xen, so the above assumes you have more than 512M total in the system; if you still have problems, reduce the amount further).

2.2. How do I hide a pci device from dom0?

In Xen 2.x, you can add the physdev_dom0_hide parameter (see XenBooting for more parameters) to hide one or more pci devices to Dom0, so you can affect them to domU.

Pci slots MUST be formatted like this:

• (nn:nn.n)

so

• (03:06.1) is correct

• (03:6.1) is NOT correct

You can get the pci bus address on a Linux system by using the lspci command (only the last bits are relevant):

lspci | grep Ethernet
0000:02:03.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
0000:02:03.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (Copper) (rev 01)

Then:

• to hide the second network interface from dom0, you can then append physdev_dom0_hide=(02:03.1) to your kernel /boot/xen-2.0.gz.

• to hide multiple pci devices, simply concatenate all the pci slots address like this: physdev_dom0_hide=(02:03.0)(02:03.1)

If everything went ok, you should see the following lines after a reboot:

(XEN) Hiding PCI device 02:03.0 from DOM0
(XEN) Hiding PCI device 02:03.1 from DOM0

2.3. Error about root device still mounted when it's not mounted, zombie domU that can't be killed, domU hangs under heavy I/O (e.g disk) access

This is an unresolved problem with Xen 3.0.

You may try to pass nousb to dom0 kernel command line, or pass ignorebiostables, or try to disable software IRQ affinity for 1850/2850 systems.

• Discussion in xen-users mailing list

3. Networking Issues

3.1. What is veth, vif or xenbr0?

You should read XenNetworking http://wiki.xensource.com/xenwiki/XenNetworking

3.2. Why can't I ssh into or ping a newly created domain?

In the default configuration we rely on the Linux bridge-utils in domain 0 to set up virtual networking. After you've created a new domain (e.g., domain 1) you should be able to run ifconfig in domain 0 and see an interface with a name like vif1.0; you should also be able to check that bridging is working by typing brctl show xen-br0. Finally, you can check the IP confiuration in the new domain by logging into it via the console (xm console) and running standard tools such as ifconfig and route.

3.3. Why does my new domain receive no network traffic until after it initiates an outgoing connection?

This is an issue that occurs under the following circumstances:

• You do not specify the domain's MAC address, causing a random MAC address to be selected at domain creation time
• The upstream router has a local ARP cache

When a domain is destroyed, the host system's ARP cache is purged of addresses for the defunct virtual NIC. If the domain is recreated it is likely to be allocated a different random MAC address. This is no problem for the host machine, but the external switch/router still knows the "old" MAC address. The switch/router requires an outbound packet to the outside world to implicitly eradicate the old ARP-cache entry. This is not really Xen's fault at all, just a property of the implementation of ARP resolution.

Fixes: Either manually declare the MAC address in your VM config file, or upgrade to the 2.0-testing tree where the problem is fixed. This fix will also find its way into the forthcoming 2.0.4 maintenance release.

3.4. How do I fix MTU issues resulting in "Received packet needs 8 bytes more headroom" in dmesg or /var/log/message output ?

See: http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00226.html

You can work around the bug by reducing the MTU of eth0 in the dom0 e.g. "ifconfig eth0 mtu 1400". Put this in your networking scripts (e.g.: /etc/sysconfig/network/ifcfg-eth0 for RH/RPM systems or /etc/network/interfaces for Debian/Deb systems). See: http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/ref-guide/s1-networkscripts-interfaces.html for RH syntax, 'man interfaces' for Debian.

This bug is understood and a fix has been applied to the testing tree.

It only effects dom0 kernels built with the -xen config rather than -xen0.

Basically, the -xen kernel config turns on so much stuff that the area reserved for the max possible header length is too big. This causes a netfront slow-path to be exercised that copies the SKB. Unfortunately, this path hadn't been exercised before, and guess what, it was subtly broken for checksum-offloaded packets.

3.5. TCP and UDP checksum errors, ping but nothing else, ipsec tunnels don't form, DNAT translation doesn't work

Try running the following command in each domU:

ethtool -K eth0 tx off

This command disable TX checksumming

To check checksum problems, you can use tcpdump:

tcpdump -vv -n -i eth0

Read http://lists.xensource.com/archives/html/xen-users/2006-04/msg00032.html

This may or may not work.

Patch for network-bridge

3.6. Xen and Shorewall

There is a document about configuring Shorewall in Dom0 at http://www.shorewall.net/Xen.html

http://www1.shorewall.net/XenMyWay.html can be useful also.

3.7. Too many vethX and vif0.X

By default, 8 vethX and vif0.X are created. These interfaces are "cheap" but clutter list of interfaces. If you want to allocate only the necessary number, pass netloop.nloopbacks=NUMBER parameter to kernel command line.

3.8. I can't use more than 3 network interfaces in domU

This is a limitation in Xen 3. Xen 3.1 supports 8 network interfaces.

4. Running Linux on Xen

4.1. A large warning message is displayed when I boot Linux, saying something about '/lib/tls'. What should I do?

Do as the message advises! Some modern distributions ship with a 'TLS' version of glibc that is not fully compatible with Xen. To use Xen reliably and with maximum performance you must disable the incompatible glibc. You can easily do this by moving the /lib/tls directory (e.g., mv /lib/tls /lib/tls.disabled).

Alternatively, install a Xen-specific GLIBC, see XenSpecificGlibc.

4.2. Lots of programs crash with 'Segmentation fault' when I boot my system. What's going wrong?

See the answer to the previous question. In some environments (for example, when running Xen within a VMware product) you may experience severe problems with TLS libraries. If these are so severe that you cannot even log in to your system, you can obtain a command prompt by temporarily adding the boot parameter init=/bin/bash to your Linux command line.

4.3. I can not export block device as /dev/sdaX in DomU

You have errors like

Registering block device major 8
register_blkdev: cannot get major 8 for sd
xen_blk: can't get major 8 with name sd

all you need to do is disable the entire scsi subsystem, ie:

CONFIG_SCSI=n

This allows the xen block frontend to register on device ID's normally occupied by the scsi subsystem.

4.4. When I start a 9th domain using a loop-back file as its virtual disk why do I get "Error: vbd: Segment not found: uname=file:/path/to/image"?

By default, Linux only allows up to 8 loop-back devices to be active. You can increase this number by setting max_loop=32 (etc) on the domain 0 kernel command line in Grub. You also need to create new loop[n] device nodes in /dev.

4.5. I'm trying to get X working in domain 0 but the server seems to hang (or domain 0 panics). What's going on?

It's likely your X server is trying to use agpgart, support for which is currently included only in the 2.0-testing and unstable trees. Try downloading a new tarball, or clone one of the BK repositories.

4.6. I get "Badness in local_bh_enable at kernel/softirq.c" messages, why is this?

This is fairly likely to be caused by a module compiled for native i386 rather than Xen. When building modules outside of the Xen build tree, use make ARCH=xen ....

Alternative, this may be a driver that uses interrupt en/disabling instructions directly rather than the proper API....

4.7. How do I load and run an initial ram disk (initrd)?

To specify an initrd file for domain 0 add an extra "module" line to your GRUB config, after your kernel-module line (e.g., module /boot/initrd-2.6-xen0.img). To use an initrd file for other domains, set the "ramdisk" variable in your domain configuration file (e.g., ramdisk = path/to/initrd-2.6-xenU.img).

4.8. The initrd file that came with my distro doesn't work well with Xen. How do I edit it?

There are various forms of initrd image, but the two most common formats are a gzip'ed filesystem and a gzip'ed cpio file. In either case you can take the following steps to modify a file initrd.img (when running as root):

# mkdir initrd.mnt
# gzip -d -S ".img" /path/to/initrd.img
# mount -o loop /path/to/initrd initrd.mnt OR (cd initrd.mnt && cpio -ic <../path/to/initrd)
# ...edit files in directory initrd.mnt....
# umount initrd.mnt OR (cd initrd.mnt && find . | cpio -oc --quiet >../path/to/initrd)
# gzip -9 -S ".img" /path/to/initrd

If you use cpio then you may get warnings about truncated inode numbers. These can be ignored. Some distros (notably SLES 9) have a mkinitrd that generates initrd files with garbage at the end of them. Xen will not load this initrds. You can work around this problem by gunzip'ing and gzip'ing the initrd again.

4.9. When I use the balloon driver, why do tools like 'top' and 'free' not show a change in memory usage?

The balloon driver requests memory from the system allocator which it then directly gives back to Xen. From the kernel's perspective, however, that memory is still part of the system and is being used by the balloon driver. Hence the unintuitive memory statistics.

4.10. I increased the memory (RAM) in a domain and rebooted it, but it still shows the old memory value?

You need to "re-create" the domain using the

xm create <domU>

command again. Simply rebooting the domain does not cause it to read the new value from the domain definition file.

4.11. Booting a guest complains about trying to use an already-mounted block device, but it isn't mounted?!

This is a bug to do with Xen incorrectly freeing guest resources. Take a look at the last line in /var/log/xen-hotplug.log, something like:

xenstore-read: couldn't read path /local/domain/9/vm

Take a note of that number, and run xenstore-rm backend/vbd/9, replacing 9 with the number you read. The guest machine should then boot happily.

Be careful when doing this that there are no guests effectively using the device in question, damage will probably be incurred in that case!

5. Building Linux for Xen

5.1. How do I configure a Linux domain 0 kernel for my hardware?

You should be able to edit the .config file and then just type make at the top level. For example, to use Linux's graphical config editor:

# cd linux-2.6-xen0
# make xconfig (configure however you want) (on old version you need to specify ARCH=xen after make)
# cd ..
# make

If there's already a .config present in the kernel tree we leave it well alone. If there's none present and a ./dist/install/boot/config-2.6-xen0 exists then we copy that to .config. As a last resort, we select one of the arch/xen/configs/defconfig files.

5.2. Why does my build fail with "error: isa_virt_to_bus_is_UNSUPPORTED undeclared"?

Certain drivers (primarily for ISA devices) do not work with Xen due to the restricted memory addresses that the device can access. For most of these drivers we report the error you have seen, in preference to failing weirdly at run time. We aim for Xen to support all modern x86 hardware: this restriction applies only to a few old NICs and disc controllers.

5.3. How do I build Xen from source with a more recent kernel?

Modifying a kernel source tree to run Xen is not a simple task. It might seem that one should be able to run a few patches on any kernel and make it a Xen kernel but this not the case.

Currently, the best option is to look to the distribution vendors. For instance, Fedora 7 ships with Xen 3.1.0 and kernel 2.6.20. Fedora provides source RPMS that might be of interest to anyone who wants to build their own 2.6.20 Xen kernel.

6. Linux Distributions

6.1. What Debian packages do I need for Xen?

To run Xen and a single OS from a binary release you need install no extra packages. To build Xen, XenLinux and documentation from source you will need: make, libncurses5-dev, libncurses5, gcc, libc6-dev, zlib1g-dev, python, python-dev, python-twisted, openssl, openssl-dev, bridge-utils, iproute, libcurl3, libcurl3-dev, bzip2, module-init-tools, latex, latex2html, transfig, and tgif.

6.2. Why does Fedora Core 3 stop working after printing the line "Freeing unused kernel memory: ..."?

FC3 uses the new udev system for managing device nodes in /dev. To successfully boot, and to get any console output from init, you either need to manually create some device nodes or you need to load and run a suitable initrd. The former solution requires you to mount the root filesystem and then:

# mknod /path/to/dev/null c 1 3
# mknod /path/to/dev/console c 5 1

If you instead wish to load an initrd file then you can use one provided in the /boot directory of your FC3 filesystem, or you can use the slightly-modified one that we supply. To load and run your initrd file, or to modify it, see this and this above.

You may also want to disable X by editing /etc/inittab if you do not use X, or if X is configured incorrectly and is causing your boot to fail. To do this, change "id:5:initdefault:" to "id:3:initdefault:".

6.3. Why can't I save VM instances or migrate with Fedora Core 3 or Debian Sarge? I get a connection refused error from twisted referring to port 111.

Fedora Core 3 and DebianSarge are missing /usr/lib/libcurl.so.2 (they come with libcurl.so.3.0.0 instead when updated to the latest RPMs as of 5/2/2005). This causes /usr/sbin/xfrd to fail and it is required for save (and other) operations. A workaround tried on both FC3 and Sarge is to just set up a symlink for libcurl.so.2 which points at libcurl.so.3.0.0. With Debian, you can instead try installing the libcurl2 package from oldstable, though this workaround is untested and may cause you other problems.

The problem does not occur in xen-testing, which uses curl3.

6.4. How do I fix the 4gb seg fixup messages in my syslog?

These messages are from a glibc that was built with negative GS register offsets. Xen can run with negative GS register references, but it must "trap and emulate" this condition, which is relatively resource intensive, and quite slow.

If you don't have a patched glibc, see the no-tls-direct-seg-refs patch mentioned in XenSpecificGlibc.

Most newer distributions are shipped with a "-mno-tls-direct-seg-refs" compiled glibc, but it might not be enabled by default.

Some newer distributions like the RHEL5 Beta1/Beta2 require the ld.so.conf.d trick below to refer to the patched glibc.

Some slightly older distributions like Ubuntu Dapper can be fixed with a simple rebuild of glibc: (see UbuntuDapperHowTo).

 echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf && ldconfig

Reference Link: http://lists.xensource.com/archives/html/xen-users/2006-11/msg00026.html and http://www.mail-archive.com/fedora-xen@redhat.com/msg00041.html

7. Using XEN

7.1. How do I connect to or detach from a console?

You can connect to a console directly when starting a domU:

# xm create -c your-DomU-config

This command will start a domU and connect to it's console immediately.

If you want to connect to a domU console that is already running you first have to find the domU id using:

# xm list
Name Id Mem(MB) CPU State Time(s) Console
Domain-0 0 123 0 r---- 2482.9
testXM1 30 127 1 -b--- 4.9 9630
testXM2 32 127 1 -b--- 4.7 9632
testXM3 34 128 1 -b--- 0.3 9634

Now you can connect to for example the testXM2 domU with Id 32 using:

# xm console 32

You can detach from a console using the escape character: <CTRL> ']'.

On a German keyboard try <CTRL> '5' if the above doesn't work. This works both in Putty and gnome-terminal. On a Swedish keyboard (mac running iTerm) <CTRL> 'å' works.

7.2. In 3.0, how can I expose a console on a TCP port like in 2.0

Xen 3.0 no longer exposes consoles on TCP ports to provide stronger security out of the box. If this feature is desired (and you're on an isolated network), you can achieve the same results using inetd or xinetd. The following is an xinetd service that can be enabled by placing it in /etc/xinetd.d/xen:

# default: on
# description: Xen console port
service xen1
{
        disable         = yes
        type            = UNLISTED
        socket_type     = stream
        protocol        = tcp
        user            = root
        wait            = no
        port            = 9001
        server          = /usr/sbin/xm
        server_args     = console 1
}
service xen2
{
        disable         = yes
        type            = UNLISTED
        socket_type     = stream
        protocol        = tcp
        user            = root
        wait            = no
        port            = 9002
        server          = /usr/sbin/xm
        server_args     = console 2
}

Adding entries for each domid that you want to expose (and the port you want to expose it on). In the above example, entries are only made for domain IDs 1 and 2. Any additional xinetd parameters can be used to limit where the connections come from and to implement logging.

7.3. In DomU, how can I use 3D graphics.

In a para-virtualized domain, you could give access to the graphics card by hiding the device from Dom0 and giving the DomU of choice access to this card. [Sorry, I don't know more details than this - maybe someone else can fill in more here. --Mats Petersson] The frequently asked question comes from people who want to use Windows in a HVM domain to play games or run high-end 3D graphics applications, and the answer here is "Not yet". The problem here is that the OS isn't loaded directly at address zero when it is a guest-OS (DomU), so we're lying to the OS about it's location (because most OS's don't quite understand the concept of memory that doesn't actually start at address zero). So, the OS thinks that it's been loaded at an address of, say, 0..256MB, but in reality it's been loaded at 256MB..512MB. When it then talks to the graphics cards, it will give the graphics card a physical address of somewhere between 0..256MB, which of course doesn't work for the graphics card - as it should really be reading somewhere between 256MB..512MB. So what can we do about it? There are a number of different solutions that could work: 1. Rewrite the graphics driver so that it understands Xen. Possible, but hard work, even if you have easy access to the source-code. Most graphics chip manufacturers do not have open-source drivers. 2. Use some alternative way to access the desktop, such as Remote Desktop. This doesn't necessarily work for real high-end apps or games, but it's probably better than the graphics card emulation in HVM. 3. Wait for hardware and software to support IOMMU. An IOMMU is a device that will support mapping memory addresses for a domian to some different address when that domain's hardware is being used. There is currently (high-end) server hardware that support this, but no desktop machines. Expect this to change in the future, but not very soon.

8. Help With Xen

8.1. Where do I find documentation on Xen?

There's a list of pages on this wiki which might be helpful.

8.2. What support is available for Xen?

There is both paid and volunteer support available for Xen. Commercial support is provided by Citrix Systems, Inc. (XenSource, Inc.), the proprietors of this wiki. Volunteer support is available via the Xen mailing lists and on Internet Relay Chat (IRC):

1. channel #xen on irc.oftc.net is frequented by Xen developers

2. channel ##xen ("about Xen") on irc.freenode.net is a slightly larger, unofficial channel which also provides support

A search tool is available to search all Xen mailing lists.

8.3. Where and how can i report a bug?

This information will need an upgrade, but by now, as reported by Ewan Mellor and Keir Fraser, is to:

1. Place question in xen-users mailing list, just to be sure that this is not a faq or something like that

2. If it is not resolved in xen-users mailing list, and it IS a bug, you have to:

   1. Place bug in bugzilla

   2. Flag the problem (with link to bugzilla, and all that you think is necessary) in xen-devel mailing list

8.4. Where and how can I submit a patch?

To submit a patch, put it in xen-devel mailing list, as reported by Keir Fraser, in plain-text unified diff format inline in the main body or as an attachment. Also, please add

Signed-off-by: Name Surname <email@domain>

to your email with patch, as reported by Ewan Mellor.

If there is a bug in bugzilla report the id of the bug, but by now do not expect anything from bugzilla (see above: "Where and how can i report a bug?").

XenFaq (2008-03-28 10:00:44에 SamuelThibault가(이) 마지막으로 수정)