Editing How to use OpenVZ as a XEN guest OS (for x86 platform)

1. Intro

Recently released RHEL5 kernel supports compatibility with XEN 3.0. It means that
RHEL5 kernel built with config that enables XEN can be used as guest OS in both privileged Dom0 and unprivileged DomU XEN domains. RHEL5 - based OpenVZ kernel 028stab033 also contains support for XEN compatibility. Current article reports on how to virtualize xen DomU by using xen-compatible OpenVZ kernel el5.028stab034.

Detailed description of XEN 3.0, user guides and howtos on XEN 3.0 can be found here:
http://www.xensource.com/products/xen/documentation.html. But we'll give here some common information on XEN 3.0 in purpose to make a story more clear. XEN system consists of 3 parts:

1) Xen hypervisor - lowest level part of system that virtualizes drivers and architecture dependent part of the system.

2) XenLinux kernel - guest OS that works over hypervisor in privileged domain Dom0 or unprivileged domain DomU. Privileged Dom0 domain is used for  creating, destroying and supervising of unprivileged domains (DomUs). XenLinux kernel that is running in Dom0 contains 2 sets of drivers: physical and virtual. DomU XenLinux kernel can contain the only set of virtual drivers.

3) XM Tool - userland program that is used from Dom0 for creating, destroying and supervising DomUs.

As OpenVZ is an OS level virtualization solution and do not affect drivers, it is possible to run OpenVZ within XEN DomU. Text listed below guides through XEN 3.0, XM tool and OpenVZ-XenLinux PAE kernel installation on x86. Guide assumes RHEL5 Linux is used as an OS on your hardware.

Well. Login as root. Create 2 directories:
<pre>
# mkdir xen
# mkdir openvz
</pre>

2. Installing XEN itself, Dom0 and XEN tools.

XEN 3.0 and XM tool can be installed in 2 ways: either from tarball containing prebuilt binaries or by building from sources.

- Installing from tarball

Download XEN 3.1 32 bit PAE SMP tarball from http://bits.xensource.com/ and unpack:
<pre>
# cd xen
# wget  http://bits.xensource.com/oss-xen/release/3.1.0/bin.tgz/xen-3.1.0-install-x86_32p.tgz
# gunzip xen-3.1.0-install-x86_32p.tgz
# tar -xvf xen-3.1.0-install-x86_32p.tar
</pre>
Run the install.sh script within ./dist folder:
<pre>
# ./dist/install.sh

Installing Xen from './dist/install' to '/'...
 - installing for udev-based system
 - modifying permissions
All done.
Checking to see whether prerequisite tools are installed...
Xen CHECK-INSTALL  Thu May 24 14:30:20 MSD 2007
Checking check_brctl: OK
Checking check_crypto_lib: OK
Checking check_iproute: OK
Checking check_libvncserver: unused, OK
Checking check_python: OK
Checking check_python_xml: OK
Checking check_sdl: unused, OK
Checking check_udev: OK
Checking check_zlib_lib: OK
All done.

#
</pre>
install.sh checks the presence of the required packages and installs hypervisor xen.gz and XenLinux vmlinuz-2.6.16.33-xen
to /boot and xm tool to /usr/sbin. Now update /etc/grub.conf by adding the xen entry in the following way:

/etc/grub.conf:
<pre>
title Xen Linux (2.6.16.33-xen)
        root (hd0,1)
        kernel /boot/xen.gz dom0_mem=1048576
        module /boot/vmlinuz-2.6.16.33-xen ro root=LABEL=/ console=ttyS0,115200 console=tty silencelevel=8 
</pre>
Now you can reboot the machine. Choosing the Xen Linux (2.6.16.33-xen) from grub sequence will cause to booting xen hypervisor and Dom0 running Xen Linux (2.6.16.33-xen). In case of success you'll see the usual login prompt. If kernel panic takes place during boot - reboot the machine into your usual working Linux and check if the xen related /etc/grub.conf entry is correct. If /etc/grub.conf entry is correct and xen kernel is unable to boot,- the reason is prebuild vmlinuz-2.6.16.33-xen does not contain
some drivers for your hardware, that are needed to be loaded by using initrd. In this case follow instructions below - "Building from sources".

- Building from sources

Download the package containing sources of Xen-3.1 and unpack it:
<pre>
# wget http://bits.xensource.com/oss-xen/release/3.1.0/src.tgz/xen-3.1.0-src.tgz
# gunzip xen-3.1.0-src.tgz
# tar -xvf xen-3.1.0-src.tar
</pre>
Change directory to ./xen-3.1.0-src:
<pre>
# cd ./xen-3.1.0-src
</pre>
and start building:
<pre>
# make XEN_TARGET_X86_PAE=y
</pre>
Building process will do the following:
- build hypervizor xen.gz
- download linux kernel sources 2.6.16 from kernel.org, patch them with a set of xen patches 
- from sources obtained build the vmlinuz-2.6.16.33-xen XenLinux kernel that will be able to boot both in Dom0 and DomU
- build and install xm tool

After building is complete run ./install.sh script:
<pre>
# ./install.sh
</pre>
As a result vmlinuz-2.6.16.33-xen, xen.gz will be placed to boot, xm tool will be placed to /usr/sbin. Now it is time to create initrd for vmlinuz-2.6.16.33-xen. Check that /lib/modules contain recently created directory 2.6.16.33-xen. If it is, type:
<pre>
# mkinitrd -f -v /boot/initrd-2.6.16.33-xen.img 2.6.16.33-xen
</pre>
After initrd-2.6.16.33-xen.img is succesfully created update corresponding strings in /etc/grub.conf:
<pre>
title Xen Linux (2.6.16.33-xen)
        root (hd0,1)
        kernel /boot/xen.gz dom0_mem=1048576
        module /boot/vmlinuz-2.6.16.33-xen ro root=LABEL=/ console=ttyS0,115200 console=tty silencelevel=8 
        module /boot/initrd-2.6.16.33-xen.img
</pre>
Xen 3.0 doesn't support TLS. To make xen work correct disable TLS on your machine bu renaming /lib/tls to /lib/tls.disabled:
<pre>
# mv /lib/tls /lib/tls.disabled
</pre>
Check that kernel command line options are correct (after ro ...) and reboot the machine. In grub loader menu select "Xen Linux (2.6.16.33-xen)". In case of success - machine boots into Xen's Dom0 that is running vmlinuz-2.6.16.33-xen XenLinux. Now it is time to prepare OpenVZ XenLinux kernel for DomU.

3. Installing OpenVZ XenLinux kernel.

OpenVZ XenLinux kernel can be installed in 2 ways: either from rpm containing or by
building from sources.

- install from rpm

Change directory to openvz and download rpm package with OpenVZ XenLinux kernel for x86:
<pre>
# cd
# cd openvz
# wget http://download.openvz.org/kernel/branches/rhel5-2.6.18/028stab034.1/ovzkernel-PAE-2.6.18-8.el5.xen.028stab034.i686.rpm
</pre>
Install downloaded rpm:
<pre>
# rpm -ihv ovzkernel-PAE-2.6.18-8.el5.xen.028stab033.i686.rpm
</pre>
If something goes wrong during installation the prebuilt kernel - you can build the OpenVZ XenLinux kernel from sources as described below.

- install from sources

Download rpm with OpenVZ kernel sources:
<pre>
# wget http://download.openvz.org/kernel/branches/rhel5-2.6.18/028stab034.1/ovzkernel-2.6.18-8.el5.028stab034.1.src.rpm
</pre>
Unpack downloaded rpm package:
<pre>
# rpm2cpio ovzkernel-2.6.18-8.el5.028stab034.1.src.rpm > ovzkernel-2.6.18-8.el5.028stab034.1.src.cpio
# cpio -i < ovzkernel-2.6.18-8.el5.028stab034.1.src.cpio
</pre>
Check that ./kernel-ovz.spec contains the following strings:
<pre>
...
# Whether to build the Xen kernels, disable if you want.
%define buildxen 1
...
</pre>
If "buildxen is set to 0" - update kernel-ovz.spec using text editor, set buildxen to 1.

Prepare sources for building:
<pre>
# rpmbuild -bp --define "_topdir $PWD" --define "_sourcedir $PWD" --define "_builddir $PWD" --target i686-linux kernel-ovz.spec
# cd ./ovzkernel-2.6.18/linux-2.6.18.i686
</pre>
Use xen config file:
<pre>
# cp configs/kernel-2.6.18-i686-PAE-xen.config.ovz ./.config
</pre>
Update Makefile: change the value of EXTRAVERSION from  "-prep" to something better, for example "-openvzxen":
<pre>
EXTRAVERSION = -openvzxen
</pre>
And build the OpenVZ XenLinux:
<pre>
# make oldconfig
# make
</pre>
After build is complete, - install modules and the kernel:
<pre>
# make modules_install
# make install
</pre>
Check that /lib/modules contains the directory 2.6.18-openvzxen.
And prepare initrd image for built OpenVZ XenLinux:
<pre>
# mkinitrd -f -v --omit-scsi-modules --preload=xenblk /boot/initrd-2.6.18-openvzxen.img 2.6.18-openvzxen
</pre>
OpenVZ XenLinux kernel prepared. Now OpenVZ tools are to be installed. In case you faced problem while building the kernel turn to article http://wiki.openvz.org/Kernel_build .

- install OpenVZ tools:

Now download and install latest versions of OpenVZ tools vzctl, vzquota that are necessary for starting/stopping VEs:
<pre>
# wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-3.0.16-1.i386.rpm .
# wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-lib-3.0.16-1.i386.rpm .
# wget http://download.openvz.org/utils/vzquota/3.0.9/vzquota-3.0.9-1.i386.rpm .
# rpm -Uhv --nodeps vzctl-3.0.16-1.i386.rpm vzctl-lib-3.0.16-1.i386.rpm vzquota-3.0.9-1.i386.rpm
</pre>
More information on installing OpenVZ tools can be found here: http://wiki.openvz.org/Quick_installation.

- download OpenVZ guest template

To make it possible to create VEs download one of the precreated OpenVZ templates and place it to /vz/template/cache:
<pre>
# wget http://download.openvz.org/template/precreated/fedora-core-5-i386-minimal.tar.gz
# mv fedora-core-5-i386-minimal.tar.gz /vz/template/cache/
</pre>

4. Preparing guest partition and configuring XEN.

Xen users manual http://www.xensource.com/products/xen/documentation.html describes a set of different ways of creating disk for DomU. Disk can be created using loopback or LVM or physical partition. Consider we have a standalone physical partition /dev/sda7 that will be used for creating DomU disk. Assume /dev/sda7 is empty and it contains ext3 filesystem.

Prepare disk with DomU guest Linux:
<pre>
# mount -t ext3 /dev/sda7 /mnt
# cp -a /bin /mnt
# cp -a /dev /mnt
# cp -a /etc /mnt
# cp -a /lib /mnt
# cp -a /net /mnt
# cp -a /opt /mnt
# cp -a /root /mnt
# cp -a /sbin /mnt
# cp -a /srv /mnt
# cp -a /tmp /mnt
# cp -a /usr /mnt
# cp -a /var /mnt
# cp -a /vz /mnt
# mkdir /mnt/sys
# mkdir /mnt/proc
# mkdir /mnt/mnt
# mkdir /mnt/home
</pre>
Now create configuration file for starting DomU:
<pre>
# touch /etc/xen/xmDomU
# vim /etc/xen/xmDomU
</pre>
and update xmDomU in the following way:
<pre>
# Kernel image file.
kernel = "/boot/vmlinuz-2.6.18-openvzxen"

# Optional ramdisk.
ramdisk = "/boot/initrd-2.6.18-openvzxen.img"

# Initial memory allocation (in megabytes) for the new domain.
#
# WARNING: Creating a domain with insufficient memory may cause out of
#          memory errors. The domain needs enough memory to boot kernel
#          and modules. Allocating less than 32MBs is not recommended.
memory = 256

# A name for your domain. All domains must have different names.
name = "Domain-U"

# Define network interfaces.
vif = [ '' ]

# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.
disk = [ 'phy:sda7,xvda1,w' ]

# Set root device.
root = "/dev/xvda1"

# Sets runlevel 4.
extra = "4 debug"
</pre>
Update /mnt/etc/fstab in the following way:
<pre>
/dev/xvda1              /                       ext3    defaults        1 1
</pre>
Umount /dev/sda7:
<pre>
# umount /mnt
</pre>
Preparations completed. At this point we have everything that is necessary for cretaing DomU and starting OpenVZ XenLinux within DomU. Reboot into Xen: select Xen Linux (2.6.16.33-xen) item from grub menu. After reboot machine will be running vmlinuz-2.6.16.33-xen (see above) in Dom0 over hypervisor. Follow instructions below.

5. Starting OpenVZ guest in DomU.

- starting

Start xend daemon to make xm tool work:
<pre>
# xend start
</pre>
List the set of running domains:
<pre>
[root@dhcp0-131 ~]# xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  1024     2     r-----     53.2
[root@dhcp0-131 ~]#
</pre>

Now only privileged domain Dom0 is running. Create DomU:
<pre>
[root@dhcp0-131 ~]# xm create -c /etc/xen/xmDomU
</pre>
This call culminates by showing us the DomU login prompt. Current console is now DomU console.

Open another terminal on the machine (correspond to Dom0). And list the machines:
<pre>
# xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  1024     2     r-----     75.0
Domain-U                                   1   255     1     r-----     19.5
#
</pre>
Switch back to DomU console and test OpenVZ.

DomU console:
<pre>
[root@localhost ~]# uname
Linux localhost.localdomain 2.6.18-ovzxen-4-uxen #13 SMP Tue May 22 22:22:35
MSD 2007 i686 i686 i386 GNU/Linux
[root@localhost ~]# vzctl create 134 --ostemplate fedora-core-5-i386-minimal
Creating VE private area (fedora-core-5-i386-minimal)
Performing postcreate actions
VE private area was created
[root@localhost ~]# vzlist -a
      VEID      NPROC STATUS  IP_ADDR         HOSTNAME
       134          - stopped -               -
[root@localhost ~]# vzctl start 134
Starting VE ...
VE is mounted
Setting CPU units: 1000
VE start in progress...
[root@localhost ~]# vzlist -a
      VEID      NPROC STATUS  IP_ADDR         HOSTNAME
       134          - running -               -
[root@localhost ~]# vzctl enter 134
entered into VE 134
[root@localhost /]#
...
some actions within VE 134
...
[root@localhost /]# exit
exited from VE 156
[root@localhost ~]# vzctl stop 134
Stopping VE ...
VE was stopped
VE is unmounted
[root@localhost ~]#
</pre>
It works.

- next steps

As OpenVZ works in DomU it is the best time for configuring network both in DomU and VEs. This process is described in http://www.xensource.com/products/xen/documentation.html (DomU) and http://wiki.openvz.org/Category:Networking (VEs). After this all of us will feel the full power of virtualization!

6. RHEL5 based OpenVZ in Dom0. Known problems.

OpenVZ XenLinux kernel is able to work also in Dom0. It can be tested just by updating /etc/grub.conf on the hardware node. But in this case it will be impossible to start DomUs. It is a known bug and it is related not to OpenVZ, but to RHEL5 kernel.
Hope it is fixed in next RHEL5 update. 

7. Appreciates

Thanks to Sergey Ya. Korshunoff seyko@ for starting the topic and contribution.