Creating OpenVZ LiveCD based on Centos 4.4

This article is a step by step guide explaining how to add OpenVZ functionality to Centos 4.4 LiveCD. As a result, you will have a LiveCD, that can boot in various OpenVZ kernels (rhel4-2.6.9, 2.6.18, rhel5-2.6.18, 2.6.20) and in original CentOS kernel also. After booting from the CD, you will be able to create VEs, start them, enter them - i.e. perform common OpenVZ activity.

CentOS 5 LiveCD is already announced and as soon as it will be available we'll update the article.

General actions

It is quite easy to add OpenVZ functionality to existent LiveCD. The most difficult work is already done by the original LiveCD creators. But it requires pretty much hand job. Probably some day some good guy will automatize this process. ;)

Actually you need to accomplish the following general actions - everything else is only technical details:

• Get a copy of CentOS 4.4 LiveCD image
• Use unsquashfs tool to extract the contents of *.mo files, located on the image
• Construct booted LiveCD root filesystem from extracted files
• Create OpenVZ kernel RPM-packages with squashfs support
• Compile unionfs kernel modules against OpenVZ kernels
• Get OpenVZ tools RPM-packages: vzctl, vzquota, vzpkg, etc. Get template metadatas and precreated templates
• Remove some RPM-packages from booted LiveCD root filesystem (conflicting with OpenVZ kernel packages and redundant packages - to free some disk space)
• Install OpenVZ kernels, tools and templates in booted LiveCD root filesystem
• Using mksquashfs tool compress booted CD root filesystem
• Modify initrd image: add there kernel modules for OpenVZ kernels
• Introduce small modifications to isolinux configuration file
• Create new iso image

Technical details

This part contains detailed description of LiveCD creation. I've removed the output of some commands in order to simplify the reading.

Creating booted LiveCD root filesystem

Create a directory, that will be our "workbench":

# cd ~
# mkdir workbench
# cd workbench

First of all we need some raw material - CentOS 4.4 LiveCD image:

# wget http://mirror.nsc.liu.se/CentOS/4.4/isos/i386/CentOS-4.4-i386-LiveCD.iso
# wget http://mirror.nsc.liu.se/CentOS/4.4/isos/i386/md5sum.livecd
# md5sum --check md5sum.livecd 
CentOS-4.4-i386-LiveCD.iso: OK

Copy the contents of CD to the directory, where we will form the OpenVZ LiveCD filesystem. I named this directory "ovz-livecd":

# mkdir ovz-livecd
# mkdir mount
# mount -o loop -t iso9660 CentOS-4.4-i386-LiveCD.iso mount/
# cp -a mount/* ovz-livecd/
# umount mount
# rm -rf mount

You can see, that ovz-livecd/base/ directory owns several *.mo files - images of squashfs filesystem, and each image holds one directory of booted LiveCD root filesystem: bin, usr and so on:

# ls -1 ovz-livecd/base
bin.mo etc.mo lib.mo root.mo sbin.mo usr.mo var.mo

In order to decompress these images we need unsquashfs tool. squashfs source archive contains patches for various kernel versions (to be able mount squashfs), and user-space tools for compressing usual filesytem into squashfs filesystem and for decompressing it back. Well, we don't need kernel patches right now (but we will need them later), we only want to install squashfs user-space tools:

# wget wget http://switch.dl.sourceforge.net/sourceforge/squashfs/squashfs3.2-r2.tar.gz
# tar xzf squashfs3.2-r2.tar.gz
# cd squashfs3.2-r2/squashfs-tools/
# make
cc mksquashfs.o read_fs.o sort.o -lz -lpthread -lm -o mksquashfs
cc unsquashfs.o -lz -o unsquashfs
# cp mksquashfs /usr/bin/
# cp unsquashfs /usr/bin/
# cd ~/workbench

Well, having the intruments in hand, why not extract the images? Create a directory, called livecd-root, that will store the root filesystem of booted LiveCD. I.e. it will contain /bin, /sbin, /etc and other directories, which you will see after booting the LiveCD. Unfortunately unsquashfs tool is not very convenient, and I found only this way to use it:

# mkdir livecd-root

# unsquashfs ovz-livecd/base/bin.mo 
# mv squashfs-root/bin/ livecd-root/
# rm -rf squashfs-root

# unsquashfs ovz-livecd/base/etc.mo 
# mv squashfs-root/etc/ livecd-root/
# rm -rf squashfs-root

# unsquashfs ovz-livecd/base/lib.mo 
# mv squashfs-root/lib/ livecd-root/
# rm -rf squashfs-root

# unsquashfs ovz-livecd/base/root.mo 
# mv squashfs-root/root/ livecd-root/
# rm -rf squashfs-root

# unsquashfs ovz-livecd/base/sbin.mo 
# mv squashfs-root/sbin/ livecd-root/
# rm -rf squashfs-root

# unsquashfs ovz-livecd/base/usr.mo    # this action takes some time, be patient ;)
# mv squashfs-root/usr/ livecd-root/
# rm -rf squashfs-root

# unsquashfs ovz-livecd/base/var.mo
# mv squashfs-root/var/ livecd-root/
# rm -rf squashfs-root

Yet we have nearly complete root filesystem of booted LiveCD in livecd-root directory. However some things are missing, and I guess you know what are they: /dev, /proc, /sys, /tmp and /boot directories. We need them, because some tools (rpm, bash, ...) will not work without them in chrooted environment later.

# mkdir root/dev
# cp -a /dev/* root/dev
# mkdir root/proc
# mount -t proc proc root/proc/
# mkdir root/sys
# mount -t sysfs sysfs root/sys/
# mkdir root/tmp
# mkdir root/boot

That's it. We have booted LiveCD root filesystem in livecd-root directory. Now we can chroot in this directory and install all needed packages. But what packages do we need?

• OpenVZ kernels
• OpenVZ tools: vzctl, vzquota, vzpkg, vzrpm43, vzrpm44, vzyum
• OpenVZ template metadatas: centos-4, fc-3, fc4 and fc-5
• Some OpenVZ precreated templates
• Some CentOS tool updates: sqlite package

Tools and templates related packages can be simply downloaded and installed. But it is not so easy with kernel. Usual OpenVZ kernel won't work: it doesn't support squashfs. The next chapter explains how to create RPM-package with OpenVZ kernel, that supports squashfs

Building OpenVZ kernels

Let's consider, how to build 2.6.18 OpenVZ kernel RPM-package with squashfs support. First download and install OpenVZ source RPM:

# wget http://download.openvz.org/kernel/branches/2.6.18/028stab035.1/kernel-2.6.18-ovz028stab035.1.src.rpm
# rpm -ivh kernel-2.6.18-ovz028stab035.1.src.rpm

squashfs sources contain the patch against 2.6.18. Let's copy this patch to RPM build directory:

# cp squashfs3.2-r2/kernel-patches/linux-2.6.18/squashfs3.2-patch /usr/src/redhat/SOURCES/

Now we need to modify the spec-file (/usr/src/redhat/SPECS/kernel-ovz.spec) in order to add squashfs patch to build process. Below are the changes you should introduce in diff format. The meaning of changes is transparent: firstly, do not build UP, enterprise and debug kernels (our LiveCD will be based on SMP kernel) and secondly, apply squashfs patch before compilation.

--- kernel-ovz.spec.orig        2007-06-09 11:40:15.000000000 +0400
+++ kernel-ovz.spec     2007-06-28 17:42:41.000000000 +0400
@@ -5,13 +5,13 @@ Summary: The Linux kernel (the core of t

 # normal/debug/debuginfo
 %define buildnormal 1
-%define builddebug 1
+%define builddebug 0
 %define builddebuginfo 0

 # kernels we want to build
-%define buildup 1
+%define buildup 0
 %define buildsmp 1
-%define buildenterprise 1
+%define buildenterprise 0
 %define buildentnosplit 0

 # versions
@@ -161,6 +161,9 @@ Patch2104: linux-2.6.18-drbd-0.7.22-0.8.
 Patch2105: linux-2.6.18-drbd-8.0.0-8.0.2.patch
 Patch2106: linux-2.6.18-drbd-8.0.2-8.0.3.patch

+# squashfs3.2-patch
+Patch2107: squashfs3.2-patch
+
 # ----------------------------------------------------------------------------

 BuildRoot: %{_tmppath}/kernel-%{KVERREL}-root
@@ -318,6 +321,8 @@ cd linux-%{kversion}
 %patch2105 -p1
 %patch2106 -p1

+%patch2107 -p1
+
 # ----------------------------------------------------------------------------

 mkdir configs

We need one more tiny hack in order RPM-build process to walk smoothly. There are config files for various OpenVZ kernel flavours in /usr/src/redhat/SOURCES directory:

# ls /usr/src/redhat/SOURCES/*.config.ovz
/usr/src/redhat/SOURCES/kernel-2.6.18-i686.config.ovz             /usr/src/redhat/SOURCES/kernel-2.6.18-ppc64.config.ovz
/usr/src/redhat/SOURCES/kernel-2.6.18-i686-enterprise.config.ovz  /usr/src/redhat/SOURCES/kernel-2.6.18-x86_64.config.ovz
/usr/src/redhat/SOURCES/kernel-2.6.18-i686-smp.config.ovz         /usr/src/redhat/SOURCES/kernel-2.6.18-x86_64-smp.config.ovz
/usr/src/redhat/SOURCES/kernel-2.6.18-ia64.config.ovz

All these files should contain information about squashfs. In fact, only two additional lines are necessary:

CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_EMBEDDED=n

I did it with the following command:

# cd /usr/src/redhat/SOURCES/
# echo "CONFIG_SQUASHFS=m
> CONFIG_SQUASHFS_EMBEDDED=n" | tee -a *.config.ovz
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_EMBEDDED=n
# cd ~/workbench/

After all efforts, building the RPM-package is very easy:

# rpmbuild --target=i686 -ba kernel-ovz.spec

If everything goes smoothly, in several tens of minutes depending on the power of your node you willl have OpenVZ kernel RPM-package with squashfs support. Let's copy it to our local directory to_booted_livecd/kernels/:

# ls /usr/src/redhat/RPMS/i686/
kernel-smp-2.6.18-ovz028stab035.1.i686.rpm
# mkdir -p to_booted_livecd/kernels
# cp /usr/src/redhat/RPMS/i686/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm to_booted_livecd/kernels/

OpenVZ LiveCD supports several kernels. So, the actions mentioned above should be accomplished to all kernels, that you need on LiveCD. In this article I skip the description of building RPMS for all kernels. All description below are based on 028stab035 kernel, that has been just build above.

Unionfs kernel module

In most present-day LiveCDs unionfs filesystem is used. This filesystem "can appear to merge the contents of several directories, while keeping their physical content separate". For example, after the command

# mount -t unionfs -o dirs=/tmp/changes=rw:/cdrom=ro unionfs /union

you will see merged contents of /cdrom and /tmp/changes directories in /union directory. At that, cdrom will be read-only. I.e. if you will write to some file in /union directory, that resides in /cdrom in fact, this modified file will be automatically moved to /tmp/changes. Such useful property is logically used in LiveCDs: unionfs merges cdrom contents (that is read-only) and temporary in-memory filesystem. In order to add support of unionfs filesystem to OpenVZ kernel we need to compile unionfs kernel module against used kernel. Download the sources of unionfs:

# wget ftp://ftp.filesystems.org/pub/unionfs/unionfs-1.x/unionfs-1.4.tar.gz
# tar xzf unionfs-1.4.tar.gz

In previous chapter we have created an RPM-package with OpenVZ kernel. We should install it: because this packages contents kernel headers, required for building kernel modules.

# rpm -ivh to_booted_livecd/kernels/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm 
Preparing...                ########################################### [100%]
   1:kernel-smp             ########################################### [100%]

Now compile unionfs:

# cd unionfs-1.4
# echo 'LINUXSRC=/lib/modules/2.6.18-ovz028stab035.1-smp/build/
> TOPINC=-I$(LINUXSRC)/include
> MODDIR=/lib/modules/2.6.18-ovz028stab035.1-smp/' > fistdev.mk
# make
# ls unionfs.ko 
unionfs.ko
# cd ..

Later we will place unionfs.ko to appropriate place in booted LiveCD filesystem. At the moment copy it to local to_booted_livecd/unionfs-modules/2.6.18/ directory:

# mkdir -p to_booted_livecd/unionfs-modules/2.6.18/
# cp unionfs-1.4/unionfs.ko to_booted_livecd/unionfs-modules/2.6.18/

I repeat once again: if you want to use several kernels on LiveCD - the operation above should be repeated for each wishful kernel.

OpenVZ tools and templates. CentOS updates

We have now the kernel RPM-package. But we also need OpenVZ tools RPM-packages. It is much easier with them: just download these packages from http://openvz.org:

# mkdir to_booted_livecd/tools
# cd to_booted_livecd/tools/

# wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-3.0.16-1.i386.rpm
# wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-lib-3.0.16-1.i386.rpm
# wget http://download.openvz.org/utils/vzquota/3.0.9/vzquota-3.0.9-1.i386.rpm
# wget http://download.openvz.org/template/utils/vzpkg/2.7.0-18/vzpkg-2.7.0-18.noarch.rpm
# wget http://download.openvz.org/template/utils/vzrpm43/current/vzrpm43-4.3.3-7_nonptl.6.i386.rpm
# wget http://download.openvz.org/template/utils/vzrpm43/current/vzrpm43-python-4.3.3-7_nonptl.6.i386.rpm
# wget http://download.openvz.org/template/utils/vzrpm44/current/vzrpm44-4.4.1-22.5.i386.rpm
# wget http://download.openvz.org/template/utils/vzrpm44/current/vzrpm44-python-4.4.1-22.5.i386.rpm
# wget http://download.openvz.org/template/utils/vzyum/current/vzyum-2.4.0-11.noarch.rpm

# mkdir ../template-metadatas
# cd ../template-metadatas

# wget http://download.openvz.org/template/metadata/centos-4/vztmpl-centos-4-2.0-2.i386.rpm
# wget http://download.openvz.org/template/metadata/fedora-core-3/vztmpl-fedora-core-3-2.0-2.i386.rpm
# wget http://download.openvz.org/template/metadata/fedora-core-4/vztmpl-fedora-core-4-2.0-2.i386.rpm
# wget http://download.openvz.org/template/metadata/fedora-core-5/vztmpl-fedora-core-5-2.0-2.i386.rpm

# mkdir ../precreated
# cd ../precreated

# wget http://download.openvz.org/template/precreated/centos-4-i386-minimal.tar.gz
# wget http://download.openvz.org/template/precreated/debian-3.1-i386-minimal.tar.gz
# wget http://download.openvz.org/template/precreated/fedora-core-5-i386-minimal.tar.gz

# cd ~/workbench/

As you understand, you can add other precreated templates to LiveCD. The main thing here is that usual CD is not rubber and precreated templates are rather big! ;)

Centos 4.4 LiveCD has a serios problem: sqlite package is broken and consequently yum doesn't work. In order to fix this problem we need updated sqlite package:

# mkdir to_booted_livecd/updates
# cd to_booted_livecd/updates
# wget http://centos.arcticnetwork.ca/4.5/os/i386/CentOS/RPMS/sqlite-3.3.6-2.i386.rpm
# cd ~/workbench/

Install all these finally

We have all components that will be on booted LiveCD ready. Let's install them in chrooted environment. But before uninstall lsctp-tools package, because these tools need kernel version prior to 2.6.10. Then update sqlite package. Also we need to remove some other packages in order to free some disk space: I've removed openoffice.org and related packages:

# cp -r to_booted_livecd/ root/tmp/
# chroot root/

# rpm -e lksctp-tools-1.0.2-6.4E.1.i386
# rpm -e openoffice.org-libs-1.1.5-4.6.0.EL4 openoffice.org-1.1.5-4.6.0.EL4

# cd /tmp/to_booted_livecd/

# rpm -Uvh updates/sqlite-3.3.6-2.i386.rpm

# rpm -ivh kernels/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm 
# rpm -ivh tools/*.rpm
# rpm -ivh template-metadatas/*.rpm

Copy precreated templates to appropriate place:

# cp  precreated/* /vz/template/cache/

Add unionfs.ko to other kernel modules:

# mkdir /lib/modules/2.6.18-ovz028stab035.1-smp/kernel/fs/unionfs
# cp unionfs-modules/2.6.18/unionfs.ko /lib/modules/2.6.18-ovz028stab035.1-smp/kernel/fs/unionfs/
# exit
exit

Booted LiveCD filesystem is ready, congratulations!

Compressing booted LiveCD filesystem

Since mksquashfs tool is already installed we can simply use it to create *.mo files:

# rm -f ovz-livecd/base/*.mo
# mksquashfs root/lib/ ovz-livecd/base/bin.mo -keep-as-directory
# mksquashfs root/etc/ ovz-livecd/base/etc.mo -keep-as-directory
# mksquashfs root/lib/ ovz-livecd/base/lib.mo -keep-as-directory
# mksquashfs root/root/ ovz-livecd/base/root.mo -keep-as-directory
# mksquashfs root/sbin/ ovz-livecd/base/sbin.mo -keep-as-directory
# mksquashfs root/usr/ ovz-livecd/base/usr.mo -keep-as-directory         # it can take long...
# mksquashfs root/var/ ovz-livecd/base/var.mo -keep-as-directory

Vmlinux, initrd and isolinux.cfg

The next step is to force isolinux (cdrom boot loader) to boot OpenVZ kernel. First, copy kernel image to OpenVZ CD:

# cp root/boot/vmlinuz-2.6.18-ovz028stab035.1-smp ovz-livecd/boot/

Modify initrd image. More precisely, we just need to add OpenVZ kernel modules there:

# mkdir initrd.d
# cp ovz-livecd/boot/initrd.gz initrd.d/
# cd initrd.d/
# mkdir mount
# gunzip initrd.gz 
# mount -o loop initrd mount/
# cp -r ../root/lib/modules/2.6.18-ovz028stab035.1-smp/ mount/lib/modules/
# rm -rf mount/lib/modules/2.6.18-ovz028stab035.1-smp/build                 # we don't neet it on initrd image
# sync
# umount mount/
# gzip initrd
# /bin/cp initrd.gz ../ovz-livecd/boot/initrd.gz
# cd ..

Now we should modify ovz-livecd/isolinux.cfg file, that controls isolinux boot loader. We inform boot loader, that new kernel was added by introducing the following changes (in diff format):

--- ovz-livecd/isolinux.cfg.orig        2006-08-19 08:53:09.000000000 +0400
+++ ovz-livecd/isolinux.cfg     2007-06-29 18:22:29.000000000 +0400
@@ -1,5 +1,5 @@
 display boot/splash.cfg
-default linux
+default ovz-2.6.18
 prompt 1
 timeout 300
 F1 boot/help1.txt
@@ -11,5 +11,9 @@ label linux
 kernel boot/vmlinuz
 append max_loop=32 vga=791 initrd=boot/initrd.gz init=linuxrc load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=40000 root=/dev/ram0 rw rhgb quiet
 
+label ovz-2.6.18 
+kernel boot/vmlinuz-2.6.18-ovz028stab035.1-smp
+append max_loop=32 vga=791 initrd=boot/initrd.gz init=linuxrc load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=40000 root=/dev/ram0 rw rhgb quiet
+
 label memtest
 kernel boot/memtest

Creating ISO

Centos 4.4 LiveCD creators were very gentle and have placed make_iso.sh script on CD. Using this sctipt, it is very easy to create iso:

./make_iso.sh ../CentOS-4.4-i386-OpenVZ-LiveCD.iso

External Links
* Official CentOS community website
* Linux kernel website
* RedHat Inc. website
* squashfs website
* uninonfs official homepage