Creating OpenVZ LiveCD

The idea of creating OpenVZ LiveCD for a long time was in the air. Such CD would be useful for various demonstrations, test-drives and some other purposes. Additionally we received reports that some problems persist in creating LiveCD based on OpenVZ kernel. In order to eliminate these problems and give users the ability to use OpenVZ technology in LiveCD manner we decided to create it. This article describes how I was doing about it. Probably there are some easier and more effective ways, so feel free to improve the article. We chose KNOPPIX as a base for our CD for two reasons. Firstly KNOPPIX is probably the most widespread LiveCD in the world. Secondly KNOPPIX was interested in integrating OpenVZ in one of its special releases. The article starts from conceptual actions required to create OpenVZ LiveCD and then step-by-step describes technical aspects. I want to note, that technical aspects depend a bit on certain version of kernel, aufs and cloop you're using. For example some small changes in code (functions argument type change and alike) are necessary. I used aufs-20061113 and cloop-2.05-1 because these versions were used on in KNOPPIX 5.1.1. 2.6.18-8.el5-028stab027 OpenVZ kernel was used.

Concepts

1. Download KNOPPIX iso and get access to its content
2. Extract KNOPPIX compressed image
3. Somehow obtain OpenVZ Debian packages: kernel, vzctl, vzquota and template
4. Uninstall previous Linux kernel and related packages from extracted tree
5. Install OpenVZ Debian packages there
6. Compress the tree
7. Build aufs and cloop kernel modules against OpenVZ kernel
8. Copy these modules in appropriate places on the disc
9. Modify initrd image of KNOPPIX
10. Create new ISO image

Step-by-step actions

First we need to download KNOPPIX iso image. Here is a list of mirrors where you can download it. I used KNOPPIX 5.1.1 from the nearest mirror:

mkdir openvz-livecd
cd openvz-livecd
wget ftp://ftp.chg.ru/pub/Linux/knoppix/KNOPPIX_V5.1.1CD-2007-01-04-EN.iso

Let's copy contents of this iso to to the hard drive:

mkdir mnt
mount -o loop KNOPPIX_V5.1.1CD-2007-01-04-EN.iso mnt
mkdir cd-contents
cp -a mnt/* cd-contents/
umount mnt
rm -r mnt

When you're booted in KNOPPIX LiveCD you can observe a lot of files and use various applications. On CD all this information is stored in a single file:

ls -lh cd-contents/KNOPPIX/KNOPPIX
-rw-r--r--  1 vass vass 691M Jan  4 03:50 cd-contents/KNOPPIX/KNOPPIX

As you can see its size is 691M, but after uncompressing it'll be approximately 2Gb. In order to uncompress this file we need special tools: cloop-utils. You can download them from this site. I used cloop-2.05-1, because exactly this version is installed on KNOPPIX 5.1.1.

wget http://debian-knoppix.alioth.debian.org/sources/cloop_2.05-1.tar.gz
tar xzf cloop_2.05-1.tar.gz
cd cloop_2.05-1
make extract_compressed_fs
make create_compressed_fs # this tool will be required later, when compressing modified tree
cd ..

After we have extract_compressed_fs tool in hand, let's extract KNOPPIX image:

cloop_2.05-1/extract_compressed_fs cd-contents/KNOPPIX/KNOPPIX > KNOPPIX.img
ls -lh KNOPPIX.img
-rw-rw-r--  1 vass vass 2.0G Apr 12 17:38 KNOPPIX.img

KNOPPIX.img file can be mounted using usual '-o loop' option. Next our step is to create a copy of KNOPPIX file system tree on the hard disk. Note, that we can't use 'cp' tool here, because of hardlinks and special files. We should use rsynс:

mkdir mnt
mount -o loop KNOPPIX.img mnt
mkdir KNOPPIX-content
rsync -Hav mnt KNOPPIX-content
umount mnt
rm -r mnt

Now we should set up OpenVZ environment in obtained tree: kernel, vzctl, vzquota, template. For vzctl and vzquota we can use precompiled packages from http://debian.systs.org. But at the moment we have to create kernel Debian package by self, because we need additional patches in order all to work smoothly (but it will be unnecessary soon, when patches will be included in main OpenVZ patch). This is important to use KNOPPIX default kernel config! The creators of LiveCD take a lot of care about customizing config file in such manner, that CD will suite as many users as possible. We shouldn't throw away such great job. We do `make oldconfig` and press enter all the time. Additionally CONFIG_MATH_EMULATION should be turn off at the moment. (http://bugzilla.openvz.org/show_bug.cgi?id=433) I created a Debian VE on my machine and install there kernel-package package, that allows to create Debian kernel packages easily. After that the creation of package is something like that:

mkdir ovz-kernel
cd ovz-kernel
wget ftp://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2
wget http://download.openvz.org/kernel/branches/rhel5-2.6.18/028stab027.1/patches/patch-8.el5.028stab027.1-combined.gz
wget http://download.openvz.org/kernel/branches/rhel5-2.6.18/028stab027.1/configs/kernel-2.6.18-i686.config.ovz
wget http://download.openvz.org/~vass/diff-ubc-mm-null-pointer-for-kthread-20070406 # this patch will be not necessary soon.
tar xjf linux-2.6.18.tar.bz2
zcat patch-8.el5.028stab027.1-combined.gz | patch -p1 -d linux-2.6.18
patch -p1 -d linux-2.6.18 < diff-ubc-mm-null-pointer-for-kthread-20070406
mv linux-2.6.18 linux-2.6.18-8.el5-028test027
vim linux-2.6.18-8.el5-028test027/Makefile                   # here we add right extraversion: -8.el5-028test027
cp linux-2.6.19 linux-2.6.18/.config
cd linux-2.6.18-8.el5-028test027
make oldconfig                                               # pressing enter here
vim .config                                                  # switching off CONFIG_MATH_EMULATION
make-kpkg --revision=.1 kernel_image
cd ..
cd ..

If everything went smoothly, in the top directory we'll be able to see linux-image file:

ls -l linux-image-2.6.18-8.el5-028stab027_.1_i386.deb
-rw-r--r-- 1 vass vass 13364308 Apr 16 11:06 linux-image-2.6.18-8.el5-028stab027_.1_i386.deb

Don't delete sources and compilation of this kernel. A bit later we'll need to build aufs and cloop kernel modules against this kernel. Now I download vzctl and vzquota Debian packages from http://debian.systs.org and a template:

wget  http://debian.systs.org/debian/pool/openvz/v/vzctl/vzctl_3.0.16-1dso2~sarge0_i386.deb
wget http://debian.systs.org/debian/pool/openvz/v/vzquota/vzquota_3.0.9-1dso1_i386.deb
wget http://download.openvz.org/template/precreated/debian-3.1-i386-minimal.tar.gz

Copying packages to KNOPPIX tree and chroot there:

cp linux-image-2.6.18-8.el5-028stab027_.1_i386.deb KNOPPIX-content/root
cp vzctl_3.0.16-1dso2~sarge0_i386.deb KNOPPIX-content/root
cp vzquota_3.0.9-1dso1_i386.deb KNOPPIX-content/root
cp debian-3.1-i386-minimal.tar.gz KNOPPIX-content/root
chroot KNOPPIX-content
mount -t proc proc /proc
mount -t tmpfs tmpfs /tmp # in order not to think about trash in /tmp later
cd /root

In chrooted environment we need to remove all kernel related packages. In KNOPPIX 5.1.1 2.6.19 vanilla kernel is used, so we can easily identify all kernel-related packages and eliminate them using this knowledge:

dpkg-query --list | grep 2.6.19
apt-get remove aufs-cvs-modules-2.6.19
apt-get remove cloop-module-2.6.19
apt-get remove madwifi-modules-2.6.19
apt-get remove ndiswrapper-modules-2.6.19
apt-get remove rt2x00-cvs-modules-2.6.19
apt-get remove loop-aes-modules-2.6.19
apt-get remove linux-image-2.6.19

Installing OpenVZ tools (don't be afraid about warnings: they are ok)

apt-get install linux-image-2.6.18-8.el5-028stab027_.1_i386.deb
apt-get install vzquota_3.0.9-1dso1_i386.deb
apt-get install vzctl_3.0.16-1dso2~sarge0_i386.deb
mv debian-3.1-i386-minimal.tar.gz /var/lib/vz/template/cache/

Here I want to note one thing. We added some data to the KNOPPIX image, so its size increased. If you want to burn iso on CD/DVD later you should think of removing some packages. I miss this point, but you're aware. After all we need to clean up chrooted environment:

umount /proc
umount /tmp
rm -rf /root/*
exit

That's great, now we have all the tree ready, let's compress it in the new image using create_compressed_fs tool and copy to the right place:

mkisofs -R -iso-level 4 -force-rr -J  ./KNOPPIX-content-modif/ > KNOPPIX-modif.img
./cloop-2.05/create_compressed_fs KNOPPIX-modif.img 65536 > KNOPPIX-modif
cp KNOPPIX-modif cd-contents/KNOPPIX/KNOPPIX

The next our step is to compile aufs and cloop kernel modules against OpenVZ kernel. Let's start from cloop module. A definition in source must be commented, because we're using RHEL-based kernel. The patch below shows what to do:

--- cloop-2.05/compressed_loop.c.rh     2006-10-13 23:39:41.000000000 +0400
+++ cloop-2.05/compressed_loop.c        2007-04-12 17:21:24.000000000 +0400
@@ -31,7 +31,7 @@
 #endif

 /* Define this if you are using Greenshoe Linux */
-/* #define REDHAT_KERNEL */
+ #define REDHAT_KERNEL

 #include <linux/kernel.h>
 #include <linux/version.h>

Getting the sources, building:

wget http://debian-knoppix.alioth.debian.org/sources/cloop_2.05-1.tar.gz
tar xzf cloop_2.05-1.tar.gz
cd cloop-2.05/
vim compressed_loop.c                                      # uncommenting define here
make KERNEL_DIR=../linux-2.6.18-8.el5-028stab027/ module

After that you should have cloop.ko file in current directory. Later we will copy it to appropriate place. What about aufs? Get the sources from cvs. I used 20061113 version, because this version is used in KNOPPIX 5.1.1. The following changes are required in aufs, because we're using rhel5-based kernel:

--- aufs/fs/aufs/cpup.c.orig    2006-10-23 12:58:33.000000000 +0000
+++ aufs/fs/aufs/cpup.c 2007-04-16 14:46:51.000000000 +0000
@@ -85,9 +85,6 @@ void cpup_attr_all(struct inode *inode)
                inode->i_rdev = hidden_inode->i_rdev;
        }
        inode->i_blkbits = hidden_inode->i_blkbits;
-#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,19)
-       inode->i_blksize = hidden_inode->i_blksize;
-#endif
 }

 /* ---------------------------------------------------------------------- */
--- aufs/fs/aufs/dir.c.orig     2006-10-23 12:59:29.000000000 +0000
+++ aufs/fs/aufs/dir.c  2007-04-16 14:58:34.000000000 +0000
@@ -223,7 +223,7 @@ struct test_empty_arg {
 };

 static int test_empty_cb(void *__arg, const char *__name, int namelen,
-                        loff_t offset, filldir_ino_t ino, unsigned int d_type)
+                        loff_t offset, u64 ino, unsigned int d_type)
 {
        struct test_empty_arg *arg = __arg;
        char *name = (void*)__name;
--- aufs/fs/aufs/vdir.c.orig    2006-10-23 13:02:20.000000000 +0000
+++ aufs/fs/aufs/vdir.c 2007-04-16 15:01:44.000000000 +0000
@@ -389,7 +389,7 @@ struct fillvdir_arg {
 };

 static int fillvdir(void *__arg, const char *__name, int namelen, loff_t offset,
-                   filldir_ino_t hidden_ino, unsigned int d_type)
+                   u64 hidden_ino, unsigned int d_type)
 {
        struct fillvdir_arg *arg = __arg;
        ino_t ino;

Building aufs:

cvs -d:pserver:anonymous@aufs.cvs.sourceforge.net:/cvsroot/aufs login
cvs -z3 -d:pserver:anonymous@aufs.cvs.sourceforge.net:/cvsroot/aufs co -D20061113 aufs
cd aufs
vim fs/aufs/cpup.c                                        #
vim fs/aufs/dir.c                                         # Introducing required changes
vim fs/aufs/vdir.c                                        #
make KDIR=../../linux-2.6.18-8.el5-028stab027/ -f local.mk

Now you have an aufs.ko file in current directory. Next we need to add these modules to the disc:

cp aufs.ko cloop.ko cd-contents/KNOPPIX/modules/

When booting KNOOPIX uses initrd image, where it stores some kernel modules required to get access to the CD-drive. We need to remove old modules and add new one to initrd image. Extracting and mounting initrd:

mkdir initrd
cd initrd
cp cd-contents/boot/isolinux/minirt.gz ./
gunzip minirt.gz
mkdir mnt
mount -o loop minirt mnt

The modules directory on initrd contents kernel modules. Now we should replace these modules by equivalents from OpenVZ kernel. Note, that some modules are already build in OpenVZ kernel, so they are missing. After this operation is done copy new minirt.gz to the appropriate place:

umount mnt
gzip minirt
cp minirt.gz ../cd-contents/boot/isolinux/
cd ..

The last step is to copy vmlinuz to isolinux directory:

cp vmlinuz-2.6.18-8.el5-028stab027 cd-contents/boot/isolinux/linux

Wow! We're ready to create is with KNOPPIX OpenVZ CD:

chmod -R u+w cd-contents
rm cd-contents/boot/isolinux/boot.cat
mkisofs -b boot/isolinux/isolinux.bin -o KNOPPIX_V5.1.1-OPENVZ-CD-2007-01-04-EN.iso -no-emul-boot -l -J -r -boot-load-size 4 -boot-info-table KNOPPIX_V5.1.1CD-2007-01-04-EN.iso-content

Result

[todo]

Links

1. http://www.knopper.net/knoppix/index-en.html
2. http://www.knoppix.net/wiki/Knoppix_Remastering_Howto
3. http://debian.systs.org/