Creating OpenVZ LiveCD based on Centos 4.4
This article is a step by step guide explaining how to add OpenVZ functionality to Centos 4.4 LiveCD. As a result, you will have a LiveCD, that can boot in various OpenVZ kernels (rhel4-2.6.9, 2.6.18, rhel5-2.6.18, 2.6.20) and in original CentOS kernel also. After booting from the CD, you will be able to create VEs, start them, enter them - i.e. perform common OpenVZ activity.
CentOS 5 LiveCD is already announced and as soon as it will be available we'll update the article.
General actions
It is quite easy to add OpenVZ functionality to existent LiveCD. The most difficult work is already done by the original LiveCD creators. But it requires pretty much hand job. Probably some day some good guy will automatize this process. ;)
Actually you need to accomplish the following general actions - everything else is only technical details:
- Get a copy of CentOS 4.4 LiveCD image
- Use unsquashfs tool to extract the contents of *.mo files, located on the image
- Construct booted LiveCD root filesystem from extracted files
- Create OpenVZ kernel RPM-packages with squashfs support
- Compile unionfs kernel modules against OpenVZ kernels
- Get OpenVZ tools RPM-packages: vzctl, vzquota, vzpkg, etc. Get template metadatas and precreated templates
- Remove some RPM-packages from booted LiveCD root filesystem (conflicting with OpenVZ kernel packages and redundant packages - to free some disk space)
- Install OpenVZ kernels, tools and templates in booted LiveCD root filesystem
- Using mksquashfs tool compress booted CD root filesystem
- Modify initrd image: add there kernel modules for OpenVZ kernels
- Introduce small modifications to isolinux configuration file
- Create new iso image
Technical details
This part contains detailed description of LiveCD creation. I've removed the output of some commands in order to simplify the reading.
Creating booted LiveCD root filesystem
Create a directory, that will be our "workbench":
# cd ~ # mkdir workbench # cd workbench
First of all we need some raw material - CentOS 4.4 LiveCD image:
# wget http://mirror.nsc.liu.se/CentOS/4.4/isos/i386/CentOS-4.4-i386-LiveCD.iso # wget http://mirror.nsc.liu.se/CentOS/4.4/isos/i386/md5sum.livecd # md5sum --check md5sum.livecd CentOS-4.4-i386-LiveCD.iso: OK
Copy the contents of CD to the directory, where we will form the OpenVZ LiveCD filesystem. I named this directory "ovz-livecd":
# mkdir ovz-livecd # mkdir mount # mount -o loop -t iso9660 CentOS-4.4-i386-LiveCD.iso mount/ # cp -a mount/* ovz-livecd/ # umount mount # rm -rf mount
You can see, that ovz-livecd/base/ directory owns several *.mo files - images of squashfs filesystem, and each image holds one directory of booted LiveCD root filesystem: bin, usr and so on:
# ls -1 ovz-livecd/base bin.mo etc.mo lib.mo root.mo sbin.mo usr.mo var.mo
In order to decompress these images we need unsquashfs tool. squashfs source archive contains patches for various kernel versions (to be able mount squashfs), and user-space tools for compressing usual filesytem into squashfs filesystem and for decompressing it back. Well, we don't need kernel patches right now (but we will need them later), we only want to install squashfs user-space tools:
# wget wget http://switch.dl.sourceforge.net/sourceforge/squashfs/squashfs3.2-r2.tar.gz # tar xzf squashfs3.2-r2.tar.gz # cd squashfs3.2-r2/squashfs-tools/ # make cc mksquashfs.o read_fs.o sort.o -lz -lpthread -lm -o mksquashfs cc unsquashfs.o -lz -o unsquashfs # cp mksquashfs /usr/bin/ # cp unsquashfs /usr/bin/ # cd ~/workbench
Well, having the intruments in hand, why not extract the images? Create a directory, called livecd-root, that will store the root filesystem of booted LiveCD. I.e. it will contain /bin, /sbin, /etc and other directories, which you will see after booting the LiveCD. Unfortunately unsquashfs tool is not very convenient, and I found only this way to use it:
# mkdir livecd-root # unsquashfs ovz-livecd/base/bin.mo # mv squashfs-root/bin/ livecd-root/ # rm -rf squashfs-root # unsquashfs ovz-livecd/base/etc.mo # mv squashfs-root/etc/ livecd-root/ # rm -rf squashfs-root # unsquashfs ovz-livecd/base/lib.mo # mv squashfs-root/lib/ livecd-root/ # rm -rf squashfs-root # unsquashfs ovz-livecd/base/root.mo # mv squashfs-root/root/ livecd-root/ # rm -rf squashfs-root # unsquashfs ovz-livecd/base/sbin.mo # mv squashfs-root/sbin/ livecd-root/ # rm -rf squashfs-root # unsquashfs ovz-livecd/base/usr.mo # this action takes some time, be patient ;) # mv squashfs-root/usr/ livecd-root/ # rm -rf squashfs-root # unsquashfs ovz-livecd/base/var.mo # mv squashfs-root/var/ livecd-root/ # rm -rf squashfs-root
Yet we have nearly complete root filesystem of booted LiveCD in livecd-root directory. However some things are missing, and I guess you know what are they: /dev, /proc, /sys, /tmp and /boot directories. We need them, because some tools (rpm, bash, ...) will not work without them in chrooted environment later.
# mkdir root/dev # cp -a /dev/* root/dev # mkdir root/proc # mount -t proc proc root/proc/ # mkdir root/sys # mount -t sysfs sysfs root/sys/ # mkdir root/tmp # mkdir root/boot
That's it. We have booted LiveCD root filesystem in livecd-root directory. Now we can chroot in this directory and install all needed packages. But what packages do we need?
- OpenVZ kernels
- OpenVZ tools: vzctl, vzquota, vzpkg, vzrpm43, vzrpm44, vzyum
- OpenVZ template metadatas: centos-4, fc-3, fc4 and fc-5
- Some OpenVZ precreated templates
- Some CentOS tool updates: sqlite package
Tools and templates related packages can be simply downloaded and installed. But it is not so easy with kernel. Usual OpenVZ kernel won't work: it doesn't support squashfs. The next chapter explains how to create RPM-package with OpenVZ kernel, that supports squashfs
Building OpenVZ kernels
Let's consider, how to build 2.6.18 OpenVZ kernel RPM-package with squashfs support. First download and install OpenVZ source RPM:
# wget http://download.openvz.org/kernel/branches/2.6.18/028stab035.1/kernel-2.6.18-ovz028stab035.1.src.rpm # rpm -ivh kernel-2.6.18-ovz028stab035.1.src.rpm
squashfs sources contain the patch against 2.6.18. Let's copy this patch to RPM build directory:
# cp squashfs3.2-r2/kernel-patches/linux-2.6.18/squashfs3.2-patch /usr/src/redhat/SOURCES/
Now we need to modify the spec-file (/usr/src/redhat/SPECS/kernel-ovz.spec) in order to add squashfs patch to build process. Below are the changes you should introduce in diff format. The meaning of changes is transparent: firstly, do not build UP, enterprise and debug kernels (our LiveCD will be based on SMP kernel) and secondly, apply squashfs patch before compilation.
--- kernel-ovz.spec.orig 2007-06-09 11:40:15.000000000 +0400
+++ kernel-ovz.spec 2007-06-28 17:42:41.000000000 +0400
@@ -5,13 +5,13 @@ Summary: The Linux kernel (the core of t
# normal/debug/debuginfo
%define buildnormal 1
-%define builddebug 1
+%define builddebug 0
%define builddebuginfo 0
# kernels we want to build
-%define buildup 1
+%define buildup 0
%define buildsmp 1
-%define buildenterprise 1
+%define buildenterprise 0
%define buildentnosplit 0
# versions
@@ -161,6 +161,9 @@ Patch2104: linux-2.6.18-drbd-0.7.22-0.8.
Patch2105: linux-2.6.18-drbd-8.0.0-8.0.2.patch
Patch2106: linux-2.6.18-drbd-8.0.2-8.0.3.patch
+# squashfs3.2-patch
+Patch2107: squashfs3.2-patch
+
# ----------------------------------------------------------------------------
BuildRoot: %{_tmppath}/kernel-%{KVERREL}-root
@@ -318,6 +321,8 @@ cd linux-%{kversion}
%patch2105 -p1
%patch2106 -p1
+%patch2107 -p1
+
# ----------------------------------------------------------------------------
mkdir configs
We need one more tiny hack in order RPM-build process to walk smoothly. There are config files for various OpenVZ kernel flavours in /usr/src/redhat/SOURCES directory:
# ls /usr/src/redhat/SOURCES/*.config.ovz /usr/src/redhat/SOURCES/kernel-2.6.18-i686.config.ovz /usr/src/redhat/SOURCES/kernel-2.6.18-ppc64.config.ovz /usr/src/redhat/SOURCES/kernel-2.6.18-i686-enterprise.config.ovz /usr/src/redhat/SOURCES/kernel-2.6.18-x86_64.config.ovz /usr/src/redhat/SOURCES/kernel-2.6.18-i686-smp.config.ovz /usr/src/redhat/SOURCES/kernel-2.6.18-x86_64-smp.config.ovz /usr/src/redhat/SOURCES/kernel-2.6.18-ia64.config.ovz
All these files should contain information about squashfs. In fact, only two additional lines are necessary:
CONFIG_SQUASHFS=m CONFIG_SQUASHFS_EMBEDDED=n
I did it with the following command:
# cd /usr/src/redhat/SOURCES/ # echo "CONFIG_SQUASHFS=m > CONFIG_SQUASHFS_EMBEDDED=n" | tee -a *.config.ovz CONFIG_SQUASHFS=m CONFIG_SQUASHFS_EMBEDDED=n # cd ~/workbench/
After all efforts, building the RPM-package is very easy:
# rpmbuild --target=i686 -ba kernel-ovz.spec
If everything goes smoothly, in several tens of minutes depending on the power of your node you willl have OpenVZ kernel RPM-package with squashfs support. Let's copy it to our local directory to_booted_livecd/kernels/:
# ls /usr/src/redhat/RPMS/i686/ kernel-smp-2.6.18-ovz028stab035.1.i686.rpm # mkdir -p to_booted_livecd/kernels # cp /usr/src/redhat/RPMS/i686/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm to_booted_livecd/kernels/
OpenVZ LiveCD supports several kernels. So, the actions mentioned above should be accomplished to all kernels, that you need on LiveCD. In this article I skip the description of building RPMS for all kernels. All description below are based on 028stab035 kernel, that has been just build above.
Unionfs kernel module
In most present-day LiveCDs unionfs filesystem is used. This filesystem "can appear to merge the contents of several directories, while keeping their physical content separate". For example, after the command
# mount -t unionfs -o dirs=/tmp/changes=rw:/cdrom=ro unionfs /union
you will see merged contents of /cdrom and /tmp/changes directories in /union directory. At that, cdrom will be read-only. I.e. if you will write to some file in /union directory, that resides in /cdrom in fact, this modified file will be automatically moved to /tmp/changes. Such useful property is logically used in LiveCDs: unionfs merges cdrom contents (that is read-only) and temporary in-memory filesystem. In order to add support of unionfs filesystem to OpenVZ kernel we need to compile unionfs kernel module against used kernel. Download the sources of unionfs:
# wget ftp://ftp.filesystems.org/pub/unionfs/unionfs-1.x/unionfs-1.4.tar.gz # tar xzf unionfs-1.4.tar.gz
In previous chapter we have created an RPM-package with OpenVZ kernel. We should install it: because this packages contents kernel headers, required for building kernel modules.
# rpm -ivh to_booted_livecd/kernels/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm Preparing... ########################################### [100%] 1:kernel-smp ########################################### [100%]
Now compile unionfs:
# cd unionfs-1.4 # echo 'LINUXSRC=/lib/modules/2.6.18-ovz028stab035.1-smp/build/ > TOPINC=-I$(LINUXSRC)/include > MODDIR=/lib/modules/2.6.18-ovz028stab035.1-smp/' > fistdev.mk # make # ls unionfs.ko unionfs.ko # cd ..
Later we will place unionfs.ko to appropriate place in booted LiveCD filesystem. At the moment copy it to local to_booted_livecd/unionfs-modules/2.6.18/ directory:
# mkdir -p to_booted_livecd/unionfs-modules/2.6.18/ # cp unionfs-1.4/unionfs.ko to_booted_livecd/unionfs-modules/2.6.18/
I repeat once again: if you want to use several kernels on LiveCD - the operation above should be repeated for each wishful kernel.
OpenVZ tools and templates. CentOS updates
We have now the kernel RPM-package. But we also need OpenVZ tools RPM-packages. It is much easier with them: just download these packages from http://openvz.org:
# mkdir to_booted_livecd/tools # cd to_booted_livecd/tools/ # wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-3.0.16-1.i386.rpm # wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-lib-3.0.16-1.i386.rpm # wget http://download.openvz.org/utils/vzquota/3.0.9/vzquota-3.0.9-1.i386.rpm # wget http://download.openvz.org/template/utils/vzpkg/2.7.0-18/vzpkg-2.7.0-18.noarch.rpm # wget http://download.openvz.org/template/utils/vzrpm43/current/vzrpm43-4.3.3-7_nonptl.6.i386.rpm # wget http://download.openvz.org/template/utils/vzrpm43/current/vzrpm43-python-4.3.3-7_nonptl.6.i386.rpm # wget http://download.openvz.org/template/utils/vzrpm44/current/vzrpm44-4.4.1-22.5.i386.rpm # wget http://download.openvz.org/template/utils/vzrpm44/current/vzrpm44-python-4.4.1-22.5.i386.rpm # wget http://download.openvz.org/template/utils/vzyum/current/vzyum-2.4.0-11.noarch.rpm # mkdir ../template-metadatas # cd ../template-metadatas # wget http://download.openvz.org/template/metadata/centos-4/vztmpl-centos-4-2.0-2.i386.rpm # wget http://download.openvz.org/template/metadata/fedora-core-3/vztmpl-fedora-core-3-2.0-2.i386.rpm # wget http://download.openvz.org/template/metadata/fedora-core-4/vztmpl-fedora-core-4-2.0-2.i386.rpm # wget http://download.openvz.org/template/metadata/fedora-core-5/vztmpl-fedora-core-5-2.0-2.i386.rpm # mkdir ../precreated # cd ../precreated # wget http://download.openvz.org/template/precreated/centos-4-i386-minimal.tar.gz # wget http://download.openvz.org/template/precreated/debian-3.1-i386-minimal.tar.gz # wget http://download.openvz.org/template/precreated/fedora-core-5-i386-minimal.tar.gz # cd ~/workbench/
As you understand, you can add other precreated templates to LiveCD. The main thing here is that usual CD is not rubber and precreated templates are rather big! ;)
Centos 4.4 LiveCD has a serios problem: sqlite package is broken and consequently yum doesn't work. In order to fix this problem we need updated sqlite package:
# mkdir to_booted_livecd/updates # cd to_booted_livecd/updates # wget http://centos.arcticnetwork.ca/4.5/os/i386/CentOS/RPMS/sqlite-3.3.6-2.i386.rpm # cd ~/workbench/
Install all these finally
We have all components that will be on booted LiveCD ready. Let's install them in chrooted environment. But before uninstall lsctp-tools package, because these tools need kernel version prior to 2.6.10. Then update sqlite package. Also we need to remove some other packages in order to free some disk space: I've removed openoffice.org, gimp, dia, gnome-games, tetex, latex, k3b and printing related packages:
# cp -r to_booted_livecd/ root/tmp/ # chroot root/ # rpm -e lksctp-tools-1.0.2-6.4E.1.i386 # rpm -e openoffice.org-libs openoffice.org # rpm -e gimp-data-extras xsane-gimp gimp-print-plugin gimp-print-utils # rpm -e gimp gimp-print gimp-help gimp-gap # rpm -e dia # rpm -e planner # rpm -e gnome-games # rpm -e tetex-latex jadetex xmltex passivetex tetex tetex-afm \ tetex-dvips tetex-fonts tetex-xdvi # rpm -e k3b # rpm -e foomatic Omni Omni-foomatic system-config-printer \ system-config-printer-gui hal-cups-utils desktop-printing # cd /tmp/to_booted_livecd/ # rpm -Uvh updates/sqlite-3.3.6-2.i386.rpm # rpm -ivh kernels/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm # rpm -ivh tools/*.rpm # rpm -ivh template-metadatas/*.rpm
Copy precreated templates to appropriate place:
# cp precreated/* /vz/template/cache/
Add unionfs.ko to other kernel modules and run depmod:
# mkdir /lib/modules/2.6.18-ovz028stab035.1-smp/kernel/fs/unionfs # cp unionfs-modules/2.6.18/unionfs.ko /lib/modules/2.6.18-ovz028stab035.1-smp/kernel/fs/unionfs/ # depmod 2.6.18-ovz028stab035.1-smp # exit exit
While being in chrooted environment bash history was written to ~/.bash_history file. Clean it:
# rm root/root/.bash_history
Booted LiveCD filesystem is ready, congratulations!
Compressing booted LiveCD filesystem
Since mksquashfs tool is already installed we can simply use it to create *.mo files:
# rm -f ovz-livecd/base/*.mo # mksquashfs root/lib/ ovz-livecd/base/bin.mo -keep-as-directory # mksquashfs root/etc/ ovz-livecd/base/etc.mo -keep-as-directory # mksquashfs root/lib/ ovz-livecd/base/lib.mo -keep-as-directory # mksquashfs root/root/ ovz-livecd/base/root.mo -keep-as-directory # mksquashfs root/sbin/ ovz-livecd/base/sbin.mo -keep-as-directory # mksquashfs root/usr/ ovz-livecd/base/usr.mo -keep-as-directory # it can take long... # mksquashfs root/var/ ovz-livecd/base/var.mo -keep-as-directory # mksquashfs root/vz/ ovz-livecd/base/vz.mo -keep-as-directory
Vmlinux, initrd and isolinux.cfg
The next step is to force isolinux (cdrom boot loader) to boot OpenVZ kernel. First, copy kernel image to OpenVZ CD. It is important to give short name (not greater than 8 characters) to kernel image, otherwise isolinux will not see it.
# cp root/boot/vmlinuz-2.6.18-ovz028stab035.1-smp ovz-livecd/boot/ovz18
Modify initrd image. More precisely, we need to add OpenVZ kernel modules there and apply small patch to liblinuxlive file. Mount initrd:
# mkdir initrd.d # cp ovz-livecd/boot/initrd.gz initrd.d/ # cd initrd.d/ # mkdir mount # gunzip initrd.gz # mount -o loop initrd mount/
Copy modules to initrd:
# cp -r ../root/lib/modules/2.6.18-ovz028stab035.1-smp/ mount/lib/modules/ # rm -rf mount/lib/modules/2.6.18-ovz028stab035.1-smp/build # we don't neet it on initrd image
initrd image, that goes with Centos 4.4 LiveCD doesn't load cdrom kernel modules. It works well for CentOS kernel, because cdrom support is built-in there. But OpenVZ kernel supports cdrom (usual ide cdrom) via kernel module. Consequetnly we need to apply the following patch to liblinuxlive file:
--- liblinuxlive.orig 2006-08-19 08:53:11.000000000 +0400
+++ liblinuxlive 2007-07-02 17:50:34.000000000 +0400
@@ -337,6 +337,9 @@ modprobe_essential_modules()
{
echolog "starting loop device support"
modprobe_module loop max_loop=32
+ echolog "starting cdrom support"
+ modprobe_module cdrom
+ modprobe_module ide-cd
echolog "starting cdrom filesystem support"
modprobe_module isofs
echolog "starting squashfs support"
Finishing actions with initrd:
# sync # umount mount/ # gzip initrd # /bin/cp initrd.gz ../ovz-livecd/boot/initrd.gz # cd ..
Now we should modify ovz-livecd/isolinux.cfg file, that controls isolinux boot loader. We inform boot loader, that new kernel was added by introducing the following changes (in diff format):
--- ovz-livecd/isolinux.cfg.orig 2006-08-19 08:53:09.000000000 +0400 +++ ovz-livecd/isolinux.cfg 2007-07-02 19:10:51.000000000 +0400 @@ -1,5 +1,5 @@ display boot/splash.cfg -default linux +default ovz18 prompt 1 timeout 300 F1 boot/help1.txt @@ -11,5 +11,9 @@ label linux kernel boot/vmlinuz append max_loop=32 vga=791 initrd=boot/initrd.gz init=linuxrc load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=40000 root=/dev/ram0 rw rhgb quiet +label ovz18 +kernel boot/ovz18 +append max_loop=32 vga=791 initrd=boot/initrd.gz init=linuxrc load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=40000 root=/dev/ram0 rw rhgb quiet + label memtest kernel boot/memtest
Creating ISO
Centos 4.4 LiveCD creators were very gentle and have placed make_iso.sh script on CD. Using this script, it is very easy to create iso:
./make_iso.sh ../CentOS-4.4-i386-OpenVZ-LiveCD.iso
Debugging LiveCD
If you have trouble with created OpenVZ LiveCD, boot the kernel with
vga=0 debug
command line options. It will help you to diagnose the problem more precisely.
