Difference between revisions of "Creating OpenVZ LiveCD based on Centos 4.4"

From OpenVZ Virtuozzo Containers Wiki
Jump to: navigation, search
(some info added)
m (Reverted edits by 3qdesigns123 (Talk) to last revision by Kirshil)
 
(55 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
This article is a step by step guide explaining how to add OpenVZ functionality to Centos 4.4 LiveCD.
 
This article is a step by step guide explaining how to add OpenVZ functionality to Centos 4.4 LiveCD.
 
As a result, you will have a LiveCD, that can boot in various OpenVZ kernels (rhel4-2.6.9, 2.6.18,
 
As a result, you will have a LiveCD, that can boot in various OpenVZ kernels (rhel4-2.6.9, 2.6.18,
rhel5-2.6.18, 2.6.20) and in original CentOS kernel also. After booting from CD, you will be able  
+
rhel5-2.6.18, 2.6.20) and in original CentOS kernel also. After booting from the CD, you will be able  
 
to create [[VE]]s, start them, enter them - i.e. perform [[Getting started with OpenVZ live CD | common OpenVZ activity]].
 
to create [[VE]]s, start them, enter them - i.e. perform [[Getting started with OpenVZ live CD | common OpenVZ activity]].
  
Line 11: Line 11:
 
automatize this process. ;)
 
automatize this process. ;)
  
Actually you need to know the following general actions - everything else is only technical details:
+
Actually you need to accomplish the following general actions - everything else is only technical details:
* Get a copy of CentOS 4.4 LiveCD iso image  
+
* Get a copy of CentOS 4.4 LiveCD image  
* Use <tt>unsquashfs</tt> tool to extract the contents of *.mo files from CD and construct booted LiveCD root filesystem from extracted files
+
* Use <tt>unsquashfs</tt> tool to extract the contents of *.mo files, located on the image
* Create OpenVZ kernel packages  with <tt>squashfs</tt> and <tt>unionfs</tt> support
+
* Construct '''booted''' LiveCD root filesystem from extracted files
* Get OpenVZ user-space tools packages: vzctl, vzquota, vzpkg, ... and some precreated templates
+
* Create OpenVZ kernel RPM-packages  with <tt>squashfs</tt> support
* Remove some packages from booted LiveCD root filesystem (conflicting with OpenVZ kernel packages and unnecessary packages to free some space)
+
* Compile <tt>unionfs</tt> kernel modules against OpenVZ kernels
* Install OpenVZ kernels, tools and templates in LiveCD root filesystem
+
* Get OpenVZ tools RPM-packages: vzctl, vzquota, vzpkg, etc. Get template metadatas and precreated templates
* Using <tt>mksquashfs</tt> tool compress booted CD root filesystem
+
* Remove some RPM-packages from '''booted''' LiveCD root filesystem (conflicting with OpenVZ kernel packages and redundant packages - to free some disk space)
 +
* Install OpenVZ kernels, tools and templates in '''booted''' LiveCD root filesystem
 +
* Using <tt>mksquashfs</tt> tool compress '''booted''' CD root filesystem
 
* Modify initrd image: add there kernel modules for OpenVZ kernels
 
* Modify initrd image: add there kernel modules for OpenVZ kernels
 
* Introduce small modifications to isolinux configuration file
 
* Introduce small modifications to isolinux configuration file
Line 24: Line 26:
  
 
== Technical details ==
 
== Technical details ==
This part contents detailed description of LiveCD creation. I've removed the output of some
+
This part contains detailed description of LiveCD creation. I've removed the output of some
 
commands in order to simplify the reading.
 
commands in order to simplify the reading.
 +
 +
Note: before proceed with commands below, one needs to disable SE Linux on the host, otherwise rpm operations
 +
in cdroot-ed environment described later on will fail due to inability to restore security file context. So, please,
 +
do:
 +
<pre>
 +
# setenforce 0
 +
</pre>
 +
This will put SE linux in so-called 'permissive' mode, where all the blockers will be substituted by warnings.
 +
If you wanna get rid of the warnings (rather annoying), the only way to do is to edit <tt>/etc/sysconfig/selinux</tt>
 +
to read <tt>SELINUX=disabled</tt> and then reboot the host.
  
 
=== Creating booted LiveCD root filesystem ===
 
=== Creating booted LiveCD root filesystem ===
Line 35: Line 47:
 
</pre>
 
</pre>
  
First of all we need some raw material - CentOS 4.4 LiveCD:
+
First of all we need some raw material - CentOS 4.4 LiveCD image:
 
<pre>
 
<pre>
 
# wget http://mirror.nsc.liu.se/CentOS/4.4/isos/i386/CentOS-4.4-i386-LiveCD.iso
 
# wget http://mirror.nsc.liu.se/CentOS/4.4/isos/i386/CentOS-4.4-i386-LiveCD.iso
Line 55: Line 67:
  
 
You can see, that <tt>ovz-livecd/base/</tt> directory owns several *.mo files - images of squashfs filesystem,
 
You can see, that <tt>ovz-livecd/base/</tt> directory owns several *.mo files - images of squashfs filesystem,
and each image holds one directory of booted LiveCD root filesystem: <tt>bin</tt>, <tt>usr</tt> and so on:
+
and each image holds one directory of '''booted''' LiveCD root filesystem: <tt>bin</tt>, <tt>usr</tt> and so on:
 
<pre>
 
<pre>
 
# ls -1 ovz-livecd/base
 
# ls -1 ovz-livecd/base
Line 61: Line 73:
 
</pre>
 
</pre>
  
In order to decompress these images we need <tt>squashfs</tt> tools. The source archive of <tt>squashfs</tt> contents
+
In order to decompress these images we need <tt>unsquashfs</tt> tool. <tt>squashfs</tt> source archive contains
patch for the kernel (to be able mount <tt>squashfs</tt>), and user-space tools for compressing/decomprassing usual filesytem
+
patches for various kernel versions (to be able to mount <tt>squashfs</tt>) and user-space tools for compressing usual filesytem into <tt>squashfs</tt> filesystem and for decompressing it back. Well, we don't need kernel patches right now (but we will need them later), we only want to install <tt>squashfs</tt> user-space tools.
into squashfs filesystem. Well, we don't need the kernel patch right now (but we will need it later), we only want to install <tt>squashfs</tt> user-space tools:
+
Here is '''an important''' note. We want, our LiveCD to be able to boot original Centos 4.4 kernel. This original kernel doesn't support
 +
<tt>squashfs</tt> of version <tt>3</tt>. Consequently we use <tt>sqashfs2.2-r2</tt> for <tt>mksquasfs</tt> tool, but for <tt>unsquashfs</tt> tool and later for compiling fresh OpenVZ kernels with <tt>squashfs</tt> support we use <tt>sqashfs3.2-r2</tt> sources/patches, because they are back compatible with <tt>sqashfs v2</tt>.
 
<pre>
 
<pre>
# wget wget http://switch.dl.sourceforge.net/sourceforge/squashfs/squashfs3.2-r2.tar.gz
+
# wget http://belnet.dl.sourceforge.net/sourceforge/squashfs/squashfs2.2-r2.tar.gz
 +
# tar xzf squashfs2.2-r2.tar.gz
 +
# cd squashfs2.2-r2/squashfs-tools/
 +
# make
 +
cc mksquashfs.o read_fs.o sort.o -lz -o mksquashfs
 +
# cp mksquashfs /usr/bin/
 +
 
 +
# cd -
 +
# wget http://belnet.dl.sourceforge.net/sourceforge/squashfs/squashfs3.2-r2.tar.gz
 
# tar xzf squashfs3.2-r2.tar.gz
 
# tar xzf squashfs3.2-r2.tar.gz
 
# cd squashfs3.2-r2/squashfs-tools/
 
# cd squashfs3.2-r2/squashfs-tools/
Line 71: Line 92:
 
cc mksquashfs.o read_fs.o sort.o -lz -lpthread -lm -o mksquashfs
 
cc mksquashfs.o read_fs.o sort.o -lz -lpthread -lm -o mksquashfs
 
cc unsquashfs.o -lz -o unsquashfs
 
cc unsquashfs.o -lz -o unsquashfs
# cp mksquashfs /usr/bin/
 
 
# cp unsquashfs /usr/bin/
 
# cp unsquashfs /usr/bin/
# cd ~/workbench
+
# cd -
 
</pre>
 
</pre>
 
Well, having the intruments in hand, why not extract the images? Create a directory, called <tt>livecd-root</tt>,
 
Well, having the intruments in hand, why not extract the images? Create a directory, called <tt>livecd-root</tt>,
that will store the root filesystem for booted LiveCD. I.e. it will content <tt>/bin</tt>, <tt>/sbin</tt>,
+
that will store the root filesystem of '''booted''' LiveCD. I.e. it will contain <tt>/bin</tt>, <tt>/sbin</tt>,
 
<tt>/etc</tt> and other directories, which you will see after booting the LiveCD. Unfortunately <tt>unsquashfs</tt>
 
<tt>/etc</tt> and other directories, which you will see after booting the LiveCD. Unfortunately <tt>unsquashfs</tt>
 
tool is not very convenient, and I found only this way to use it:
 
tool is not very convenient, and I found only this way to use it:
Line 82: Line 102:
 
# mkdir livecd-root
 
# mkdir livecd-root
  
# unsquashfs ovz-livecd/base/bin.mo
+
# for f in bin etc lib root sbin usr var; do \
# mv squashfs-root/bin/ livecd-root/
+
    unsquashfs ovz-livecd/base/$f.mo; \
# rm -rf squashfs-root
+
    mv squashfs-root/$f/ livecd-root; \
 
+
    rm -rf squashfs-root; \
# unsquashfs ovz-livecd/base/etc.mo
+
  done
# mv squashfs-root/etc/ livecd-root/
+
</pre>
# rm -rf squashfs-root
 
 
 
# unsquashfs ovz-livecd/base/lib.mo
 
# mv squashfs-root/lib/ livecd-root/
 
# rm -rf squashfs-root
 
 
 
# unsquashfs ovz-livecd/base/root.mo  
 
# mv squashfs-root/root/ livecd-root/
 
# rm -rf squashfs-root
 
 
 
# unsquashfs ovz-livecd/base/sbin.mo
 
# mv squashfs-root/sbin/ livecd-root/
 
# rm -rf squashfs-root
 
  
# unsquashfs ovz-livecd/base/usr.mo    # this action takes some time, be patient ;)
+
Be patient, this will take a few minutes.
# mv squashfs-root/usr/ livecd-root/
 
# rm -rf squashfs-root
 
  
# unsquashfs ovz-livecd/base/var.mo
+
Yet we have nearly complete root filesystem of '''booted''' LiveCD in <tt>livecd-root</tt> directory. However some things
# mv squashfs-root/var/ livecd-root/
 
# rm -rf squashfs-root
 
</pre>
 
Yet we have nearly complete root filesystem of booted LiveCD in <tt>livecd-root</tt> directory. However some things
 
 
are missing, and I guess you know what are they: <tt>/dev</tt>, <tt>/proc</tt>, <tt>/sys</tt>, <tt>/tmp</tt> and <tt>/boot</tt> directories. We need them, because some tools (<tt>rpm</tt>, <tt>bash</tt>, ...) will not work without them in chrooted environment later.
 
are missing, and I guess you know what are they: <tt>/dev</tt>, <tt>/proc</tt>, <tt>/sys</tt>, <tt>/tmp</tt> and <tt>/boot</tt> directories. We need them, because some tools (<tt>rpm</tt>, <tt>bash</tt>, ...) will not work without them in chrooted environment later.
 
<pre>
 
<pre>
# mkdir root/dev
+
# cd livecd-root
# cp -a /dev/* root/dev
+
# mkdir dev
# mkdir root/proc
+
# cp -a /dev/* dev/
# mount -t proc proc root/proc/
+
# mkdir proc
# mkdir root/sys
+
# mount -t proc proc proc/
# mount -t sysfs sysfs root/sys/
+
# mkdir sys
# mkdir root/tmp
+
# mount -t sysfs sysfs sys/
# mkdir root/boot
+
# mkdir tmp
 +
# mkdir boot
 +
# cd -
 
</pre>
 
</pre>
  
That's it. We have booted LiveCD root filesystem in <tt>livecd-root</tt> directory.
+
That's it. We have '''bootstrapped''' LiveCD root filesystem in <tt>livecd-root</tt> directory.
 
Now we can <tt>chroot</tt> in this directory and install all needed packages. But what
 
Now we can <tt>chroot</tt> in this directory and install all needed packages. But what
 
packages do we need?
 
packages do we need?
Line 130: Line 133:
 
* OpenVZ template metadatas: centos-4, fc-3, fc4 and fc-5
 
* OpenVZ template metadatas: centos-4, fc-3, fc4 and fc-5
 
* Some OpenVZ precreated templates
 
* Some OpenVZ precreated templates
* Some CentOS tools updates: sqlite package
+
* Some CentOS tool updates: sqlite package
 
Tools and templates related packages can be simply downloaded and installed.
 
Tools and templates related packages can be simply downloaded and installed.
 
But it is not so easy with kernel. Usual OpenVZ kernel won't work: it doesn't
 
But it is not so easy with kernel. Usual OpenVZ kernel won't work: it doesn't
support <tt>squashfs</tt>. The next chapter explains how to create RPM package with
+
support <tt>squashfs</tt>. The next chapter explains how to create RPM-package with
 
OpenVZ kernel, that supports <tt>squashfs</tt>
 
OpenVZ kernel, that supports <tt>squashfs</tt>
  
=== Building OpenVZ kernels and kernel modules ===
+
=== Building OpenVZ kernels ===
Let's consider, how to build <tt>2.6.18</tt> OpenVZ kernel RPM package with squashfs support.
+
Let's consider, how to build <tt>2.6.18</tt> OpenVZ kernel RPM-package with <tt>squashfs</tt> support.
 
First  download and install OpenVZ source RPM:
 
First  download and install OpenVZ source RPM:
 
<pre>
 
<pre>
# wget http://download.openvz.org/kernel/branches/2.6.18/028stab033.1/kernel-2.6.18-ovz028stab033.1.src.rpm
+
# wget http://download.openvz.org/kernel/branches/2.6.18/028stab035.1/kernel-2.6.18-ovz028stab035.1.src.rpm
# rpm -ivh kernel-2.6.18-ovz028stab033.1.src.rpm
+
# rpm -ivh kernel-2.6.18-ovz028stab035.1.src.rpm
 
</pre>
 
</pre>
<tt>squashfs</tt> sources content the patch against <tt>2.6.18</tt>. Let's copy this patch to RPM build directory:
+
<tt>squashfs</tt> sources contain the patch against <tt>2.6.18</tt>. Let's copy this patch to RPM build directory:
 
<pre>
 
<pre>
 
# cp squashfs3.2-r2/kernel-patches/linux-2.6.18/squashfs3.2-patch /usr/src/redhat/SOURCES/
 
# cp squashfs3.2-r2/kernel-patches/linux-2.6.18/squashfs3.2-patch /usr/src/redhat/SOURCES/
 
</pre>
 
</pre>
Now we need to modify the spec-file in order to add patch to build process. Below is the changes you should add in diff-based format:
+
Now we need to modify the spec-file (<tt>/usr/src/redhat/SPECS/kernel-ovz.spec</tt>) in order to add <tt>squashfs</tt> patch to build process. Below are the changes you should introduce in diff format.
 +
The meaning of changes is transparent: firstly, do not build UP, enterprise and debug kernels (our LiveCD will be
 +
based on SMP kernel) and secondly, apply <tt>squashfs</tt> patch before compilation.
 
<pre>
 
<pre>
 +
--- kernel-ovz.spec.orig        2007-06-09 11:40:15.000000000 +0400
 +
+++ kernel-ovz.spec    2007-06-28 17:42:41.000000000 +0400
 
@@ -5,13 +5,13 @@ Summary: The Linux kernel (the core of t
 
@@ -5,13 +5,13 @@ Summary: The Linux kernel (the core of t
  
Line 170: Line 177:
 
  Patch2106: linux-2.6.18-drbd-8.0.2-8.0.3.patch
 
  Patch2106: linux-2.6.18-drbd-8.0.2-8.0.3.patch
  
+# squashfs
+
+# squashfs3.2-patch
 
+Patch2107: squashfs3.2-patch
 
+Patch2107: squashfs3.2-patch
 
+
 
+
Line 183: Line 190:
 
+
 
+
 
  # ----------------------------------------------------------------------------
 
  # ----------------------------------------------------------------------------
 +
 +
mkdir configs
 +
</pre>
 +
We need one more tiny hack in order RPM-build process to walk smoothly.
 +
There are config files for various OpenVZ kernel flavours
 +
in <tt>/usr/src/redhat/SOURCES</tt> directory:
 +
<pre>
 +
# ls /usr/src/redhat/SOURCES/*.config.ovz
 +
/usr/src/redhat/SOURCES/kernel-2.6.18-i686.config.ovz            /usr/src/redhat/SOURCES/kernel-2.6.18-ppc64.config.ovz
 +
/usr/src/redhat/SOURCES/kernel-2.6.18-i686-enterprise.config.ovz  /usr/src/redhat/SOURCES/kernel-2.6.18-x86_64.config.ovz
 +
/usr/src/redhat/SOURCES/kernel-2.6.18-i686-smp.config.ovz        /usr/src/redhat/SOURCES/kernel-2.6.18-x86_64-smp.config.ovz
 +
/usr/src/redhat/SOURCES/kernel-2.6.18-ia64.config.ovz
 +
</pre>
 +
'''All''' these files should contain information about <tt>squashfs</tt>.
 +
In fact, only two additional lines are necessary:
 +
<pre>
 +
CONFIG_SQUASHFS=m
 +
CONFIG_SQUASHFS_EMBEDDED=n
 +
</pre>
 +
I did it with the following command:
 +
<pre>
 +
# cd /usr/src/redhat/SOURCES/
 +
# echo "CONFIG_SQUASHFS=m
 +
> CONFIG_SQUASHFS_EMBEDDED=n" | tee -a *.config.ovz
 +
CONFIG_SQUASHFS=m
 +
CONFIG_SQUASHFS_EMBEDDED=n
 
</pre>
 
</pre>
Building the RPM:
+
After all efforts, building the RPM-package is very easy:
 
<pre>
 
<pre>
 +
# cd /usr/src/redhat/SPECS/
 
# rpmbuild --target=i686 -ba kernel-ovz.spec
 
# rpmbuild --target=i686 -ba kernel-ovz.spec
 
</pre>
 
</pre>
If everything goes ok, then you will have a ready rpm kernel:
+
If everything goes smoothly, in several tens of minutes depending on the power of your node you willl have OpenVZ kernel RPM-package with <tt>squashfs</tt> support. Let's copy it to our local directory <tt>to_booted_livecd/kernels/</tt>:
 
<pre>
 
<pre>
# ls -l
+
# cd ~/workbench/
 +
# ls /usr/src/redhat/RPMS/i686/
 +
kernel-smp-2.6.18-ovz028stab035.1.i686.rpm
 +
# mkdir -p to_booted_livecd/kernels
 +
# cp /usr/src/redhat/RPMS/i686/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm to_booted_livecd/kernels/
 
</pre>
 
</pre>
 +
OpenVZ LiveCD supports several kernels. So, the actions mentioned above should be accomplished to all kernels,
 +
that you need on LiveCD. In this article I skip the description of building RPMS for all kernels. All description
 +
below are based on <tt>028stab035</tt> kernel, that has been just build above. But some important differences in building
 +
kernel of version other then 2.6.18 one is summarized below.
 +
==== Peculiarities of building 2.6.9-rhel4 ====
 +
* You should also include this patch (http://git.openvz.org/?p=linux-2.6.18-openvz;a=commit;h=681d0b65dedd54d8c716cef56418ac1a478f94da)
 +
in building process. Hopefully will be fixed in <tt>2.6.9-044stab044.8</tt> kernel.
 +
* Use <tt>unionfs 1.0.14</tt>, but not <tt>unionfs 1.1.5</tt>, as offered at its official site!
 +
* Small problems while building <tt>unionfs</tt>: arguments of <tt>vfs_permissions()</tt>. Just add <tt>NULL</tt>
 +
to appropriate places.
 +
 +
==== Peculiarities of building 2.6.18-rhel5 ====
 +
* Note, <tt>squashfs</tt> patch is already there, but you should use <tt>3.2</tt>! :)
 +
* While building <tt>unionfs</tt> some small fixes: <tt>i_blksize</tt> field is missing in <tt>inode</tt> structure, <tt>ino_t</tt> should be changed to <tt>u64</tt> in some functions.
 +
 +
==== Peculiarities of building 2.6.20 ====
 +
* Only <tt>unionfs</tt> <tt>2.0</tt> is available for 2.6.20 kernel and it is distributed as a patch. Consequently modify <tt>spec</tt>-file and configs while building the kernel
 +
* <tt>unionfs</tt> <tt>2.0</tt> doesn't support unionctl. Consequently you have to modify <tt>liblinuxlive</tt> file on <tt>initrd</tt> image to mount all branches at once, but don't add them by ioctls.
  
== Installing RPMs ==
+
=== Unionfs kernel module ===
We have the RPM package for the kernel. But we also need
+
In most present-day LiveCDs <tt>unionfs</tt> filesystem is used. This filesystem "can appear to merge the contents of several directories, while keeping their physical content separate". For example, after the command
 
<pre>
 
<pre>
<
+
# mount -t unionfs -o dirs=/tmp/changes=rw:/cdrom=ro unionfs /union
 +
</pre>
 +
you will see merged contents of <tt>/cdrom</tt> and <tt>/tmp/changes</tt> directories in <tt>/union</tt> directory.
 +
At that, <tt>cdrom</tt> will be read-only. I.e. if you will write to some file in <tt>/union</tt> directory,
 +
that resides in <tt>/cdrom</tt> in fact, this modified file will be automatically moved to <tt>/tmp/changes</tt>.
 +
Such useful property is logically used in LiveCDs: <tt>unionfs</tt> merges cdrom contents (that is read-only)
 +
and temporary in-memory filesystem.
 +
In order to add support of <tt>unionfs</tt> filesystem to OpenVZ kernel we need to compile <tt>unionfs</tt> kernel
 +
module against used kernel. Download the sources of unionfs:
 +
<pre>
 +
# wget ftp://ftp.filesystems.org/pub/unionfs/unionfs-1.x/unionfs-1.4.tar.gz
 +
# tar xzf unionfs-1.4.tar.gz
 +
</pre>
 +
In previous chapter we have created an RPM-package with OpenVZ kernel. We should install it: because
 +
this packages contents kernel headers, required for building kernel modules.
 +
<pre>
 +
# rpm -ivh to_booted_livecd/kernels/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm
 +
Preparing...                ########################################### [100%]
 +
  1:kernel-smp            ########################################### [100%]
 +
</pre>
 +
Now compile <tt>unionfs</tt>:
 +
<pre>
 +
# cd unionfs-1.4
 +
# echo 'LINUXSRC=/lib/modules/2.6.18-ovz028stab035.1-smp/build/
 +
> TOPINC=-I$(LINUXSRC)/include
 +
> MODDIR=/lib/modules/2.6.18-ovz028stab035.1-smp/' > fistdev.mk
 +
# make
 +
# ls unionfs.ko
 +
unionfs.ko
 +
# cd ..
 +
</pre>
 +
Later we will place <tt>unionfs.ko</tt> to appropriate place in '''booted''' LiveCD filesystem.
 +
At the moment copy it to local <tt>to_booted_livecd/unionfs-modules/2.6.18/</tt> directory:
 +
<pre>
 +
# mkdir -p to_booted_livecd/unionfs-modules/2.6.18/
 +
# cp unionfs-1.4/unionfs.ko to_booted_livecd/unionfs-modules/2.6.18/
 +
</pre>
 +
I repeat once again: if you want to use several kernels on LiveCD - the operation above should
 +
be repeated for each wishful kernel.
 +
 
 +
=== OpenVZ tools and templates. CentOS updates ===
 +
We have now the kernel RPM-package. But we also need OpenVZ tools RPM-packages.
 +
It is much easier with them: just download these packages from http://openvz.org:
 +
<pre>
 +
# mkdir to_booted_livecd/tools
 +
# cd to_booted_livecd/tools/
  
To be continued...
+
# wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-3.0.16-1.i386.rpm
 +
# wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-lib-3.0.16-1.i386.rpm
 +
# wget http://download.openvz.org/utils/vzquota/3.0.9/vzquota-3.0.9-1.i386.rpm
 +
# wget http://download.openvz.org/template/utils/vzpkg/2.7.0-18/vzpkg-2.7.0-18.noarch.rpm
 +
# wget http://download.openvz.org/template/utils/vzrpm43/current/vzrpm43-4.3.3-7_nonptl.6.i386.rpm
 +
# wget http://download.openvz.org/template/utils/vzrpm43/current/vzrpm43-python-4.3.3-7_nonptl.6.i386.rpm
 +
# wget http://download.openvz.org/template/utils/vzrpm44/current/vzrpm44-4.4.1-22.5.i386.rpm
 +
# wget http://download.openvz.org/template/utils/vzrpm44/current/vzrpm44-python-4.4.1-22.5.i386.rpm
 +
# wget http://download.openvz.org/template/utils/vzyum/current/vzyum-2.4.0-11.noarch.rpm
  
 +
# mkdir ../template-metadatas
 +
# cd ../template-metadatas
  
 +
# wget http://download.openvz.org/template/metadata/centos-4/vztmpl-centos-4-2.0-2.i386.rpm
 +
# wget http://download.openvz.org/template/metadata/fedora-core-3/vztmpl-fedora-core-3-2.0-2.i386.rpm
 +
# wget http://download.openvz.org/template/metadata/fedora-core-4/vztmpl-fedora-core-4-2.0-2.i386.rpm
 +
# wget http://download.openvz.org/template/metadata/fedora-core-5/vztmpl-fedora-core-5-2.0-2.i386.rpm
  
 +
# mkdir ../precreated
 +
# cd ../precreated
  
=== Obrezki ===
+
# wget http://download.openvz.org/template/precreated/centos-4-i386-minimal.tar.gz
TODO
+
# wget http://download.openvz.org/template/precreated/debian-3.1-i386-minimal.tar.gz
rpms/
+
# wget http://download.openvz.org/template/precreated/fedora-core-5-i386-minimal.tar.gz
rpms/kernel
 
rpms/tools
 
rpms/update
 
  
Uninstall <tt>lsctp-tools</tt> package, because these tools they need kernel version prior to 2.6.10:
+
# cd ~/workbench/
 +
</pre>
 +
As you understand, you can add other precreated templates to LiveCD. The main thing here is that usual CD is
 +
not rubber and precreated templates are rather big! ;)
 +
 
 +
Centos 4.4 LiveCD has a serios problem: <tt>sqlite</tt> package is broken and consequently
 +
<tt>yum</tt> doesn't work. In order to fix this problem we need updated <tt>sqlite</tt> package:
 
<pre>
 
<pre>
rpm -e lksctp-tools-1.0.2-6.4E.1.i386
+
# mkdir to_booted_livecd/updates
 +
# cd to_booted_livecd/updates
 +
# wget http://centos.arcticnetwork.ca/4.5/os/i386/CentOS/RPMS/sqlite-3.3.6-2.i386.rpm
 +
# cd ~/workbench/
 
</pre>
 
</pre>
Uninstall kernel-module-xfs package, because these modules are compiled against original
+
 
 +
=== Install all these finally ===
 +
We have all components that will be on '''booted''' LiveCD ready. Let's install them
 +
in <tt>chrooted</tt> environment. But before uninstall <tt>lsctp-tools</tt> package,
 +
because these tools need kernel version prior to 2.6.10. Then update <tt>sqlite</tt>
 +
package. Also we need to remove some other packages in order to free some disk space:
 +
I've removed <tt>openoffice.org</tt>, <tt>gimp</tt>, <tt>dia</tt>, <tt>gnome-games</tt>,
 +
<tt>tetex</tt>, <tt>latex</tt>, <tt>k3b</tt> and printing related packages:
 
<pre>
 
<pre>
rpm -e kernel-module-xfs-2.6.9-42.livecd.c4-0.1-3.i686
+
# cp -r to_booted_livecd/ livecd-root/tmp/
 +
# chroot livecd-root/
 +
 
 +
# rpm -e lksctp-tools-1.0.2-6.4E.1.i386
 +
 
 +
# rpm -e openoffice.org-libs openoffice.org
 +
# rpm -e gimp-data-extras xsane-gimp gimp-print-plugin gimp-print-utils
 +
# rpm -e gimp gimp-print gimp-help gimp-gap
 +
# rpm -e dia
 +
# rpm -e planner
 +
# rpm -e gnome-games
 +
# rpm -e tetex-latex jadetex xmltex passivetex tetex tetex-afm \
 +
tetex-dvips tetex-fonts tetex-xdvi
 +
# rpm -e k3b
 +
# rpm -e foomatic Omni Omni-foomatic system-config-printer \
 +
system-config-printer-gui hal-cups-utils desktop-printing
 +
 
 +
# cd /tmp/to_booted_livecd/
 +
 
 +
# rpm -Uvh updates/sqlite-3.3.6-2.i386.rpm
 +
 
 +
# rpm -ivh kernels/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm
 +
# rpm -ivh tools/*.rpm
 +
# rpm -ivh template-metadatas/*.rpm
 
</pre>
 
</pre>
  
cd /tmp/rpms/kernel
+
Copy precreated templates to appropriate place:
rpm -ivh kernel-smp-2.6.18-ovz028stab033.1.i686.rpm
+
<pre>
rpm -e kernel-2.6.9-42.livecd.c4
+
# cp  precreated/* /vz/template/cache/
 +
</pre>
 +
Add <tt>unionfs.ko</tt> to other kernel modules and run <tt>depmod</tt>:
 +
<pre>
 +
# mkdir /lib/modules/2.6.18-ovz028stab035.1-smp/kernel/fs/unionfs
 +
# cp unionfs-modules/2.6.18/unionfs.ko /lib/modules/2.6.18-ovz028stab035.1-smp/kernel/fs/unionfs/
 +
# depmod 2.6.18-ovz028stab035.1-smp
 +
# exit
 +
exit
 +
</pre>
 +
While being in chrooted environment bash history was written to <tt>~/.bash_history</tt> file.
 +
Clean it:
 +
<pre>
 +
# rm livecd-root/root/.bash_history
 +
</pre>
 +
Here we have to make one step to workaround [[Checkpointing and live migration]] problem. The thing is that at the moment
 +
chekpointing isn't supported on <tt>unionfs</tt> filesystem ([http://bugzilla.openvz.org/show_bug.cgi?id=606  Bug #606]).
 +
But we want of course checkpoiting to work on LiveCD. If we mount <tt>tmpfs</tt> on <tt>/vz/private/</tt> on '''booted''' LiveCD,
 +
then all newly created VEs reside on <tt>tmpfs</tt> and checkpointing works. Moreover we do not introduce any additional memory usage,
 +
because even without <tt>tmpfs</tt> all VEs reside in RAM (they contain new/updated files for <tt>unionfs</tt>).
 +
 
 +
Create <tt>/vz/private</tt> directory, later, while modifying initrd image we will add the line that will mount <tt>tmpfs</tt>
 +
on this directory:
 +
<pre>
 +
mkdir livecd-root/vz/private/
 +
</pre>
 +
At the moment vzquota doesn't support <tt>tmpfs</tt> ([http://bugzilla.openvz.org/show_bug.cgi?id=558  Bug #558]).
 +
We're working on this problem, but for now '''switching the disk quota off''' in <tt>/etc/vz/vz.conf</tt>: <tt>DISK_QUOTA=no</tt>.
 +
But when disk quota is off <tt>vzmigrate</tt> fails! ;) This is the [http://bugzilla.openvz.org/show_bug.cgi?id=558  Bug #512].
 +
So, don't forget to apply the patch attached to this bug report to <tt>vzmigrate</tt>.
 +
 
 +
'''Booted''' LiveCD filesystem is ready, congratulations!
 +
 
 +
=== Compressing '''booted''' LiveCD filesystem ===
 +
Since <tt>mksquashfs</tt> tool is already installed we can simply use it to create
 +
<tt>*.mo</tt> files:
 +
<pre>
 +
# rm -f ovz-livecd/base/*.mo
 +
# for f in bin etc lib root sbin usr var vz; do \
 +
    mksquashfs livecd-root/$f ovz-livecd/base/$f.mo -keep-as-directory; \
 +
  done
 +
</pre>
 +
 
 +
=== Vmlinux, initrd and isolinux.cfg ===
 +
The next step is to force <tt>isolinux</tt> (cdrom boot loader) to boot OpenVZ kernel.
 +
First, copy kernel image to OpenVZ CD. It is important to give '''short name''' (not greater
 +
than 8 characters) to kernel image, otherwise <tt>isolinux</tt> will not see it.
 +
<pre>
 +
# cp livecd-root/boot/vmlinuz-2.6.18-ovz028stab035.1-smp ovz-livecd/boot/ovz18
 +
</pre>
 +
Modify initrd image. More precisely, we need to add OpenVZ kernel modules there
 +
and apply small patches to <tt>liblinuxlive</tt> file. Initrd image that goes with Centos 4.4 is
 +
of size 40Mb. OpenVZ LiveCD will contain several kernels, so initrd should have several sets of
 +
kernel modules (for each kernel) and 40Mb isn't enough in this case. I created an initrd of size 100Mb.
 +
Don't forget to update <tt>ramdisk_size</tt> in <tt>isolinux.cfg</tt> file later!
 +
 
 +
Creating "big" initrd with original content:
 +
<pre>
 +
# mkdir initrd.d
 +
# cp ovz-livecd/boot/initrd.gz initrd.d/initrd-old.gz
 +
# cd initrd.d/
 +
# mkdir mount-old
 +
# gunzip initrd-old.gz
 +
# mount -o loop initrd-old mount-old/
 +
# dd if=/dev/zero of=initrd bs=1048576 count=100
 +
# mke2fs initrd
 +
initrd is not a block special device.
 +
Proceed anyway? (y,n) y
 +
...
 +
# mkdir mount
 +
# mount -o loop initrd mount/
 +
# cp -a mount-old/* mount/
 +
# umount mount-old/
 +
# rm -rf mount-old initrd-old
 +
</pre>
 +
Copy modules to initrd:
 +
<pre>
 +
# cp -r ../livecd-root/lib/modules/2.6.18-ovz028stab035.1-smp/ mount/lib/modules/
 +
# rm -rf mount/lib/modules/2.6.18-ovz028stab035.1-smp/build                # we don't neet it on initrd image
 +
</pre>
 +
I also delete some not necessary kernel modules from initrd to free space.
 +
They are stil on CD, so init sctipts will be able to load them later.
 +
 
 +
initrd image, that goes with Centos 4.4 LiveCD doesn't load cdrom kernel modules.
 +
It works well for CentOS kernel, because cdrom support is built-in there.
 +
But OpenVZ kernel supports cdrom (usual ide cdrom) via kernel module.
 +
Consequetnly we need to apply the following patch to <tt>liblinuxlive</tt> file:
 +
<pre>
 +
--- liblinuxlive.orig  2006-08-19 08:53:11.000000000 +0400
 +
+++ liblinuxlive        2007-07-02 17:50:34.000000000 +0400
 +
@@ -337,6 +337,9 @@ modprobe_essential_modules()
 +
{
 +
    echolog "starting loop device support"
 +
    modprobe_module loop max_loop=32
 +
+  echolog "starting cdrom support"
 +
+  modprobe_module cdrom
 +
+  modprobe_module ide-cd
 +
    echolog "starting cdrom filesystem support"
 +
    modprobe_module isofs
 +
    echolog "starting squashfs support"
 +
</pre>
 +
As mentioned earlier we want to mount <tt>tmpfs</tt> on /vz/private automatically. Just apply this patch to
 +
<tt>liblinuxlive</tt> file:
 +
<pre>
 +
--- liblinuxlive.orig-fstab    2007-07-02 17:50:34.000000000 +0400
 +
+++ mount/liblinuxlive  2007-07-03 12:33:30.000000000 +0400
 +
@@ -414,6 +414,7 @@ activate_fstab()
 +
    echo "tmpfs            /                tmpfs      defaults        0  0" >$FSTAB
 +
    echo "devpts          /dev/pts        devpts      gid=5,mode=620  0  0" >>$FSTAB
 +
    echo "proc            /proc            proc        defaults        0  0" >>$FSTAB
 +
+  echo "tmpfs            /vz/private      tmpfs      defaults        0  0" >>$FSTAB
 +
 
 +
    # now done by fstab-sync
 +
    # list_cdrom_devices | while read DEVICE; do
 +
</pre>
 +
Finishing actions with initrd:
 +
<pre>
 +
# sync
 +
# umount mount/
 +
# gzip initrd
 +
# cp initrd.gz ../ovz-livecd/boot/initrd.gz
 +
# cd ..
 +
</pre>
 +
Now we should modify <tt>ovz-livecd/isolinux.cfg</tt> file, that controls <tt>isolinux</tt> boot loader.
 +
We inform boot loader, that new kernel was added by introducing the following changes (in diff format):
 +
<pre>
 +
--- ovz-livecd/isolinux.cfg.orig  2006-08-19 08:53:09.000000000 +0400
 +
+++ ovz-livecd/isolinux.cfg    2007-07-02 19:10:51.000000000 +0400
 +
@@ -1,5 +1,5 @@
 +
display boot/splash.cfg
 +
-default linux
 +
+default ovz18
 +
prompt 1
 +
timeout 300
 +
F1 boot/help1.txt
 +
@@ -11,5 +11,9 @@ label linux
 +
kernel boot/vmlinuz
 +
append max_loop=32 vga=791 initrd=boot/initrd.gz init=linuxrc load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=40000 root=/dev/ram0 rw rhgb quiet
 +
 
 +
+label ovz18
 +
+kernel boot/ovz18
 +
+append max_loop=32 vga=791 initrd=boot/initrd.gz init=linuxrc load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=102400 root=/dev/ram0 rw rhgb
 +
+
 +
label memtest
 +
kernel boot/memtest
 +
 
 +
</pre>
 +
 
 +
Also '''don't forget''' to change <tt>ramdisk_size</tt> to 102400 for all kernels including Centos 4.4 default!
 +
 
 +
And I suggest to remove <tt>quiet</tt> parameter, to see what happens: it will help greatly if some problem will happen.
  
cd /tmp/rpms/updates
+
=== Embellishment ===
rpm -Uvh sqlite-3.3.6-2.i386.rpm
+
If you want to release LiveCD to other people some "embelishments" should be added.
  
<tt>
+
First, edit <tt>splash.cfg</tt>, <tt>help1.txt</tt>, <tt>help2.txt</tt> and <tt>help3.txt</tt>
# rpm -ivh vz*
+
files (that are in <tt>~/workbench/ovz-livecd/boot/help</tt> directory) to contain appropriate information.
Preparing...               ########################################### [100%]
+
These messages appear on the LiveCD initial boot: they help to choose kernel and kernel options.
  1:vzrpm44                ########################################### [  8%]
+
 
  2:vzrpm44-python        ########################################### [ 15%]
+
Also, you can note, that CentOS logo appears at this moment. This logo resides in <tt>ovz-livecd/boot/splash.lss</tt>
  3:vzrpm43-python        ########################################### [ 23%]
+
file. I added OpenVZ logo there. The following tools will help you: <tt>lss16toppm</tt>, <tt>ppmtolss16</tt> and
  4:vzrpm43                ########################################### [ 31%]
+
of course <tt>gimp</tt> :)
  5:vzyum                  ########################################### [ 38%]
+
 
  6:vzquota                ########################################### [ 46%]
+
One more thing we change is default desktop background. It resides in <tt>usr/share/backgrounds/images/default.png</tt>.
  7:vzctl-lib              ########################################### [ 54%]
+
I added OpenVZ logo there.
  8:vzctl                  ########################################### [ 62%]
+
 
  9:vzpkg                  ########################################### [ 69%]
+
=== Creating ISO ===
  10:vztmpl-centos-4       ########################################### [ 77%]
+
Centos 4.4 LiveCD creators were very gentle and have placed <tt>make_iso.sh</tt> script on CD.
  11:vztmpl-fedora-core-3  ########################################### [ 85%]
+
Using this script, it is very easy to create iso:
  12:vztmpl-fedora-core-4  ########################################### [ 92%]
+
<pre>
  13:vztmpl-fedora-core-5  ########################################### [100%]
+
# cd ovz-livecd
</tt>
+
# ./make_iso.sh ../CentOS-4.4-i386-OpenVZ-LiveCD.iso
 +
</pre>
 +
 
 +
== Debugging LiveCD ==
 +
If you have trouble with created OpenVZ LiveCD, boot the kernel with
 +
<pre>
 +
vga=0 debug
 +
</pre>
 +
command line options. It will help you to diagnose the problem more precisely.
 +
Also it can be useful to redirect kernel output to serial console:
 +
<pre>
 +
console=ttyS0,115200 console=tty
 +
</pre>
 +
 
 +
See more at [[remote console setup]].
  
 
== External Links ==
 
== External Links ==
Line 250: Line 567:
 
* [http://squashfs.sourceforge.net/ squashfs website]
 
* [http://squashfs.sourceforge.net/ squashfs website]
 
* [http://www.filesystems.org/project-unionfs.html uninonfs official homepage]
 
* [http://www.filesystems.org/project-unionfs.html uninonfs official homepage]
 +
 +
[[Category: HOWTO]]
 +
[[Category: Live CD]]

Latest revision as of 07:55, 26 August 2010

This article is a step by step guide explaining how to add OpenVZ functionality to Centos 4.4 LiveCD. As a result, you will have a LiveCD, that can boot in various OpenVZ kernels (rhel4-2.6.9, 2.6.18, rhel5-2.6.18, 2.6.20) and in original CentOS kernel also. After booting from the CD, you will be able to create VEs, start them, enter them - i.e. perform common OpenVZ activity.

CentOS 5 LiveCD is already announced and as soon as it will be available we'll update the article.

General actions[edit]

It is quite easy to add OpenVZ functionality to existent LiveCD. The most difficult work is already done by the original LiveCD creators. But it requires pretty much hand job. Probably some day some good guy will automatize this process. ;)

Actually you need to accomplish the following general actions - everything else is only technical details:

  • Get a copy of CentOS 4.4 LiveCD image
  • Use unsquashfs tool to extract the contents of *.mo files, located on the image
  • Construct booted LiveCD root filesystem from extracted files
  • Create OpenVZ kernel RPM-packages with squashfs support
  • Compile unionfs kernel modules against OpenVZ kernels
  • Get OpenVZ tools RPM-packages: vzctl, vzquota, vzpkg, etc. Get template metadatas and precreated templates
  • Remove some RPM-packages from booted LiveCD root filesystem (conflicting with OpenVZ kernel packages and redundant packages - to free some disk space)
  • Install OpenVZ kernels, tools and templates in booted LiveCD root filesystem
  • Using mksquashfs tool compress booted CD root filesystem
  • Modify initrd image: add there kernel modules for OpenVZ kernels
  • Introduce small modifications to isolinux configuration file
  • Create new iso image

Technical details[edit]

This part contains detailed description of LiveCD creation. I've removed the output of some commands in order to simplify the reading.

Note: before proceed with commands below, one needs to disable SE Linux on the host, otherwise rpm operations in cdroot-ed environment described later on will fail due to inability to restore security file context. So, please, do:

# setenforce 0

This will put SE linux in so-called 'permissive' mode, where all the blockers will be substituted by warnings. If you wanna get rid of the warnings (rather annoying), the only way to do is to edit /etc/sysconfig/selinux to read SELINUX=disabled and then reboot the host.

Creating booted LiveCD root filesystem[edit]

Create a directory, that will be our "workbench":

# cd ~
# mkdir workbench
# cd workbench

First of all we need some raw material - CentOS 4.4 LiveCD image:

# wget http://mirror.nsc.liu.se/CentOS/4.4/isos/i386/CentOS-4.4-i386-LiveCD.iso
# wget http://mirror.nsc.liu.se/CentOS/4.4/isos/i386/md5sum.livecd
# md5sum --check md5sum.livecd 
CentOS-4.4-i386-LiveCD.iso: OK

Copy the contents of CD to the directory, where we will form the OpenVZ LiveCD filesystem. I named this directory "ovz-livecd":

# mkdir ovz-livecd
# mkdir mount
# mount -o loop -t iso9660 CentOS-4.4-i386-LiveCD.iso mount/
# cp -a mount/* ovz-livecd/
# umount mount
# rm -rf mount

You can see, that ovz-livecd/base/ directory owns several *.mo files - images of squashfs filesystem, and each image holds one directory of booted LiveCD root filesystem: bin, usr and so on:

# ls -1 ovz-livecd/base
bin.mo etc.mo lib.mo root.mo sbin.mo usr.mo var.mo

In order to decompress these images we need unsquashfs tool. squashfs source archive contains patches for various kernel versions (to be able to mount squashfs) and user-space tools for compressing usual filesytem into squashfs filesystem and for decompressing it back. Well, we don't need kernel patches right now (but we will need them later), we only want to install squashfs user-space tools. Here is an important note. We want, our LiveCD to be able to boot original Centos 4.4 kernel. This original kernel doesn't support squashfs of version 3. Consequently we use sqashfs2.2-r2 for mksquasfs tool, but for unsquashfs tool and later for compiling fresh OpenVZ kernels with squashfs support we use sqashfs3.2-r2 sources/patches, because they are back compatible with sqashfs v2.

# wget http://belnet.dl.sourceforge.net/sourceforge/squashfs/squashfs2.2-r2.tar.gz
# tar xzf squashfs2.2-r2.tar.gz
# cd squashfs2.2-r2/squashfs-tools/
# make
cc mksquashfs.o read_fs.o sort.o -lz -o mksquashfs
# cp mksquashfs /usr/bin/

# cd -
# wget http://belnet.dl.sourceforge.net/sourceforge/squashfs/squashfs3.2-r2.tar.gz
# tar xzf squashfs3.2-r2.tar.gz
# cd squashfs3.2-r2/squashfs-tools/
# make
cc mksquashfs.o read_fs.o sort.o -lz -lpthread -lm -o mksquashfs
cc unsquashfs.o -lz -o unsquashfs
# cp unsquashfs /usr/bin/
# cd -

Well, having the intruments in hand, why not extract the images? Create a directory, called livecd-root, that will store the root filesystem of booted LiveCD. I.e. it will contain /bin, /sbin, /etc and other directories, which you will see after booting the LiveCD. Unfortunately unsquashfs tool is not very convenient, and I found only this way to use it:

# mkdir livecd-root

# for f in bin etc lib root sbin usr var; do \
    unsquashfs ovz-livecd/base/$f.mo; \
    mv squashfs-root/$f/ livecd-root; \
    rm -rf squashfs-root; \
  done

Be patient, this will take a few minutes.

Yet we have nearly complete root filesystem of booted LiveCD in livecd-root directory. However some things are missing, and I guess you know what are they: /dev, /proc, /sys, /tmp and /boot directories. We need them, because some tools (rpm, bash, ...) will not work without them in chrooted environment later.

# cd livecd-root
# mkdir dev
# cp -a /dev/* dev/
# mkdir proc
# mount -t proc proc proc/
# mkdir sys
# mount -t sysfs sysfs sys/
# mkdir tmp
# mkdir boot
# cd -

That's it. We have bootstrapped LiveCD root filesystem in livecd-root directory. Now we can chroot in this directory and install all needed packages. But what packages do we need?

  • OpenVZ kernels
  • OpenVZ tools: vzctl, vzquota, vzpkg, vzrpm43, vzrpm44, vzyum
  • OpenVZ template metadatas: centos-4, fc-3, fc4 and fc-5
  • Some OpenVZ precreated templates
  • Some CentOS tool updates: sqlite package

Tools and templates related packages can be simply downloaded and installed. But it is not so easy with kernel. Usual OpenVZ kernel won't work: it doesn't support squashfs. The next chapter explains how to create RPM-package with OpenVZ kernel, that supports squashfs

Building OpenVZ kernels[edit]

Let's consider, how to build 2.6.18 OpenVZ kernel RPM-package with squashfs support. First download and install OpenVZ source RPM:

# wget http://download.openvz.org/kernel/branches/2.6.18/028stab035.1/kernel-2.6.18-ovz028stab035.1.src.rpm
# rpm -ivh kernel-2.6.18-ovz028stab035.1.src.rpm

squashfs sources contain the patch against 2.6.18. Let's copy this patch to RPM build directory:

# cp squashfs3.2-r2/kernel-patches/linux-2.6.18/squashfs3.2-patch /usr/src/redhat/SOURCES/

Now we need to modify the spec-file (/usr/src/redhat/SPECS/kernel-ovz.spec) in order to add squashfs patch to build process. Below are the changes you should introduce in diff format. The meaning of changes is transparent: firstly, do not build UP, enterprise and debug kernels (our LiveCD will be based on SMP kernel) and secondly, apply squashfs patch before compilation.

--- kernel-ovz.spec.orig        2007-06-09 11:40:15.000000000 +0400
+++ kernel-ovz.spec     2007-06-28 17:42:41.000000000 +0400
@@ -5,13 +5,13 @@ Summary: The Linux kernel (the core of t

 # normal/debug/debuginfo
 %define buildnormal 1
-%define builddebug 1
+%define builddebug 0
 %define builddebuginfo 0

 # kernels we want to build
-%define buildup 1
+%define buildup 0
 %define buildsmp 1
-%define buildenterprise 1
+%define buildenterprise 0
 %define buildentnosplit 0

 # versions
@@ -161,6 +161,9 @@ Patch2104: linux-2.6.18-drbd-0.7.22-0.8.
 Patch2105: linux-2.6.18-drbd-8.0.0-8.0.2.patch
 Patch2106: linux-2.6.18-drbd-8.0.2-8.0.3.patch

+# squashfs3.2-patch
+Patch2107: squashfs3.2-patch
+
 # ----------------------------------------------------------------------------

 BuildRoot: %{_tmppath}/kernel-%{KVERREL}-root
@@ -318,6 +321,8 @@ cd linux-%{kversion}
 %patch2105 -p1
 %patch2106 -p1

+%patch2107 -p1
+
 # ----------------------------------------------------------------------------

 mkdir configs

We need one more tiny hack in order RPM-build process to walk smoothly. There are config files for various OpenVZ kernel flavours in /usr/src/redhat/SOURCES directory:

# ls /usr/src/redhat/SOURCES/*.config.ovz
/usr/src/redhat/SOURCES/kernel-2.6.18-i686.config.ovz             /usr/src/redhat/SOURCES/kernel-2.6.18-ppc64.config.ovz
/usr/src/redhat/SOURCES/kernel-2.6.18-i686-enterprise.config.ovz  /usr/src/redhat/SOURCES/kernel-2.6.18-x86_64.config.ovz
/usr/src/redhat/SOURCES/kernel-2.6.18-i686-smp.config.ovz         /usr/src/redhat/SOURCES/kernel-2.6.18-x86_64-smp.config.ovz
/usr/src/redhat/SOURCES/kernel-2.6.18-ia64.config.ovz

All these files should contain information about squashfs. In fact, only two additional lines are necessary:

CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_EMBEDDED=n

I did it with the following command:

# cd /usr/src/redhat/SOURCES/
# echo "CONFIG_SQUASHFS=m
> CONFIG_SQUASHFS_EMBEDDED=n" | tee -a *.config.ovz
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_EMBEDDED=n

After all efforts, building the RPM-package is very easy:

# cd /usr/src/redhat/SPECS/
# rpmbuild --target=i686 -ba kernel-ovz.spec

If everything goes smoothly, in several tens of minutes depending on the power of your node you willl have OpenVZ kernel RPM-package with squashfs support. Let's copy it to our local directory to_booted_livecd/kernels/:

# cd ~/workbench/
# ls /usr/src/redhat/RPMS/i686/
kernel-smp-2.6.18-ovz028stab035.1.i686.rpm
# mkdir -p to_booted_livecd/kernels
# cp /usr/src/redhat/RPMS/i686/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm to_booted_livecd/kernels/

OpenVZ LiveCD supports several kernels. So, the actions mentioned above should be accomplished to all kernels, that you need on LiveCD. In this article I skip the description of building RPMS for all kernels. All description below are based on 028stab035 kernel, that has been just build above. But some important differences in building kernel of version other then 2.6.18 one is summarized below.

Peculiarities of building 2.6.9-rhel4[edit]

in building process. Hopefully will be fixed in 2.6.9-044stab044.8 kernel.

  • Use unionfs 1.0.14, but not unionfs 1.1.5, as offered at its official site!
  • Small problems while building unionfs: arguments of vfs_permissions(). Just add NULL

to appropriate places.

Peculiarities of building 2.6.18-rhel5[edit]

  • Note, squashfs patch is already there, but you should use 3.2! :)
  • While building unionfs some small fixes: i_blksize field is missing in inode structure, ino_t should be changed to u64 in some functions.

Peculiarities of building 2.6.20[edit]

  • Only unionfs 2.0 is available for 2.6.20 kernel and it is distributed as a patch. Consequently modify spec-file and configs while building the kernel
  • unionfs 2.0 doesn't support unionctl. Consequently you have to modify liblinuxlive file on initrd image to mount all branches at once, but don't add them by ioctls.

Unionfs kernel module[edit]

In most present-day LiveCDs unionfs filesystem is used. This filesystem "can appear to merge the contents of several directories, while keeping their physical content separate". For example, after the command

# mount -t unionfs -o dirs=/tmp/changes=rw:/cdrom=ro unionfs /union

you will see merged contents of /cdrom and /tmp/changes directories in /union directory. At that, cdrom will be read-only. I.e. if you will write to some file in /union directory, that resides in /cdrom in fact, this modified file will be automatically moved to /tmp/changes. Such useful property is logically used in LiveCDs: unionfs merges cdrom contents (that is read-only) and temporary in-memory filesystem. In order to add support of unionfs filesystem to OpenVZ kernel we need to compile unionfs kernel module against used kernel. Download the sources of unionfs:

# wget ftp://ftp.filesystems.org/pub/unionfs/unionfs-1.x/unionfs-1.4.tar.gz
# tar xzf unionfs-1.4.tar.gz

In previous chapter we have created an RPM-package with OpenVZ kernel. We should install it: because this packages contents kernel headers, required for building kernel modules.

# rpm -ivh to_booted_livecd/kernels/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm 
Preparing...                ########################################### [100%]
   1:kernel-smp             ########################################### [100%]

Now compile unionfs:

# cd unionfs-1.4
# echo 'LINUXSRC=/lib/modules/2.6.18-ovz028stab035.1-smp/build/
> TOPINC=-I$(LINUXSRC)/include
> MODDIR=/lib/modules/2.6.18-ovz028stab035.1-smp/' > fistdev.mk
# make
# ls unionfs.ko 
unionfs.ko
# cd ..

Later we will place unionfs.ko to appropriate place in booted LiveCD filesystem. At the moment copy it to local to_booted_livecd/unionfs-modules/2.6.18/ directory:

# mkdir -p to_booted_livecd/unionfs-modules/2.6.18/
# cp unionfs-1.4/unionfs.ko to_booted_livecd/unionfs-modules/2.6.18/

I repeat once again: if you want to use several kernels on LiveCD - the operation above should be repeated for each wishful kernel.

OpenVZ tools and templates. CentOS updates[edit]

We have now the kernel RPM-package. But we also need OpenVZ tools RPM-packages. It is much easier with them: just download these packages from http://openvz.org:

# mkdir to_booted_livecd/tools
# cd to_booted_livecd/tools/

# wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-3.0.16-1.i386.rpm
# wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-lib-3.0.16-1.i386.rpm
# wget http://download.openvz.org/utils/vzquota/3.0.9/vzquota-3.0.9-1.i386.rpm
# wget http://download.openvz.org/template/utils/vzpkg/2.7.0-18/vzpkg-2.7.0-18.noarch.rpm
# wget http://download.openvz.org/template/utils/vzrpm43/current/vzrpm43-4.3.3-7_nonptl.6.i386.rpm
# wget http://download.openvz.org/template/utils/vzrpm43/current/vzrpm43-python-4.3.3-7_nonptl.6.i386.rpm
# wget http://download.openvz.org/template/utils/vzrpm44/current/vzrpm44-4.4.1-22.5.i386.rpm
# wget http://download.openvz.org/template/utils/vzrpm44/current/vzrpm44-python-4.4.1-22.5.i386.rpm
# wget http://download.openvz.org/template/utils/vzyum/current/vzyum-2.4.0-11.noarch.rpm

# mkdir ../template-metadatas
# cd ../template-metadatas

# wget http://download.openvz.org/template/metadata/centos-4/vztmpl-centos-4-2.0-2.i386.rpm
# wget http://download.openvz.org/template/metadata/fedora-core-3/vztmpl-fedora-core-3-2.0-2.i386.rpm
# wget http://download.openvz.org/template/metadata/fedora-core-4/vztmpl-fedora-core-4-2.0-2.i386.rpm
# wget http://download.openvz.org/template/metadata/fedora-core-5/vztmpl-fedora-core-5-2.0-2.i386.rpm

# mkdir ../precreated
# cd ../precreated

# wget http://download.openvz.org/template/precreated/centos-4-i386-minimal.tar.gz
# wget http://download.openvz.org/template/precreated/debian-3.1-i386-minimal.tar.gz
# wget http://download.openvz.org/template/precreated/fedora-core-5-i386-minimal.tar.gz

# cd ~/workbench/

As you understand, you can add other precreated templates to LiveCD. The main thing here is that usual CD is not rubber and precreated templates are rather big! ;)

Centos 4.4 LiveCD has a serios problem: sqlite package is broken and consequently yum doesn't work. In order to fix this problem we need updated sqlite package:

# mkdir to_booted_livecd/updates
# cd to_booted_livecd/updates
# wget http://centos.arcticnetwork.ca/4.5/os/i386/CentOS/RPMS/sqlite-3.3.6-2.i386.rpm
# cd ~/workbench/

Install all these finally[edit]

We have all components that will be on booted LiveCD ready. Let's install them in chrooted environment. But before uninstall lsctp-tools package, because these tools need kernel version prior to 2.6.10. Then update sqlite package. Also we need to remove some other packages in order to free some disk space: I've removed openoffice.org, gimp, dia, gnome-games, tetex, latex, k3b and printing related packages:

# cp -r to_booted_livecd/ livecd-root/tmp/
# chroot livecd-root/

# rpm -e lksctp-tools-1.0.2-6.4E.1.i386

# rpm -e openoffice.org-libs openoffice.org
# rpm -e gimp-data-extras xsane-gimp gimp-print-plugin gimp-print-utils
# rpm -e gimp gimp-print gimp-help gimp-gap
# rpm -e dia
# rpm -e planner
# rpm -e gnome-games
# rpm -e tetex-latex jadetex xmltex passivetex tetex tetex-afm \
tetex-dvips tetex-fonts tetex-xdvi
# rpm -e k3b
# rpm -e foomatic Omni Omni-foomatic system-config-printer \
system-config-printer-gui hal-cups-utils desktop-printing

# cd /tmp/to_booted_livecd/

# rpm -Uvh updates/sqlite-3.3.6-2.i386.rpm

# rpm -ivh kernels/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm 
# rpm -ivh tools/*.rpm
# rpm -ivh template-metadatas/*.rpm

Copy precreated templates to appropriate place:

# cp  precreated/* /vz/template/cache/

Add unionfs.ko to other kernel modules and run depmod:

# mkdir /lib/modules/2.6.18-ovz028stab035.1-smp/kernel/fs/unionfs
# cp unionfs-modules/2.6.18/unionfs.ko /lib/modules/2.6.18-ovz028stab035.1-smp/kernel/fs/unionfs/
# depmod 2.6.18-ovz028stab035.1-smp
# exit
exit

While being in chrooted environment bash history was written to ~/.bash_history file. Clean it:

# rm livecd-root/root/.bash_history

Here we have to make one step to workaround Checkpointing and live migration problem. The thing is that at the moment chekpointing isn't supported on unionfs filesystem (Bug #606). But we want of course checkpoiting to work on LiveCD. If we mount tmpfs on /vz/private/ on booted LiveCD, then all newly created VEs reside on tmpfs and checkpointing works. Moreover we do not introduce any additional memory usage, because even without tmpfs all VEs reside in RAM (they contain new/updated files for unionfs).

Create /vz/private directory, later, while modifying initrd image we will add the line that will mount tmpfs on this directory:

mkdir livecd-root/vz/private/

At the moment vzquota doesn't support tmpfs (Bug #558). We're working on this problem, but for now switching the disk quota off in /etc/vz/vz.conf: DISK_QUOTA=no. But when disk quota is off vzmigrate fails! ;) This is the Bug #512. So, don't forget to apply the patch attached to this bug report to vzmigrate.

Booted LiveCD filesystem is ready, congratulations!

Compressing booted LiveCD filesystem[edit]

Since mksquashfs tool is already installed we can simply use it to create *.mo files:

# rm -f ovz-livecd/base/*.mo
# for f in bin etc lib root sbin usr var vz; do \
    mksquashfs livecd-root/$f ovz-livecd/base/$f.mo -keep-as-directory; \
  done

Vmlinux, initrd and isolinux.cfg[edit]

The next step is to force isolinux (cdrom boot loader) to boot OpenVZ kernel. First, copy kernel image to OpenVZ CD. It is important to give short name (not greater than 8 characters) to kernel image, otherwise isolinux will not see it.

# cp livecd-root/boot/vmlinuz-2.6.18-ovz028stab035.1-smp ovz-livecd/boot/ovz18

Modify initrd image. More precisely, we need to add OpenVZ kernel modules there and apply small patches to liblinuxlive file. Initrd image that goes with Centos 4.4 is of size 40Mb. OpenVZ LiveCD will contain several kernels, so initrd should have several sets of kernel modules (for each kernel) and 40Mb isn't enough in this case. I created an initrd of size 100Mb. Don't forget to update ramdisk_size in isolinux.cfg file later!

Creating "big" initrd with original content:

# mkdir initrd.d
# cp ovz-livecd/boot/initrd.gz initrd.d/initrd-old.gz
# cd initrd.d/
# mkdir mount-old
# gunzip initrd-old.gz 
# mount -o loop initrd-old mount-old/
# dd if=/dev/zero of=initrd bs=1048576 count=100
# mke2fs initrd
initrd is not a block special device.
Proceed anyway? (y,n) y
...
# mkdir mount
# mount -o loop initrd mount/
# cp -a mount-old/* mount/
# umount mount-old/
# rm -rf mount-old initrd-old

Copy modules to initrd:

# cp -r ../livecd-root/lib/modules/2.6.18-ovz028stab035.1-smp/ mount/lib/modules/
# rm -rf mount/lib/modules/2.6.18-ovz028stab035.1-smp/build                 # we don't neet it on initrd image

I also delete some not necessary kernel modules from initrd to free space. They are stil on CD, so init sctipts will be able to load them later.

initrd image, that goes with Centos 4.4 LiveCD doesn't load cdrom kernel modules. It works well for CentOS kernel, because cdrom support is built-in there. But OpenVZ kernel supports cdrom (usual ide cdrom) via kernel module. Consequetnly we need to apply the following patch to liblinuxlive file:

--- liblinuxlive.orig   2006-08-19 08:53:11.000000000 +0400
+++ liblinuxlive        2007-07-02 17:50:34.000000000 +0400
@@ -337,6 +337,9 @@ modprobe_essential_modules()
 {
    echolog "starting loop device support"
    modprobe_module loop max_loop=32
+   echolog "starting cdrom support"
+   modprobe_module cdrom
+   modprobe_module ide-cd
    echolog "starting cdrom filesystem support"
    modprobe_module isofs
    echolog "starting squashfs support"

As mentioned earlier we want to mount tmpfs on /vz/private automatically. Just apply this patch to liblinuxlive file:

--- liblinuxlive.orig-fstab     2007-07-02 17:50:34.000000000 +0400
+++ mount/liblinuxlive  2007-07-03 12:33:30.000000000 +0400
@@ -414,6 +414,7 @@ activate_fstab()
    echo "tmpfs            /                tmpfs       defaults         0   0" >$FSTAB
    echo "devpts           /dev/pts         devpts      gid=5,mode=620   0   0" >>$FSTAB
    echo "proc             /proc            proc        defaults         0   0" >>$FSTAB
+   echo "tmpfs            /vz/private      tmpfs       defaults         0   0" >>$FSTAB

    # now done by fstab-sync
    # list_cdrom_devices | while read DEVICE; do

Finishing actions with initrd:

# sync
# umount mount/
# gzip initrd
# cp initrd.gz ../ovz-livecd/boot/initrd.gz
# cd ..

Now we should modify ovz-livecd/isolinux.cfg file, that controls isolinux boot loader. We inform boot loader, that new kernel was added by introducing the following changes (in diff format):

--- ovz-livecd/isolinux.cfg.orig   2006-08-19 08:53:09.000000000 +0400
+++ ovz-livecd/isolinux.cfg     2007-07-02 19:10:51.000000000 +0400
@@ -1,5 +1,5 @@
 display boot/splash.cfg
-default linux
+default ovz18
 prompt 1
 timeout 300
 F1 boot/help1.txt
@@ -11,5 +11,9 @@ label linux
 kernel boot/vmlinuz
 append max_loop=32 vga=791 initrd=boot/initrd.gz init=linuxrc load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=40000 root=/dev/ram0 rw rhgb quiet

+label ovz18 
+kernel boot/ovz18
+append max_loop=32 vga=791 initrd=boot/initrd.gz init=linuxrc load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=102400 root=/dev/ram0 rw rhgb
+
 label memtest
 kernel boot/memtest

Also don't forget to change ramdisk_size to 102400 for all kernels including Centos 4.4 default!

And I suggest to remove quiet parameter, to see what happens: it will help greatly if some problem will happen.

Embellishment[edit]

If you want to release LiveCD to other people some "embelishments" should be added.

First, edit splash.cfg, help1.txt, help2.txt and help3.txt files (that are in ~/workbench/ovz-livecd/boot/help directory) to contain appropriate information. These messages appear on the LiveCD initial boot: they help to choose kernel and kernel options.

Also, you can note, that CentOS logo appears at this moment. This logo resides in ovz-livecd/boot/splash.lss file. I added OpenVZ logo there. The following tools will help you: lss16toppm, ppmtolss16 and of course gimp :)

One more thing we change is default desktop background. It resides in usr/share/backgrounds/images/default.png. I added OpenVZ logo there.

Creating ISO[edit]

Centos 4.4 LiveCD creators were very gentle and have placed make_iso.sh script on CD. Using this script, it is very easy to create iso:

# cd ovz-livecd
# ./make_iso.sh ../CentOS-4.4-i386-OpenVZ-LiveCD.iso

Debugging LiveCD[edit]

If you have trouble with created OpenVZ LiveCD, boot the kernel with

vga=0 debug

command line options. It will help you to diagnose the problem more precisely. Also it can be useful to redirect kernel output to serial console:

console=ttyS0,115200 console=tty

See more at remote console setup.

External Links[edit]