Changes

rhel5-2.6.18, 2.6.20) and in original CentOS kernel also. After booting from the CD, you will be able

Actually you need to know accomplish the following general actions - everything else is only technical details:* Get a copy of CentOS 4.4 LiveCD iso image * Use <tt>unsquashfs</tt> tool to extract the contents of *.mo files from CD and construct , located on the image* Construct '''booted ''' LiveCD root filesystem from extracted files* Create OpenVZ kernel RPM-packages with <tt>squashfs</tt> and support* Compile <tt>unionfs</tt> supportkernel modules against OpenVZ kernels* Get OpenVZ usertools RPM-space tools packages: vzctl, vzquota, vzpkg, etc... Get template metadatas and some precreated templates* Remove some RPM-packages from '''booted ''' LiveCD root filesystem (conflicting with OpenVZ kernel packages and unnecessary redundant packages - to free some disk space)* Install OpenVZ kernels, tools and templates in '''booted''' LiveCD root filesystem* Using <tt>mksquashfs</tt> tool compress '''booted ''' CD root filesystem

This part contents contains detailed description of LiveCD creation. I've removed the output of some

First of all we need some raw material - CentOS 4.4 LiveCDimage:

and each image holds one directory of '''booted''' LiveCD root filesystem: <tt>bin</tt>, <tt>usr</tt> and so on:

In order to decompress these images we need <tt>squashfsunsquashfs</tt> toolstool. The source archive of <tt>squashfs</tt> contentssource archive containspatch patches for the various kernel versions (to be able to mount <tt>squashfs</tt>), and user-space tools for compressing/decomprassing usual filesyteminto <tt>squashfs </tt> filesystemand for decompressing it back. Well, we don't need the kernel patch patches right now (but we will need it them later), we only want to install <tt>squashfs</tt> user-space tools:.Here is '''an important''' note. We want, our LiveCD to be able to boot original Centos 4.4 kernel. This original kernel doesn't support<tt>squashfs</tt> of version <tt>3</tt>. Consequently we use <tt>sqashfs2.2-r2</tt> for <tt>mksquasfs</tt> tool, but for <tt>unsquashfs</tt> tool and later for compiling fresh OpenVZ kernels with <tt>squashfs</tt> support we use <tt>sqashfs3.2-r2</tt> sources/patches, because they are back compatible with <tt>sqashfs v2</tt>.

# wget http://belnet.dl.sourceforge.net/sourceforge/squashfs/squashfs2.2-r2.tar.gz# tar xzf squashfs2.2-r2.tar.gz# cd squashfs2.2-r2/squashfs-tools/# makecc mksquashfs.o read_fs.o sort.o -lz -o mksquashfs# cp mksquashfs /usr/bin/ # cd -# wget http://switchbelnet.dl.sourceforge.net/sourceforge/squashfs/squashfs3.2-r2.tar.gz

# cd ~/workbench-

that will store the root filesystem for of '''booted''' LiveCD. I.e. it will content contain <tt>/bin</tt>, <tt>/sbin</tt>,

# unsquashfs ovz-livecd/base/for f in bin.mo # mv squashfs-etc lib root/bin/ livecd-root/sbin usr var; do \# rm -rf squashfs-root # unsquashfs ovz-livecd/base/etc$f.mo ; \# mv squashfs-root/etc$f/ livecd-root/; \# rm -rf squashfs-root; \ done# unsquashfs ovz-livecd/base/lib.mo # mv squashfs-root/lib/ livecd-root</# rm -rf squashfs-rootpre>

# unsquashfs ovz-livecd/base/rootBe patient, this will take a few minutes.mo # mv squashfs-root/root/ livecd-root/# rm -rf squashfs-root

# unsquashfs ovz-livecd/base/sbin.mo # mv squashfs-root/sbin/ livecd-root/# rm -rf squashfs-root # unsquashfs ovz-livecd/base/usr.mo # this action takes some time, be patient ;)# mv squashfs-root/usr/ livecd-root/# rm -rf squashfs-root # unsquashfs ovz-livecd/base/var.mo# mv squashfs-root/var/ livecd-root/# rm -rf squashfs-root</pre>Yet we have nearly complete root filesystem of '''booted ''' LiveCD in <tt>livecd-root</tt> directory. However some thingsare missing, and I guess you know what are they: <tt>/dev</tt>, <tt>/proc</tt>, <tt>/sys</tt>, <tt>/tmp</tt> and <tt>/boot</tt> directories. We need them, because some tools (<tt>rpm</tt>, <tt>bash</tt>,...) will not work without them in chrooted environment later.

# cd livecd-root# mkdir root/dev# cp -a /dev/* rootdev/dev# mkdir root/proc# mount -t proc proc root/proc/# mkdir root/sys# mount -t sysfs sysfs root/sys/# mkdir root/tmp# mkdir root/boot# cd -

That's it. We have booted '''bootstrapped''' LiveCD root filesystem in <tt>livecd-root</tt> directory.

* OpenVZ kernelkernels

* Some CentOS tools tool updates: sqlite package

support <tt>squashfs</tt>. The next chapter explains how to create RPM -package with

=== Building OpenVZ kernels and kernel modules ===Let's consider, how to build <tt>2.6.18</tt> OpenVZ kernel RPM-package with <tt>squashfs</tt> support.First download and install OpenVZ source RPM:<pre># wget http://download.openvz.org/kernel/branches/2.6.18/028stab035.1/kernel-2.6.18-ovz028stab035.1.src.rpm# rpm -ivh kernel-2.6.18-ovz028stab035.1.src.rpm</pre><tt>squashfs</tt> sources contain the patch against <tt>2.6.18</tt>. Let's copy this patch to RPM build directory:<pre># cp squashfs3.2-r2/kernel-patches/linux-2.6.18/squashfs3.2-patch /usr/src/redhat/SOURCES/</pre>Now we need to modify the spec-file (<tt>/usr/src/redhat/SPECS/kernel-ovz.spec</tt>) in order to add <tt>squashfs</tt> patch to build process. Below are the changes you should introduce in diff format.The meaning of changes is transparent: firstly, do not build UP, enterprise and debug kernels (our LiveCD will bebased on SMP kernel) and secondly, apply <tt>squashfs</tt> patch before compilation.<pre>--- kernel-ovz.spec.orig 2007-06-09 11:40:15.000000000 +0400+++ kernel-ovz.spec 2007-06-28 17:42:41.000000000 +0400@@ -5,13 +5,13 @@ Summary: The Linux kernel (the core of t

Uninstall mkdir configs</pre>We need one more tiny hack in order RPM-build process to walk smoothly.There are config files for various OpenVZ kernel flavours in <tt>lsctp-tools/usr/src/redhat/SOURCES</tt> package, because these tools they need kernel version prior to 2.6.10directory:

rpm # ls /usr/src/redhat/SOURCES/*.config.ovz/usr/src/redhat/SOURCES/kernel-2.6.18-i686.config.ovz /usr/src/redhat/SOURCES/kernel-2.6.18-ppc64.config.ovz/usr/src/redhat/SOURCES/kernel-2.6.18-i686-enterprise.config.ovz /usr/src/redhat/SOURCES/kernel-e lksctp2.6.18-toolsx86_64.config.ovz/usr/src/redhat/SOURCES/kernel-12.6.018-i686-smp.config.ovz /usr/src/redhat/SOURCES/kernel-2.6.18-x86_64-smp.config.ovz/usr/src/redhat/SOURCES/kernel-2.6.4E18-ia64.1config.i386ovz

Uninstall kernel-module-xfs package'''All''' these files should contain information about <tt>squashfs</tt>.In fact, because these modules only two additional lines are compiled against original necessary:

cd /tmp/rpms/kernelrpm -ivh kernel-smp-==== Peculiarities of building 2.6.18-ovz028stab033rhel5 ====* Note, <tt>squashfs</tt> patch is already there, but you should use <tt>3.1.i686.rpm2</tt>! :)rpm -e kernel-2.6.9-42.livecd* While building <tt>unionfs</tt> some small fixes: <tt>i_blksize</tt> field is missing in <tt>inode</tt> structure, <tt>ino_t</tt> should be changed to <tt>u64</tt> in some functions.c4

cd ==== Peculiarities of building 2.6.20 ====* Only <tt>unionfs</tmptt> <tt>2.0</rpms/updatesrpm -Uvh sqlite-3tt> is available for 2.6.320 kernel and it is distributed as a patch.6Consequently modify <tt>spec</tt>-file and configs while building the kernel* <tt>unionfs</tt> <tt>2.i3860</tt> doesn't support unionctl. Consequently you have to modify <tt>liblinuxlive</tt> file on <tt>initrd</tt> image to mount all branches at once, but don't add them by ioctls.rpm

=== Unionfs kernel module ===In most present-day LiveCDs <tt>unionfs</tt> filesystem is used. This filesystem "can appear to merge the contents of several directories, while keeping their physical content separate". For example, after the command<pre># mount -t unionfs -o dirs=/tmp/changes=rw:/cdrom=ro unionfs /union</pre>you will see merged contents of <tt>/cdrom</tt> and <tt>/tmp/changes</tt> directories in <tt>/union</tt> directory. At that, <tt>cdrom</tt> will be read-only. I.e. if you will write to some file in <tt>/union</tt> directory, that resides in <tt>/cdrom</tt> in fact, this modified file will be automatically moved to <tt>/tmp/changes</tt>.Such useful property is logically used in LiveCDs: <tt>unionfs</tt> merges cdrom contents (that is read-only)and temporary in-memory filesystem.In order to add support of <tt>unionfs</tt> filesystem to OpenVZ kernel we need to compile <tt>unionfs</tt> kernelmodule against used kernel. Download the sources of unionfs:<pre># wget ftp://ftp.filesystems.org/pub/unionfs/unionfs-1.x/unionfs-1.4.tar.gz# tar xzf unionfs-1.4.tar.gz</pre>In previous chapter we have created an RPM-package with OpenVZ kernel. We should install it: becausethis packages contents kernel headers, required for building kernel modules.<pre># rpm -ivh vz*to_booted_livecd/kernels/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm

1:vzrpm44 kernel-smp ########################################### [ 8100%] 2</pre>Now compile <tt>unionfs</tt>:vzrpm44<pre># cd unionfs-python ####1.4#echo 'LINUXSRC=/lib/modules/2.6.18-ovz028stab035.1-smp/build/> TOPINC=-I$(LINUXSRC)/include> MODDIR=/lib/modules/2.6.18-ovz028stab035.1-smp/' > fistdev.mk#make#ls unionfs.ko unionfs.ko#cd ..</pre>Later we will place <tt>unionfs.ko</tt> to appropriate place in '''booted''' LiveCD filesystem.At the moment copy it to local <tt>to_booted_livecd/unionfs-modules/2.6.18/</tt> directory:<pre>#mkdir -p to_booted_livecd/unionfs-modules/2.6.18/#cp unionfs-1.4/unionfs.ko to_booted_livecd/unionfs-modules/2.6.18/</pre>I repeat once again: if you want to use several kernels on LiveCD - the operation above shouldbe repeated for each wishful kernel. === OpenVZ tools and templates. CentOS updates ===We have now the kernel RPM-package. But we also need OpenVZ tools RPM-packages.It is much easier with them: just download these packages from http://openvz.org:<pre>#mkdir to_booted_livecd/tools#cd to_booted_livecd/tools/ #wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-3.0.16-1.i386.rpm#wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-lib-3.0.16-1.i386.rpm#wget http://download.openvz.org/utils/vzquota/3.0.9/vzquota-3.0.9-1.i386.rpm#wget http://download.openvz.org/template/utils/vzpkg/2.7.0-18/vzpkg-2.7.0-18.noarch.rpm#wget http://download.openvz.org/template/utils/vzrpm43/current/vzrpm43-4.3.3-7_nonptl.6.i386.rpm########################## [ 15%] 3wget http://download.openvz.org/template/utils/vzrpm43/current/vzrpm43-python -4.3.3-7_nonptl.6.i386.rpm#wget http://download.openvz.org/template/utils/vzrpm44/current/vzrpm44-4.4.1-22.5.i386.rpm#wget http://download.openvz.org/template/utils/vzrpm44/current/vzrpm44-python-4.4.1-22.5.i386.rpm#wget http://download.openvz.org/template/utils/vzyum/current/vzyum-2.4.0-11.noarch.rpm #mkdir ../template-metadatas#cd ../template-metadatas #wget http://download.openvz.org/template/metadata/centos-4/vztmpl-centos-4-2.0-2.i386.rpm#wget http://download.openvz.org/template/metadata/fedora-core-3/vztmpl-fedora-core-3-2.0-2.i386.rpm#wget http://download.openvz.org/template/metadata/fedora-core-4/vztmpl-fedora-core-4-2.0-2.i386.rpm#wget http://download.openvz.org/template/metadata/fedora-core-5/vztmpl-fedora-core-5-2.0-2.i386.rpm #mkdir ../precreated#cd ../precreated #wget http://download.openvz.org/template/precreated/centos-4-i386-minimal.tar.gz#wget http://download.openvz.org/template/precreated/debian-3.1-i386-minimal.tar.gz#wget http://download.openvz.org/template/precreated/fedora-core-5-i386-minimal.tar.gz ############################# [ 23%]cd ~/workbench/</pre>As you understand, you can add other precreated templates to LiveCD. The main thing here is that usual CD is not rubber and precreated templates are rather big! ;)  Centos 4.4 LiveCD has a serios problem: <tt>sqlite</tt> package is broken and consequently<tt>yum</tt> doesn't work. In order to fix this problem we need updated <tt>sqlite</tt> package:vzrpm43 ################<pre>#mkdir to_booted_livecd/updates#cd to_booted_livecd/updates#wget http://centos.arcticnetwork.ca/4.5/os/i386/CentOS/RPMS/sqlite-3.3.6-2.i386.rpm#cd ~/workbench/</pre> === Install all these finally ===We have all components that will be on '''booted''' LiveCD ready. Let's install themin <tt>chrooted</tt> environment. But before uninstall <tt>lsctp-tools</tt> package,because these tools need kernel version prior to 2.6.10. Then update <tt>sqlite</tt>package. Also we need to remove some other packages in order to free some disk space:I've removed <tt>openoffice.org</tt>, <tt>gimp</tt>, <tt>dia</tt>, <tt>gnome-games</tt>,<tt>tetex</tt>, <tt>latex</tt>, <tt>k3b</tt> and printing related packages:<pre>#cp -r to_booted_livecd/ livecd-root/tmp/#chroot livecd-root/ #rpm -e lksctp-tools-1.0.2-6.4E.1.i386 #rpm -e openoffice.org-libs openoffice.org#rpm -e gimp-data-extras xsane-gimp gimp-print-plugin gimp-print-utils#rpm -e gimp gimp-print gimp-help gimp-gap#rpm -e dia#rpm -e planner#rpm -e gnome-games#rpm -e tetex-latex jadetex xmltex passivetex tetex tetex-afm \tetex-dvips tetex-fonts tetex-xdvi#rpm -e k3b#rpm -e foomatic Omni Omni-foomatic system-config-printer \system-config-printer-gui hal-cups-utils desktop-printing #cd /tmp/to_booted_livecd/ #rpm -Uvh updates/sqlite-3.3.6-2.i386.rpm #rpm -ivh kernels/kernel-smp-2.6.18-ovz028stab035.1.i686.rpm #rpm -ivh tools/*.rpm#rpm -ivh template-metadatas/*.rpm</pre> Copy precreated templates to appropriate place:<pre>#cp precreated/* /vz/template/cache/</pre>Add <tt>unionfs.ko</tt> to other kernel modules and run <tt>depmod</tt>:<pre>#mkdir /lib/modules/2.6.18-ovz028stab035.1-smp/kernel/fs/unionfs#cp unionfs-modules/2.6.18/unionfs.ko /lib/modules/2.6.18-ovz028stab035.1-smp/kernel/fs/unionfs/#depmod 2.6.18-ovz028stab035.1-smp#exitexit</pre>While being in chrooted environment bash history was written to <tt>~/.bash_history</tt> file.Clean it:<pre># rm livecd-root/root/.bash_history</pre>Here we have to make one step to workaround [[ 31%Checkpointing and live migration]]problem. The thing is that at the moment 5chekpointing isn't supported on <tt>unionfs</tt> filesystem ([http:vzyum ##################################//bugzilla.openvz.org/show_bug.cgi?id=606 Bug ######### [ 38%606]).But we want of course checkpoiting to work on LiveCD. If we mount <tt>tmpfs</tt> on <tt>/vz/private/</tt> on '''booted''' LiveCD,then all newly created VEs reside on <tt>tmpfs</tt> and checkpointing works. Moreover we do not introduce any additional memory usage,because even without <tt>tmpfs</tt> all VEs reside in RAM (they contain new/updated files for <tt>unionfs</tt>).  6Create <tt>/vz/private</tt> directory, later, while modifying initrd image we will add the line that will mount <tt>tmpfs</tt>on this directory:<pre>mkdir livecd-root/vz/private/</pre>At the moment vzquota ############################doesn't support <tt>tmpfs</tt> ([http://bugzilla.openvz.org/show_bug.cgi?id=558 Bug ############### [ 46%558]). 7We're working on this problem, but for now '''switching the disk quota off''' in <tt>/etc/vz/vz.conf</tt>:vzctl-lib ########################################### <tt>DISK_QUOTA=no</tt>.But when disk quota is off <tt>vzmigrate</tt> fails! ;) This is the [ 54%] 8http:vzctl #############################//bugzilla.openvz.org/show_bug.cgi?id=558 Bug ############## [ 62%512].So, don't forget to apply the patch attached to this bug report to <tt>vzmigrate</tt>. '''Booted''' LiveCD filesystem is ready, congratulations! === Compressing '''booted''' LiveCD filesystem ===Since <tt>mksquashfs</tt> tool is already installed we can simply use it to create 9<tt>*.mo</tt> files:vzpkg <pre>#rm -f ovz-livecd/base/*.mo########################################## [ 69%]for f in bin etc lib root sbin usr var vz; do \ mksquashfs livecd-root/$f ovz-livecd/base/$f.mo -keep-as-directory; \ 10:vztmpldone</pre> === Vmlinux, initrd and isolinux.cfg ===The next step is to force <tt>isolinux</tt> (cdrom boot loader) to boot OpenVZ kernel.First, copy kernel image to OpenVZ CD. It is important to give '''short name''' (not greaterthan 8 characters) to kernel image, otherwise <tt>isolinux</tt> will not see it.<pre># cp livecd-root/boot/vmlinuz-2.6.18-ovz028stab035.1-centossmp ovz-livecd/boot/ovz18</pre>Modify initrd image. More precisely, we need to add OpenVZ kernel modules thereand apply small patches to <tt>liblinuxlive</tt> file. Initrd image that goes with Centos 4.4 isof size 40Mb. OpenVZ LiveCD will contain several kernels, so initrd should have several sets ofkernel modules (for each kernel) and 40Mb isn't enough in this case. I created an initrd of size 100Mb.Don't forget to update <tt>ramdisk_size</tt> in <tt>isolinux.cfg</tt> file later! Creating "big" initrd with original content:<pre>#mkdir initrd.d#cp ovz-livecd/boot/initrd.gz initrd.d/initrd-old.gz#cd initrd.d/#mkdir mount-old#gunzip initrd-old.gz #mount -o loop initrd-old mount-old/#dd if=/dev/zero of=initrd bs=1048576 count=100#mke2fs initrdinitrd is not a block special device.Proceed anyway? (y,n) y...#mkdir mount#mount -o loop initrd mount/#cp -a mount-old/* mount/#umount mount-old/#rm -rf mount-old initrd-old</pre>Copy modules to initrd:<pre>#cp -r ../livecd-root/lib/modules/2.6.18-ovz028stab035.1-smp/ mount/lib/modules/#rm -rf mount/lib/modules/2.6.18-ovz028stab035.1-smp/build ############################ [ 77%]we don't neet it on initrd image</pre>I also delete some not necessary kernel modules from initrd to free space.They are stil on CD, so init sctipts will be able to load them later. initrd image, that goes with Centos 4.4 LiveCD doesn't load cdrom kernel modules.It works well for CentOS kernel, because cdrom support is built-in there.But OpenVZ kernel supports cdrom (usual ide cdrom) via kernel module.Consequetnly we need to apply the following patch to <tt>liblinuxlive</tt> file:<pre>--- liblinuxlive.orig 2006-08-19 08:53:11:vztmpl.000000000 +0400+++ liblinuxlive 2007-fedora07-core02 17:50:34.000000000 +0400@@ -3 337,6 +337,9 @@ modprobe_essential_modules() { echolog "starting loop device support" modprobe_module loop max_loop=32+ echolog "starting cdrom support"+ ########################################### [ 85%]modprobe_module cdrom+ 12modprobe_module ide-cd echolog "starting cdrom filesystem support" modprobe_module isofs echolog "starting squashfs support"</pre>As mentioned earlier we want to mount <tt>tmpfs</tt> on /vz/private automatically. Just apply this patch to<tt>liblinuxlive</tt> file:<pre>--- liblinuxlive.orig-fstab 2007-07-02 17:50:vztmpl34.000000000 +0400+++ mount/liblinuxlive 2007-fedora07-core03 12:33:30.000000000 +0400@@ -4 414,6 +414,7 @@ activate_fstab() echo "tmpfs / tmpfs defaults 0 0" >$FSTAB echo "devpts /dev/pts devpts gid=5,mode=620 0 0" >>$FSTAB echo "proc /proc proc defaults 0 0" >>$FSTAB+ echo "tmpfs /vz/private tmpfs defaults 0 0" >>$FSTAB  #now done by fstab-sync #list_cdrom_devices | while read DEVICE; do</pre>Finishing actions with initrd:<pre>#sync#umount mount/#gzip initrd#cp initrd.gz ../ovz-livecd/boot/initrd.gz##################################### [ 92%]cd ..</pre>Now we should modify <tt>ovz-livecd/isolinux.cfg</tt> file, that controls <tt>isolinux</tt> boot loader.We inform boot loader, that new kernel was added by introducing the following changes (in diff format):<pre>--- ovz-livecd/isolinux.cfg.orig 132006-08-19 08:53:09.000000000 +0400+++ ovz-livecd/isolinux.cfg 2007-07-02 19:10:vztmpl51.000000000 +0400@@ -fedora1,5 +1,5 @@ display boot/splash.cfg-coredefault linux+default ovz18 prompt 1 timeout 300 F1 boot/help1.txt@@ -11,5 +11,9 @@ label linux kernel boot/vmlinuz append max_loop=32 vga=791 initrd=boot/initrd.gz init=linuxrc load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=40000 root=/dev/ram0 rw rhgb quiet +label ovz18 +kernel boot/ovz18+append max_loop=32 vga=791 initrd=boot/initrd.gz init=linuxrc load_ramdisk=1 prompt_ramdisk=0 ramdisk_size=102400 root=/dev/ram0 rw rhgb+ label memtest kernel boot/memtest </pre> Also '''don't forget''' to change <tt>ramdisk_size</tt> to 102400 for all kernels including Centos 4.4 default!  And I suggest to remove <tt>quiet</tt> parameter, to see what happens: it will help greatly if some problem will happen. === Embellishment ===If you want to release LiveCD to other people some "embelishments" should be added. First, edit <tt>splash.cfg</tt>, <tt>help1.txt</tt>, <tt>help2.txt</tt> and <tt>help3.txt</tt>files (that are in <tt>~/workbench/ovz-livecd/boot/help</tt> directory) to contain appropriate information.These messages appear on the LiveCD initial boot: they help to choose kernel and kernel options. Also, you can note, that CentOS logo appears at this moment. This logo resides in <tt>ovz-livecd/boot/splash.lss</tt>file. I added OpenVZ logo there. The following tools will help you: <tt>lss16toppm</tt>, <tt>ppmtolss16</tt> andof course <tt>gimp</tt> :) One more thing we change is default desktop background. It resides in <tt>usr/share/backgrounds/images/default.png</tt>.I added OpenVZ logo there. === Creating ISO ===Centos 4.4 LiveCD creators were very gentle and have placed <tt>make_iso.sh</tt> script on CD.Using this script, it is very easy to create iso:<pre>#cd ovz-livecd########################################## [100%]./make_iso.sh ../CentOS-4.4-i386-OpenVZ-LiveCD.iso</pre> == Debugging LiveCD ==If you have trouble with created OpenVZ LiveCD, boot the kernel with<pre>vga=0 debug</pre>command line options. It will help you to diagnose the problem more precisely.Also it can be useful to redirect kernel output to serial console:<pre>console=ttyS0,115200 console=tty</ttpre> See more at [[remote console setup]].