OpenVZ Virtuozzo Containers Wiki - User contributions [en]

Using veth and brctl for protecting HN and saving IP addresses

2007-07-03T11:40:33Z

Emkravts:

Configuration described below has been suggested by Ugo123. Appreciates.

Consider we are facing the following task:

1) We have limited range of IP adresses granted by ISP.
We want to assign as much granted IPs to VEs as possible.
We do not want to protect VEs from Internet.
2) We want to protect the HN OS (VE0) from Internet and make it possible to manage VEs from VE0 within local area network.

Assume we have a HN with 2 ethernet cards (interfaces eth0 and eth1), OpenVZ kernel 2.6.18-028stab033, vzctl version 3.0.16,
bridge-utils version 1.1. OpenVZ installation process is covered by http://wiki.openvz.org/Quick_installation.

Task can be effectively solved by setting up the configuration presented on Figure 1.

Figure 1: Effective configuration. 10.0.98.96-10.0.98.X - range of IP-adresses granted by ISP, 192.168.1.136 - IP address from LAN

[[Image:fig.jpg]]

Initial ifconfig output of HN is the following:
<pre>
[HN]# ifconfig
eth0 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:34
inet addr:192.168.1.136 Bcast:192.168.3.255 Mask:255.255.252.0
inet6 addr: fe80::230:48ff:fe5b:ab34/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3122 errors:0 dropped:0 overruns:0 frame:0
TX packets:246 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:325879 (318.2 KiB) TX bytes:57278 (55.9 KiB)
Interrupt:20

eth1 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:35
inet addr:192.168.0.32 Bcast:192.168.3.255 Mask:255.255.252.0
inet6 addr: fe80::213:d4ff:fe90:4d50/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:603734 errors:0 dropped:0 overruns:0 frame:0
TX packets:36627 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:21

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1376 errors:0 dropped:0 overruns:0 frame:0
TX packets:1376 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2078718 (1.9 MiB) TX bytes:2078718 (1.9 MiB)
</pre>
Let us pass through the setup process step by step.

1) Create 2 VEs on the HN as described in http://download.openvz.org/doc/OpenVZ-Users-Guide.pdf.
For testing purposes I've used opensuse-10 precreated template from openvz.org:
<pre>
[HN]# cd /vz/template/cache
[HN]# wget http://download.openvz.org/template/precreated/opensuse-10-i386-default.tar.gz
</pre>
Create VE 101 and assign it one of the IP adresses obtained from ISP:
<pre>
[HN]# vzctl create 101 --ostemplate opensuse-10-i386-default --ipadd 10.0.98.96
[HN]# vzctl set 101 --userpasswd root:XXX --save
</pre>
And do the same for VE 102 ... VE N. When ready - start VEs:
<pre>
[HN]# vzctl start 101
[HN]# vzlist -a
VEID NPROC STATUS IP_ADDR HOSTNAME
101 4 running 10.0.98.96 -
102 4 running 10.0.98.97 -
</pre>
2) By default VEs use venet device for networking (http://wiki.openvz.org/Venet). But current
configuration requires using alternative networking - through veth devices (http://wiki.openvz.org/Virtual_Ethernet_device).
Switch VE 101 to veth by doing the following:

MAC address needed by eth0 of VE 101 and veth101.0 should be generated by easymac:
<pre>
[HN]# wget http://www.easyvmx.com/software/easymac.sh
[HN]# chmod 0777 easymac.sh
[HN]# ./easymac.sh -R
00:0C:29:70:BB:34
[HN]# ./easymac.sh -R
00:0C:29:C0:2E:07
</pre>
Replace venet by veth device on HN:
<pre>
[HN]# ifconfig venet0:0 down
[HN]# vzctl set 101 --netif_add eth0,00:0C:29:70:BB:34,veth101.0,00:0C:29:C0:2E:07 --save
[HN]# ifconfig veth101.0 0
[HN]# echo 0 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
[HN]# echo 0 > /proc/sys/net/ipv4/conf/veth101.0/proxy_arp
</pre>
Enter VE and tune ifconfig within VE:
<pre>
[VE 101]# vzctl enter 101
[VE 101]# ifconfig venet0:0 down
[VE 101]# ifconfig venet0 down
[VE 101]# ifconfig eth0 0
[VE 101]# ip addr add 10.0.98.96 dev eth0
[VE 101]# ip route add default dev eth0
</pre>
The same (whole item 2) should be done for VE 102 .. VE N.
3) Now we should eliminate the IP address on eth1:
[HN]# vim /etc/sysconfig/network-scripts/ifcfg-eth1

Edit like this:
<pre>
DEVICE=eth1
#BOOTPROTO=dhcp <<== comment
HWADDR=XX:XX:XX:XX:XX:XX
ONBOOT=yes
</pre>
and save changes (:wq).
<pre>
[HN]# /etc/init.d/network restart
</pre>
And turn off forwarding and proxy_arp for eth1.
<pre>
[HN]# ifconfig eth1 0
[HN]# echo 0 > /proc/sys/net/ipv4/conf/eth1/forwarding
[HN]# echo 0 > /proc/sys/net/ipv4/conf/eth1/proxy_arp
</pre>
4) Create br0 bridge uniting eth1, veth101.0, ..., vethN.0:
<pre>
[HN]# brctl addbr br0
[HN]# brctl addif br0 eth1
[HN]# brctl addif br0 veth101.0
..., veth102.0, vethN.0 etc.
[HN]# ifconfig br0 0
</pre>
And turn off frowarding and proxy_arp for br0:
<pre>
[HN]# echo 0 > /proc/sys/net/ipv4/conf/br0/forwarding
[HN]# echo 0 > /proc/sys/net/ipv4/conf/br0/proxy_arp
</pre>
This is very important action. If skipped - network
can be broken on further steps due to incoming arp-requests provoked storm.

As a result of above listed actions the ifconfig output like the following should be listed:
<pre>
[HN]# ifconfig
br0 Link encap:Ethernet HWaddr 00:0C:29:A7:A9:D9
inet6 addr: fe80::20c:29ff:fea7:a9d9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:79 errors:0 dropped:0 overruns:0 frame:0
TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2972 (2.9 KiB) TX bytes:4390 (4.2 KiB)

eth0 Link encap:Ethernet HWaddr 00:30:48:5B:AB:34
inet addr:192.168.1.136 Bcast:192.168.3.255 Mask:255.255.252.0
inet6 addr: fe80::230:48ff:fe5b:ab34/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:347855 errors:0 dropped:0 overruns:0 frame:0
TX packets:4778 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:35964081 (34.2 MiB) TX bytes:698801 (682.4 KiB)
Interrupt:20

eth1 Link encap:Ethernet HWaddr 00:30:48:5B:AB:35
inet6 addr: fe80::230:48ff:fe5b:ab35/64 Scope:Link
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:322 errors:0 dropped:0 overruns:0 frame:0
TX packets:182 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:41943 (40.9 KiB) TX bytes:21338 (20.8 KiB)
Interrupt:21

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1376 errors:0 dropped:0 overruns:0 frame:0
TX packets:1376 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2078718 (1.9 MiB) TX bytes:2078718 (1.9 MiB)

veth101.0 Link encap:Ethernet HWaddr 00:0C:29:C0:2E:07
inet6 addr: fe80::20c:29ff:fec0:2e07/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:363 errors:0 dropped:0 overruns:0 frame:0
TX packets:397 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:31134 (30.4 KiB) TX bytes:31440 (30.7 KiB)

veth102.0 Link encap:Ethernet HWaddr 00:0C:29:A7:A9:D9
inet6 addr: fe80::20c:29ff:fea7:a9d9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:36 errors:0 dropped:0 overruns:0 frame:0
TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1840 (1.7 KiB) TX bytes:2350 (2.2 KiB)
</pre>
5) That is all. It's time to test the obtained configuration.
Now plug eth1 of HN into network wall outlet provided by ISP and carry out the following testing:

- It should be tested that VEs are accessible from Internet:
<pre>
[INET]# ssh root@10.0.98.96
[VE 101]# ...
</pre>
- HN is not accessible from Internet:
<pre>
[INET]# ssh root@192.168.1.136
inaccessible
</pre>
- VEs can be managed from HN:
<pre>
[HN]# vzctl enter 101
[VE 101]# ...
</pre>
- VEs VE 101, VE 102 .. VE N "see" each other (ping).

If all the steps are done as written, it should work.
Enjoy.

File:Fig.jpg

2007-07-03T11:38:53Z

Emkravts: Illustration

Illustration

Using veth and brctl for protecting HN and saving IP addresses

2007-07-03T11:33:25Z

Emkravts: New page: Configuration described below has been suggested by Ugo123. Appreciates. Consider we are facing the following task: 1) We have limited range of IP adresses granted by ISP. We want to as...

Configuration described below has been suggested by Ugo123. Appreciates.

Consider we are facing the following task:

1) We have limited range of IP adresses granted by ISP.
We want to assign as much granted IPs to VEs as possible.
We do not want to protect VEs from Internet.
2) We want to protect the HN OS (VE0) from Internet and make it possible to manage VEs from VE0 within local area network.

Assume we have a HN with 2 ethernet cards (interfaces eth0 and eth1), OpenVZ kernel 2.6.18-028stab033, vzctl version 3.0.16,
bridge-utils version 1.1. OpenVZ installation process is covered by http://wiki.openvz.org/Quick_installation.

Task can be effectively solved by setting up the configuration presented on Figure 1.

Figure 1: Effective configuration. 10.0.98.96-10.0.98.X - range of IP-adresses granted by ISP, 192.168.1.136 - IP address from LAN

Initial ifconfig output of HN is the following:
<pre>
[HN]# ifconfig
eth0 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:34
inet addr:192.168.1.136 Bcast:192.168.3.255 Mask:255.255.252.0
inet6 addr: fe80::230:48ff:fe5b:ab34/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3122 errors:0 dropped:0 overruns:0 frame:0
TX packets:246 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:325879 (318.2 KiB) TX bytes:57278 (55.9 KiB)
Interrupt:20

eth1 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:35
inet addr:192.168.0.32 Bcast:192.168.3.255 Mask:255.255.252.0
inet6 addr: fe80::213:d4ff:fe90:4d50/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:603734 errors:0 dropped:0 overruns:0 frame:0
TX packets:36627 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:21

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1376 errors:0 dropped:0 overruns:0 frame:0
TX packets:1376 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2078718 (1.9 MiB) TX bytes:2078718 (1.9 MiB)
</pre>
Let us pass through the setup process step by step.

1) Create 2 VEs on the HN as described in http://download.openvz.org/doc/OpenVZ-Users-Guide.pdf.
For testing purposes I've used opensuse-10 precreated template from openvz.org:
<pre>
[HN]# cd /vz/template/cache
[HN]# wget http://download.openvz.org/template/precreated/opensuse-10-i386-default.tar.gz
</pre>
Create VE 101 and assign it one of the IP adresses obtained from ISP:
<pre>
[HN]# vzctl create 101 --ostemplate opensuse-10-i386-default --ipadd 10.0.98.96
[HN]# vzctl set 101 --userpasswd root:XXX --save
</pre>
And do the same for VE 102 ... VE N. When ready - start VEs:
<pre>
[HN]# vzctl start 101
[HN]# vzlist -a
VEID NPROC STATUS IP_ADDR HOSTNAME
101 4 running 10.0.98.96 -
102 4 running 10.0.98.97 -
</pre>
2) By default VEs use venet device for networking (http://wiki.openvz.org/Venet). But current
configuration requires using alternative networking - through veth devices (http://wiki.openvz.org/Virtual_Ethernet_device).
Switch VE 101 to veth by doing the following:

MAC address needed by eth0 of VE 101 and veth101.0 should be generated by easymac:
<pre>
[HN]# wget http://www.easyvmx.com/software/easymac.sh
[HN]# chmod 0777 easymac.sh
[HN]# ./easymac.sh -R
00:0C:29:70:BB:34
[HN]# ./easymac.sh -R
00:0C:29:C0:2E:07
</pre>
Replace venet by veth device on HN:
<pre>
[HN]# ifconfig venet0:0 down
[HN]# vzctl set 101 --netif_add eth0,00:0C:29:70:BB:34,veth101.0,00:0C:29:C0:2E:07 --save
[HN]# ifconfig veth101.0 0
[HN]# echo 0 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
[HN]# echo 0 > /proc/sys/net/ipv4/conf/veth101.0/proxy_arp
</pre>
Enter VE and tune ifconfig within VE:
<pre>
[VE 101]# vzctl enter 101
[VE 101]# ifconfig venet0:0 down
[VE 101]# ifconfig venet0 down
[VE 101]# ifconfig eth0 0
[VE 101]# ip addr add 10.0.98.96 dev eth0
[VE 101]# ip route add default dev eth0
</pre>
The same (whole item 2) should be done for VE 102 .. VE N.
3) Now we should eliminate the IP address on eth1:
[HN]# vim /etc/sysconfig/network-scripts/ifcfg-eth1

Edit like this:
<pre>
DEVICE=eth1
#BOOTPROTO=dhcp <<== comment
HWADDR=XX:XX:XX:XX:XX:XX
ONBOOT=yes
</pre>
and save changes (:wq).
<pre>
[HN]# /etc/init.d/network restart
</pre>
And turn off forwarding and proxy_arp for eth1.
<pre>
[HN]# ifconfig eth1 0
[HN]# echo 0 > /proc/sys/net/ipv4/conf/eth1/forwarding
[HN]# echo 0 > /proc/sys/net/ipv4/conf/eth1/proxy_arp
</pre>
4) Create br0 bridge uniting eth1, veth101.0, ..., vethN.0:
<pre>
[HN]# brctl addbr br0
[HN]# brctl addif br0 eth1
[HN]# brctl addif br0 veth101.0
..., veth102.0, vethN.0 etc.
[HN]# ifconfig br0 0
</pre>
And turn off frowarding and proxy_arp for br0:
<pre>
[HN]# echo 0 > /proc/sys/net/ipv4/conf/br0/forwarding
[HN]# echo 0 > /proc/sys/net/ipv4/conf/br0/proxy_arp
</pre>
This is very important action. If skipped - network
can be broken on further steps due to incoming arp-requests provoked storm.

As a result of above listed actions the ifconfig output like the following should be listed:
<pre>
[HN]# ifconfig
br0 Link encap:Ethernet HWaddr 00:0C:29:A7:A9:D9
inet6 addr: fe80::20c:29ff:fea7:a9d9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:79 errors:0 dropped:0 overruns:0 frame:0
TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2972 (2.9 KiB) TX bytes:4390 (4.2 KiB)

eth0 Link encap:Ethernet HWaddr 00:30:48:5B:AB:34
inet addr:192.168.1.136 Bcast:192.168.3.255 Mask:255.255.252.0
inet6 addr: fe80::230:48ff:fe5b:ab34/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:347855 errors:0 dropped:0 overruns:0 frame:0
TX packets:4778 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:35964081 (34.2 MiB) TX bytes:698801 (682.4 KiB)
Interrupt:20

eth1 Link encap:Ethernet HWaddr 00:30:48:5B:AB:35
inet6 addr: fe80::230:48ff:fe5b:ab35/64 Scope:Link
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:322 errors:0 dropped:0 overruns:0 frame:0
TX packets:182 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:41943 (40.9 KiB) TX bytes:21338 (20.8 KiB)
Interrupt:21

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1376 errors:0 dropped:0 overruns:0 frame:0
TX packets:1376 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2078718 (1.9 MiB) TX bytes:2078718 (1.9 MiB)

veth101.0 Link encap:Ethernet HWaddr 00:0C:29:C0:2E:07
inet6 addr: fe80::20c:29ff:fec0:2e07/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:363 errors:0 dropped:0 overruns:0 frame:0
TX packets:397 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:31134 (30.4 KiB) TX bytes:31440 (30.7 KiB)

veth102.0 Link encap:Ethernet HWaddr 00:0C:29:A7:A9:D9
inet6 addr: fe80::20c:29ff:fea7:a9d9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:36 errors:0 dropped:0 overruns:0 frame:0
TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1840 (1.7 KiB) TX bytes:2350 (2.2 KiB)
</pre>
5) That is all. It's time to test the obtained configuration.
Now plug eth1 of HN into network wall outlet provided by ISP and carry out the following testing:

- It should be tested that VEs are accessible from Internet:
<pre>
[INET]# ssh root@10.0.98.96
[VE 101]# ...
</pre>
- HN is not accessible from Internet:
<pre>
[INET]# ssh root@192.168.1.136
inaccessible
</pre>
- VEs can be managed from HN:
<pre>
[HN]# vzctl enter 101
[VE 101]# ...
</pre>
- VEs VE 101, VE 102 .. VE N "see" each other (ping).

If all the steps are done as written, it should work.
Enjoy.

How to use OpenVZ as a XEN guest OS (for x86 platform)

2007-06-09T12:52:48Z

Emkravts: New page: 1. Intro Recently released RHEL5 kernel supports compatibility with XEN 3.0. It means that RHEL5 kernel built with config that enables XEN can be used as guest OS in both privileged Dom0 ...

1. Intro

Recently released RHEL5 kernel supports compatibility with XEN 3.0. It means that
RHEL5 kernel built with config that enables XEN can be used as guest OS in both privileged Dom0 and unprivileged DomU XEN domains. RHEL5 - based OpenVZ kernel 028stab033 also contains support for XEN compatibility. Current article reports on how to virtualize xen DomU by using xen-compatible OpenVZ kernel el5.028stab034.

Detailed description of XEN 3.0, user guides and howtos on XEN 3.0 can be found here:
http://www.xensource.com/products/xen/documentation.html. But we'll give here some common information on XEN 3.0 in purpose to make a story more clear. XEN system consists of 3 parts:

1) Xen hypervisor - lowest level part of system that virtualizes drivers and architecture dependent part of the system.

2) XenLinux kernel - guest OS that works over hypervisor in privileged domain Dom0 or unprivileged domain DomU. Privileged Dom0 domain is used for creating, destroying and supervising of unprivileged domains (DomUs). XenLinux kernel that is running in Dom0 contains 2 sets of drivers: physical and virtual. DomU XenLinux kernel can contain the only set of virtual drivers.

3) XM Tool - userland program that is used from Dom0 for creating, destroying and supervising DomUs.

As OpenVZ is an OS level virtualization solution and do not affect drivers, it is possible to run OpenVZ within XEN DomU. Text listed below guides through XEN 3.0, XM tool and OpenVZ-XenLinux PAE kernel installation on x86. Guide assumes RHEL5 Linux is used as an OS on your hardware.

Well. Login as root. Create 2 directories:
<pre>
# mkdir xen
# mkdir openvz
</pre>

2. Installing XEN itself, Dom0 and XEN tools.

XEN 3.0 and XM tool can be installed in 2 ways: either from tarball containing prebuilt binaries or by building from sources.

- Installing from tarball

Download XEN 3.1 32 bit PAE SMP tarball from http://bits.xensource.com/ and unpack:
<pre>
# cd xen
# wget http://bits.xensource.com/oss-xen/release/3.1.0/bin.tgz/xen-3.1.0-install-x86_32p.tgz
# gunzip xen-3.1.0-install-x86_32p.tgz
# tar -xvf xen-3.1.0-install-x86_32p.tar
</pre>
Run the install.sh script within ./dist folder:
<pre>
# ./dist/install.sh

Installing Xen from './dist/install' to '/'...
- installing for udev-based system
- modifying permissions
All done.
Checking to see whether prerequisite tools are installed...
Xen CHECK-INSTALL Thu May 24 14:30:20 MSD 2007
Checking check_brctl: OK
Checking check_crypto_lib: OK
Checking check_iproute: OK
Checking check_libvncserver: unused, OK
Checking check_python: OK
Checking check_python_xml: OK
Checking check_sdl: unused, OK
Checking check_udev: OK
Checking check_zlib_lib: OK
All done.

#
</pre>
install.sh checks the presence of the required packages and installs hypervisor xen.gz and XenLinux vmlinuz-2.6.16.33-xen
to /boot and xm tool to /usr/sbin. Now update /etc/grub.conf by adding the xen entry in the following way:

/etc/grub.conf:
<pre>
title Xen Linux (2.6.16.33-xen)
root (hd0,1)
kernel /boot/xen.gz dom0_mem=1048576
module /boot/vmlinuz-2.6.16.33-xen ro root=LABEL=/ console=ttyS0,115200 console=tty silencelevel=8
</pre>
Now you can reboot the machine. Choosing the Xen Linux (2.6.16.33-xen) from grub sequence will cause to booting xen hypervisor and Dom0 running Xen Linux (2.6.16.33-xen). In case of success you'll see the usual login prompt. If kernel panic takes place during boot - reboot the machine into your usual working Linux and check if the xen related /etc/grub.conf entry is correct. If /etc/grub.conf entry is correct and xen kernel is unable to boot,- the reason is prebuild vmlinuz-2.6.16.33-xen does not contain
some drivers for your hardware, that are needed to be loaded by using initrd. In this case follow instructions below - "Building from sources".

- Building from sources

Download the package containing sources of Xen-3.1 and unpack it:
<pre>
# wget http://bits.xensource.com/oss-xen/release/3.1.0/src.tgz/xen-3.1.0-src.tgz
# gunzip xen-3.1.0-src.tgz
# tar -xvf xen-3.1.0-src.tar
</pre>
Change directory to ./xen-3.1.0-src:
<pre>
# cd ./xen-3.1.0-src
</pre>
and start building:
<pre>
# make XEN_TARGET_X86_PAE=y
</pre>
Building process will do the following:
- build hypervizor xen.gz
- download linux kernel sources 2.6.16 from kernel.org, patch them with a set of xen patches
- from sources obtained build the vmlinuz-2.6.16.33-xen XenLinux kernel that will be able to boot both in Dom0 and DomU
- build and install xm tool

After building is complete run ./install.sh script:
<pre>
# ./install.sh
</pre>
As a result vmlinuz-2.6.16.33-xen, xen.gz will be placed to boot, xm tool will be placed to /usr/sbin. Now it is time to create initrd for vmlinuz-2.6.16.33-xen. Check that /lib/modules contain recently created directory 2.6.16.33-xen. If it is, type:
<pre>
# mkinitrd -f -v /boot/initrd-2.6.16.33-xen.img 2.6.16.33-xen
</pre>
After initrd-2.6.16.33-xen.img is succesfully created update corresponding strings in /etc/grub.conf:
<pre>
title Xen Linux (2.6.16.33-xen)
root (hd0,1)
kernel /boot/xen.gz dom0_mem=1048576
module /boot/vmlinuz-2.6.16.33-xen ro root=LABEL=/ console=ttyS0,115200 console=tty silencelevel=8
module /boot/initrd-2.6.16.33-xen.img
</pre>
Xen 3.0 doesn't support TLS. To make xen work correct disable TLS on your machine bu renaming /lib/tls to /lib/tls.disabled:
<pre>
# mv /lib/tls /lib/tls.disabled
</pre>
Check that kernel command line options are correct (after ro ...) and reboot the machine. In grub loader menu select "Xen Linux (2.6.16.33-xen)". In case of success - machine boots into Xen's Dom0 that is running vmlinuz-2.6.16.33-xen XenLinux. Now it is time to prepare OpenVZ XenLinux kernel for DomU.

3. Installing OpenVZ XenLinux kernel.

OpenVZ XenLinux kernel can be installed in 2 ways: either from rpm containing or by
building from sources.

- install from rpm

Change directory to openvz and download rpm package with OpenVZ XenLinux kernel for x86:
<pre>
# cd
# cd openvz
# wget http://download.openvz.org/kernel/branches/rhel5-2.6.18/028stab034.1/ovzkernel-PAE-2.6.18-8.el5.xen.028stab034.i686.rpm
</pre>
Install downloaded rpm:
<pre>
# rpm -ihv ovzkernel-PAE-2.6.18-8.el5.xen.028stab033.i686.rpm
</pre>
If something goes wrong during installation the prebuilt kernel - you can build the OpenVZ XenLinux kernel from sources as described below.

- install from sources

Download rpm with OpenVZ kernel sources:
<pre>
# wget http://download.openvz.org/kernel/branches/rhel5-2.6.18/028stab034.1/ovzkernel-2.6.18-8.el5.028stab034.1.src.rpm
</pre>
Unpack downloaded rpm package:
<pre>
# rpm2cpio ovzkernel-2.6.18-8.el5.028stab034.1.src.rpm > ovzkernel-2.6.18-8.el5.028stab034.1.src.cpio
# cpio -i < ovzkernel-2.6.18-8.el5.028stab034.1.src.cpio
</pre>
Check that ./kernel-ovz.spec contains the following strings:
<pre>
...
# Whether to build the Xen kernels, disable if you want.
%define buildxen 1
...
</pre>
If "buildxen is set to 0" - update kernel-ovz.spec using text editor, set buildxen to 1.

Prepare sources for building:
<pre>
# rpmbuild -bp --define "_topdir $PWD" --define "_sourcedir $PWD" --define "_builddir $PWD" --target i686-linux kernel-ovz.spec
# cd ./ovzkernel-2.6.18/linux-2.6.18.i686
</pre>
Use xen config file:
<pre>
# cp configs/kernel-2.6.18-i686-PAE-xen.config.ovz ./.config
</pre>
Update Makefile: change the value of EXTRAVERSION from "-prep" to something better, for example "-openvzxen":
<pre>
EXTRAVERSION = -openvzxen
</pre>
And build the OpenVZ XenLinux:
<pre>
# make oldconfig
# make
</pre>
After build is complete, - install modules and the kernel:
<pre>
# make modules_install
# make install
</pre>
Check that /lib/modules contains the directory 2.6.18-openvzxen.
And prepare initrd image for built OpenVZ XenLinux:
<pre>
# mkinitrd -f -v --omit-scsi-modules --preload=xenblk /boot/initrd-2.6.18-openvzxen.img 2.6.18-openvzxen
</pre>
OpenVZ XenLinux kernel prepared. Now OpenVZ tools are to be installed. In case you faced problem while building the kernel turn to article http://wiki.openvz.org/Kernel_build .

- install OpenVZ tools:

Now download and install latest versions of OpenVZ tools vzctl, vzquota that are necessary for starting/stopping VEs:
<pre>
# wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-3.0.16-1.i386.rpm .
# wget http://download.openvz.org/utils/vzctl/3.0.16/vzctl-lib-3.0.16-1.i386.rpm .
# wget http://download.openvz.org/utils/vzquota/3.0.9/vzquota-3.0.9-1.i386.rpm .
# rpm -Uhv --nodeps vzctl-3.0.16-1.i386.rpm vzctl-lib-3.0.16-1.i386.rpm vzquota-3.0.9-1.i386.rpm
</pre>
More information on installing OpenVZ tools can be found here: http://wiki.openvz.org/Quick_installation.

- download OpenVZ guest template

To make it possible to create VEs download one of the precreated OpenVZ templates and place it to /vz/template/cache:
<pre>
# wget http://download.openvz.org/template/precreated/fedora-core-5-i386-minimal.tar.gz
# mv fedora-core-5-i386-minimal.tar.gz /vz/template/cache/
</pre>

4. Preparing guest partition and configuring XEN.

Xen users manual http://www.xensource.com/products/xen/documentation.html describes a set of different ways of creating disk for DomU. Disk can be created using loopback or LVM or physical partition. Consider we have a standalone physical partition /dev/sda7 that will be used for creating DomU disk. Assume /dev/sda7 is empty and it contains ext3 filesystem.

Prepare disk with DomU guest Linux:
<pre>
# mount -t ext3 /dev/sda7 /mnt
# cp -a /bin /mnt
# cp -a /dev /mnt
# cp -a /etc /mnt
# cp -a /lib /mnt
# cp -a /net /mnt
# cp -a /opt /mnt
# cp -a /root /mnt
# cp -a /sbin /mnt
# cp -a /srv /mnt
# cp -a /tmp /mnt
# cp -a /usr /mnt
# cp -a /var /mnt
# cp -a /vz /mnt
# mkdir /mnt/sys
# mkdir /mnt/proc
# mkdir /mnt/mnt
# mkdir /mnt/home
</pre>
Now create configuration file for starting DomU:
<pre>
# touch /etc/xen/xmDomU
# vim /etc/xen/xmDomU
</pre>
and update xmDomU in the following way:
<pre>
# Kernel image file.
kernel = "/boot/vmlinuz-2.6.18-openvzxen"

# Optional ramdisk.
ramdisk = "/boot/initrd-2.6.18-openvzxen.img"

# Initial memory allocation (in megabytes) for the new domain.
#
# WARNING: Creating a domain with insufficient memory may cause out of
# memory errors. The domain needs enough memory to boot kernel
# and modules. Allocating less than 32MBs is not recommended.
memory = 256

# A name for your domain. All domains must have different names.
name = "Domain-U"

# Define network interfaces.
vif = [ '' ]

# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.
disk = [ 'phy:sda7,xvda1,w' ]

# Set root device.
root = "/dev/xvda1"

# Sets runlevel 4.
extra = "4 debug"
</pre>
Update /mnt/etc/fstab in the following way:
<pre>
/dev/xvda1 / ext3 defaults 1 1
</pre>
Umount /dev/sda7:
<pre>
# umount /mnt
</pre>
Preparations completed. At this point we have everything that is necessary for cretaing DomU and starting OpenVZ XenLinux within DomU. Reboot into Xen: select Xen Linux (2.6.16.33-xen) item from grub menu. After reboot machine will be running vmlinuz-2.6.16.33-xen (see above) in Dom0 over hypervisor. Follow instructions below.

5. Starting OpenVZ guest in DomU.

- starting

Start xend daemon to make xm tool work:
<pre>
# xend start
</pre>
List the set of running domains:
<pre>
[root@dhcp0-131 ~]# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 2 r----- 53.2
[root@dhcp0-131 ~]#
</pre>

Now only privileged domain Dom0 is running. Create DomU:
<pre>
[root@dhcp0-131 ~]# xm create -c /etc/xen/xmDomU
</pre>
This call culminates by showing us the DomU login prompt. Current console is now DomU console.

Open another terminal on the machine (correspond to Dom0). And list the machines:
<pre>
# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 2 r----- 75.0
Domain-U 1 255 1 r----- 19.5
#
</pre>
Switch back to DomU console and test OpenVZ.

DomU console:
<pre>
[root@localhost ~]# uname
Linux localhost.localdomain 2.6.18-ovzxen-4-uxen #13 SMP Tue May 22 22:22:35
MSD 2007 i686 i686 i386 GNU/Linux
[root@localhost ~]# vzctl create 134 --ostemplate fedora-core-5-i386-minimal
Creating VE private area (fedora-core-5-i386-minimal)
Performing postcreate actions
VE private area was created
[root@localhost ~]# vzlist -a
VEID NPROC STATUS IP_ADDR HOSTNAME
134 - stopped - -
[root@localhost ~]# vzctl start 134
Starting VE ...
VE is mounted
Setting CPU units: 1000
VE start in progress...
[root@localhost ~]# vzlist -a
VEID NPROC STATUS IP_ADDR HOSTNAME
134 - running - -
[root@localhost ~]# vzctl enter 134
entered into VE 134
[root@localhost /]#
...
some actions within VE 134
...
[root@localhost /]# exit
exited from VE 156
[root@localhost ~]# vzctl stop 134
Stopping VE ...
VE was stopped
VE is unmounted
[root@localhost ~]#
</pre>
It works.

- next steps

As OpenVZ works in DomU it is the best time for configuring network both in DomU and VEs. This process is described in http://www.xensource.com/products/xen/documentation.html (DomU) and http://wiki.openvz.org/Category:Networking (VEs). After this all of us will feel the full power of virtualization!

6. RHEL5 based OpenVZ in Dom0. Known problems.

OpenVZ XenLinux kernel is able to work also in Dom0. It can be tested just by updating /etc/grub.conf on the hardware node. But in this case it will be impossible to start DomUs. It is a known bug and it is related not to OpenVZ, but to RHEL5 kernel.
Hope it is fixed in next RHEL5 update.

7. Appreciates

Thanks to Sergey Ya. Korshunoff seyko@ for starting the topic and contribution.

Migration from Linux-VServer to OpenVZ

2006-12-12T07:44:54Z

Emkravts:

Current document describes the migration from Linux-VServer based virtualization solution to OpenVZ.

Description of challenge:

The challenge is migration from Linux-Vserver to OpenVZ by booting the OpenVZ kernel and updating the existing configs of
utility level in purpose to make the existing guest OSes work over OpenVZ kernel.

Details of migration process. Step by step:

1. Initial conditions: the following example of Linux-VServer based solution was used for the experiment:

- Kernel linux-2.6.17.13 was patched by the patch-2.6.17.13-vs2.0.2.1.diff and rebuild;
- Util-vserver-0.30.211 tools were used for creating Virtual Environments;

<code>
# vserver-info
Versions:
Kernel: 2.6.17.13-vs2.0.2.1
VS-API: 0x00020002
util-vserver: 0.30.211; Dec 5 2006, 17:10:21

Features:
CC: gcc, gcc (GCC) 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)
CXX: g++, g++ (GCC) 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)
CPPFLAGS: ''
CFLAGS: '-g -O2 -std=c99 -Wall -pedantic -W -funit-at-a-time'
CXXFLAGS: '-g -O2 -ansi -Wall -pedantic -W -fmessage-length=0 -funit-at-a-time'
build/host: i686-pc-linux-gnu/i686-pc-linux-gnu
Use dietlibc: yes
Build C++ programs: yes
Build C99 programs: yes
Available APIs: v13,net
ext2fs Source: kernel
syscall(2) invocation: alternative
vserver(2) syscall#: 273/glibc

Paths:
prefix: /usr/local
sysconf-Directory: ${prefix}/etc
cfg-Directory: ${prefix}/etc/vservers
initrd-Directory: $(sysconfdir)/init.d
pkgstate-Directory: ${prefix}/var/run/vservers
vserver-Rootdir: /vservers
#
</code>

VServer v345 was built using vserver vX build utility and populated by using the tarballed template of Fedora Core 4.

<code>
# vserver v345 start
Starting system logger: [ OK ]
Initializing random number generator: [ OK ]
Starting crond: l: [ OK ]
Starting atd: [ OK ]
# vserver v345 enter
[/]# ls -l
total 44
drwxr-xr-x 2 root root 4096 Oct 26 2004 bin
drwxr-xr-x 3 root root 4096 Dec 8 17:16 dev
drwxr-xr-x 27 root root 4096 Dec 8 15:21 etc
-rw-r--r-- 1 root root 0 Dec 8 15:33 halt
drwxr-xr-x 2 root root 4096 Jan 24 2003 home
drwxr-xr-x 7 root root 4096 Oct 26 2004 lib
drwxr-xr-x 2 root root 4096 Jan 24 2003 mnt
drwxr-xr-x 3 root root 4096 Oct 26 2004 opt
-rw-r--r-- 1 root root 0 Dec 7 20:17 poweroff
dr-xr-xr-x 80 root root 0 Dec 8 11:38 proc
drwxr-x--- 2 root root 4096 Dec 7 20:17 root
drwxr-xr-x 2 root root 4096 Oct 26 2004 sbin
drwxrwxrwt 2 root root 40 Dec 8 17:16 tmp
drwxr-xr-x 15 root root 4096 Jul 27 2004 usr
drwxr-xr-x 17 root root 4096 Oct 26 2004 var
[/]# sh
sh-2.05b#
.........
</code>

As a result we obtain running virtual environment v345:

<code>
# vserver-stat

CTX PROC VSZ RSS userTIME sysTIME UPTIME NAME
0 51 90.9M 26.3M 0m58s75 2m42s57 33m45s93 root server
49153 4 10.2M 2.8M 0m00s00 0m00s11 21m45s42 v345

#
</code>

2. Starting migration to OpenVZ: downloading and installing the stable OpenVZ kernel.

First of all we should download and install the latest stable version of OpenVZ kernel:

<code>
# wget http://openvz.org/download/kernel/stable/ovzkernel-2.6.9-023stab032.1.i686.rpm
# rpm -ihv ovzkernel-2.6.9-023stab032.1.i686.rpm
</code>

If you use grub bootloader, then before installing the OpenVZ kernel file /boot/grub/grub.conf looked like this:

<code>
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux AS (2.6.17.13-vs2.0.2.1)
root (hd0,0)
kernel /vmlinuz-2.6.17.13-vs2.0.2.1 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.17.13-vs2.0.2.1.img
</code>

After installing it should look like this:

<code>
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux AS (2.6.9-023stab032.1)
root (hd0,0)
kernel /vmlinuz-2.6.9-023stab032.1 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.9-023stab032.1.img
title Red Hat Enterprise Linux AS (2.6.17.13-vs2.0.2.1)
root (hd0,0)
kernel /vmlinuz-2.6.17.13-vs2.0.2.1 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.17.13-vs2.0.2.1.img
</code>

And the default parameter should be set to 0 in purpose to make the OpenVZ kernel work by default after rebooting the machine.

Some more manual configuring needed on this step:
Update the /etc/sysctl.conf file according to the following listing ( Comment all other settings that are not presented
in listing below ):

<code>
# On Hardware Node we generally need
# packet forwarding enabled and proxy arp disabled
net.ipv4.ip_forward = 1
net.ipv4.conf.default.proxy_arp = 0
# Enables source route verification
net.ipv4.conf.all.rp_filter = 1
# Enables the magic-sysrq key
kernel.sysrq = 1
# TCP Explict Congestion Notification
#net.ipv4.tcp_ecn = 0
# we do not want all our interfaces to send redirects
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
</code>

As the SELinux should be disabled, put the following line to /etc/sysconfig/selinux:

<code>
SELINUX=disabled
</code>

Now reboot the machine. After rebooting and loggin in you will see the following reply on vserver-stat call:

<code>
# vserver-stat
can not change context: migrate kernel feature missing and 'compat' API disabled: Function not implemented
#
</code>

It is a natural thing that now virtual environments v345 is unavailable. The following steps will be devoted to making them
work over OpenVZ kernel.

3. Downloading and istalling vzctl package

OpenVZ solution reqires installing a set of tools: vzctl and vzquota packages. Let us download and install this tools:

<code>
# wget http://openvz.org/download/utils/vzctl-3.0.13-1.i386.rpm
# wget http://openvz.org/download/utils/vzquota-3.0.9-1.i386.rpm
# rpm -Uhv vzctl-3.0.13-1.i386.rpm vzquota-3.0.9-1.i386.rpm
</code>

If rpm complains about unresolved dependencies, you'll have to satisfy them first, then repeat the installation.
Then launch the OpenVZ:

<code>
# /sbin/service vz start
Starting OpenVZ: [ OK ]
Bringing up interface venet0: [ OK ]
Configuring interface venet0: [ OK ]
#
</code>

Currently vzlist util is unable to find any VE:

<code>
# vzlist
VE not found
#
</code>

4. Updating different configurations in purpose to make existing templates work

Move the existing templates of guest OSes to the right place:

<code>
# cd /vz
# mkdir private
# mkdir 345
# mv /vservers/v345 /vz/private/345
</code>

Now it is time for creating configuration files for OpenVZ VPS. Use the basic sample
configuration presented in /etc/sysconfig/vz-scripts/ve-vps.basic.conf-sample file:

<code>
# cd /etc/sysconfig/vz-scripts
# cp ve-vps.basic.conf-sample 345.conf
</code>

Update the ON_BOOT string in 345.conf file by typing:

<code>
.....
ONBOOT="yes"
.....
</code>
to make it boot on node restart, and add a couple of strings related to the particular 345 VE:
<code>
.....
VE_ROOT="/vz/root/345"
VE_PRIVATE="/vz/private/345"
ORIGIN_SAMPLE="vps.basic"
HOSTNAME="test345.my.org"
IP_ADDRESS="192.168.0.145"
.....
</code>

And reboot the machine:

<code>
# reboot
</code>

5. Testing how the guest OSes succesfully work over OpenVZ. reference to Users Guide of OpenVZ (vzctl).

After rebooting you will be able to see running VE 345 that have been migrated from vserver:

<code>
# vzlist -a
VEID NPROC STATUS IP_ADDR HOSTNAME
345 5 running 192.168.0.145 test345.my.org
#
</code>

And run commands on it:

<code>
# vzctl exec 345 ls -l
total 48
drwxr-xr-x 2 root root 4096 Oct 26 2004 bin
drwxr-xr-x 3 root root 4096 Dec 11 12:42 dev
drwxr-xr-x 27 root root 4096 Dec 11 12:44 etc
-rw-r--r-- 1 root root 0 Dec 11 12:13 fastboot
-rw-r--r-- 1 root root 0 Dec 8 15:33 halt
drwxr-xr-x 2 root root 4096 Jan 24 2003 home
drwxr-xr-x 7 root root 4096 Oct 26 2004 lib
drwxr-xr-x 2 root root 4096 Jan 24 2003 mnt
drwxr-xr-x 3 root root 4096 Oct 26 2004 opt
-rw-r--r-- 1 root root 0 Dec 7 20:17 poweroff
dr-xr-xr-x 70 root root 0 Dec 11 12:42 proc
drwxr-x--- 2 root root 4096 Dec 7 20:17 root
drwxr-xr-x 2 root root 4096 Dec 11 12:13 sbin
drwxrwxrwt 2 root root 4096 Dec 8 12:40 tmp
drwxr-xr-x 15 root root 4096 Jul 27 2004 usr
drwxr-xr-x 17 root root 4096 Oct 26 2004 var
#
</code>

Potential issues:
This work has in progress status. Some issues may take place with tuning the network for VEs. To be continued.

[[Category:HOWTO]]