OpenVZ Virtuozzo Containers Wiki - User contributions [en]

HA cluster with DRBD and Heartbeat

2008-01-09T15:18:46Z

Wfischer: added live-switchover

This article shows how to setup a OpenVZ high availability (HA) cluster using the data replication software DRBD and the cluster manager Heartbeat. In this example the two machines building the cluster run on CentOS 4.3. The article also shows how to do kernel updates in the cluster, including necessary steps like recompiling of new DRBD userspace tools. For this purpose, kernel 2.6.8-022stab078.10 (containing DRBD module 0.7.17) is used as initial kernel version, and kernel 2.6.8-022stab078.14 (containing DRBD module 0.7.20) as updated kernel version.

Update: this howto currently does not describe details on OpenVZ Kernel 2.6.18, which contains DRBD version 8.*. Meanwhile, some hints on using OpenVZ Kernel 2.6.18 with DRBD 8 can be found in [http://forum.openvz.org/index.php?t=msg&th=3213&start=0& this thread in the forum].

Additional information about clustering of virtual machines can be found in the following paper: [http://www.linuxtag.org/2006/fileadmin/linuxtag/dvd/12080-paper.pdf (PDF, 145K)]

Some other additional information can be found in the documentation of the Thomas-Krenn.AG cluster (The author of this howto is working in the cluster development there, that is the reason why he was able to write this howto :-). The full documentation with interesting illustrations is currently only [http://www.thomas-krenn.com/en/service-support/knowledge-center/cluster/documentation.html available in German]:

== Prerequisites ==
The OpenVZ kernel already includes the DRBD module. The DRBD userspace tools and the cluster manager Heartbeat must be provided seperately. As the API version of the DRBD userspace tools must exactly match the API version of the module, compile them yourself. Also compile Heartbeat yourself, as at the time of this writing the CentOS extras repository only contained an old CVS version of Heartbeat.

On a hardware node for production use there should not be any application that is not really needed for running OpenVZ (any things which are not needed by OpenVZ should run in a VE for security reasons). As a result, compile DRBD and Heartbeat on another machine running CentOS 4.3 (in this example I used a virtual machine on a VMware Server).

=== Compiling Heartbeat ===
Heartbeat version 1.2.* has successfully been used in a lot of two-node-clusters around the world. As the codebase used in version 1.2.* is in production use for many years now, the code is very stable. At the time of writing, Heartbeat version 1.2.4 is the current version of the 1.2.* branch.

Get the tar.gz of the current version of the 1.2.* branch from http://linux-ha.org/download/index.html, at the time of this writing this is http://linux-ha.org/download/heartbeat-1.2.4.tar.gz. Use rpmbuild to build the package:
<pre>rpmbuild -ta heartbeat-1.2.4.tar.gz</pre>
After that, you find four rpm packes in /usr/src/redhat/RPMS/i386 (heartbeat-1.2.4-1.i386.rpm, heartbeat-ldirectord-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, heartbeat-stonith-1.2.4-1.i386.rpm). In this example only heartbeat-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, and heartbeat-stonith-1.2.4-1.i386.rpm are needed.

=== Compiling DRBD userspace tools ===
When compiling the DRBD userspace tools, you have to take care to take the version that matches the DRBD version that is included in the OpenVZ kernel you want to use. If you are unsure about the version, do the following steps while running the OpenVZ kernel that you want to use on a test machine (I used another virtual machine on a VMware server to try this):
<pre>
[root@testmachine ~]# cat /proc/version
Linux version 2.6.8-022stab078.10 (root@rhel4-32) (gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)) #1 Wed Jun 21 12:01:20 MSD 2006
[root@testmachine ~]# modprobe drbd
[root@testmachine ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Unconfigured
1: cs:Unconfigured
[root@testmachine ~]# rmmod drbd
[root@testmachine ~]#
</pre>
Here the version of the DRBD module is 0.7.17. So the userspace tools for 0.7.17 are neccessary.

Back on the buildmachine, do the following to create the rpm:
<pre>
[root@buildmachine ~]# yum install kernel-devel gcc bison flex
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Nothing to do
[root@buildmachine ~]# tar xfz drbd-0.7.17.tar.gz
[root@buildmachine ~]# cd drbd-0.7.17
[root@buildmachine drbd-0.7.17]# make rpm
[...]
You have now:
-rw-r--r-- 1 root root 288728 Jul 30 10:40 dist/RPMS/i386/drbd-0.7.17-1.i386.rpm
-rw-r--r-- 1 root root 518369 Jul 30 10:40 dist/RPMS/i386/drbd-km-2.6.9_34.0.2.EL-0.7.17-1.i386.rpm
[root@buildmachine drbd-0.7.17]#
</pre>
Note that in this way the kernel-devel from CentOS is used, but this does not matter as the created drbd-km rpm will not be used (the DRBD kernel module is already included in OpenVZ kernel). If the kernel-devel package is not the same version as the kernel package that is currently running, it is possible to execute 'make rpm KDIR=/usr/src/kernels/2.6.9-34.0.2.EL-i686/' to directly point to the kernel sources.

== Installing the two nodes ==
Install the two machines in the same way as you would install them for a normal OpenVZ installation, but do not create a filesystem for the /vz. This filesystem will be installed later on on top of DRBD.
{| class="wikitable"
|+'''Example installation configuration'''
! Parameter !! node1 !! node2
|-
! hostname
| ovz-node1
| ovz-node2
|-
! / filesystem
| hda1, 10 GB
| hda1, 10 GB
|-
! swap space
| hda2, 2048 MB
| hda2, 2048 MB
|-
! public LAN
| eth0, 192.168.1.201
| eth0, 192.168.1.202
|-
! private LAN
| eth1, 192.168.255.1 (Gbit Ethernet)
| eth1, 192.168.255.2 (Gbit Ethernet)
|-
! other install options
| no firewall, no SELinux
| no firewall, no SELinux
|-
! package groups
| deactivated everything, only kept vim-enhanced
| deactivated everything, only kept vim-enhanced
|}

== Installing OpenVZ ==

Get the OpenVZ kernel and utilities and install them on both nodes, as described in [[quick installation]]. Update grub configuration to use the OpenVZ kernel by default. Disable starting of OpenVZ on system boot on both nodes (OpenVZ will be started and stopped by Heartbeat):
<pre>
[root@ovz-node1 ~]# chkconfig vz off
[root@ovz-node1 ~]#
</pre>
Then reboot both machines.

== Setting up DRBD ==

On each of the two nodes create a partition that acts as underlying DRBD device. The partitions should have exactly the same size (I created a 10 GB partition hda3 using fdisk on each node for this example). Note that it might be necessary to reboot the machines to re-read the partition table.

Install the rpm of the DRBD userspace tools on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv drbd-0.7.17-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Then create the drbd.conf configuration file and copy it to /etc/drbd.conf on both nodes. Below is the example configuration file that is used in this article:
<pre>
resource r0 {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";

startup {
degr-wfc-timeout 120;
}

net {
on-disconnect reconnect;
}

disk {
on-io-error detach;
}

syncer {
rate 30M;
group 1;
al-extents 257;
}

on ovz-node1 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.1:7788;
meta-disk internal;
}

on ovz-node2 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.2:7788;
meta-disk internal;
}

}
</pre>
Start DRBD on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/drbd start
Starting DRBD resources: [ d0 s0 n0 ].
[root@ovz-node1 ~]#
</pre>
Then check the status of /proc/drbd:
<pre>
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Secondary ld:Inconsistent
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
[root@ovz-node1 ~]#
</pre>
Both nodes are now Secondary and Inconsistent. The latter is because the underlying storage is not yet in-sync, and DRBD has no way to know whether you want the initial sync from ovz-node1 to ovz-node2, or ovz-node2 to ovz-node1. As there is no data below it yet, it does not matter.

To start the sync from ovz-node1 to ovz-node2, do the following on ovz-node1:
<pre>
[root@ovz-node1 ~]# drbdadm -- --do-what-I-say primary all
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:SyncSource st:Primary/Secondary ld:Consistent
ns:627252 nr:0 dw:0 dr:629812 al:0 bm:38 lo:640 pe:0 ua:640 ap:0
[=>..................] sync'ed: 6.6% (8805/9418)M
finish: 0:04:51 speed: 30,888 (27,268) K/sec
[root@ovz-node1 ~]#
</pre>
As you see, DRBD syncs with about 30 MB per second, as we told it so in /etc/drbd.conf. On the SyncSource (ovz-node1 in this case) the DRBD device is already useable (although it is syncing in the background).

So you can immediately create the filesystem:
<pre>
[root@ovz-node1 ~]# mkfs.ext3 /dev/drbd0
[...]
[root@ovz-node1 ~]#
</pre>

=== Copy necessary OpenVZ files to DRBD device ===

Move the original /vz directory to /vz.orig and recreate the /vz directory to have it as a mount point '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /vz /vz.orig
[root@ovz-node1 ~]# mkdir /vz
[root@ovz-node1 ~]#
</pre>

Afterwards move the necessary OpenVZ directories (/etc/vz, /etc/sysconfig/vz-scripts, /var/vzquota) and replace them with symbolic links '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /etc/vz /etc/vz.orig
[root@ovz-node1 ~]# mv /etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts.orig
[root@ovz-node1 ~]# mv /var/vzquota /var/vzquota.orig
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/vz /etc/vz
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts
[root@ovz-node1 ~]# ln -s /vz/cluster/var/vzquota /var/vzquota
[root@ovz-node1 ~]#
</pre>

Currently, ovz-node1 is still Primary of /dev/drbd0. You can now mount it and copy the necessary files to it '''(only on ovz-node1!)''':
<pre>
[root@ovz-node1 ~]# mount /dev/drbd0 /vz
[root@ovz-node1 ~]# cp -a /vz.orig/* /vz/
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc/sysconfig
[root@ovz-node1 ~]# mkdir -p /vz/cluster/var
[root@ovz-node1 ~]# cp -a /etc/vz /vz/cluster/etc/
[root@ovz-node1 ~]# cp -a /etc/sysconfig/vz-scripts /vz/cluster/etc/sysconfig/
[root@ovz-node1 ~]# cp -a /var/vzquota /vz/cluster/var/
[root@ovz-node1 ~]# umount /dev/drbd0
[root@ovz-node1 ~]#
</pre>

== Setting up Heartbeat ==

Install the neccessary Heartbeat rpms on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv heartbeat-1.2.4-1.i386.rpm heartbeat-pils-1.2.4-1.i386.rpm heartbeat-stonith-1.2.4-1.i386.rpm
Preparing... ########################################### [100%]
1:heartbeat-pils ########################################### [ 33%]
2:heartbeat-stonith ########################################### [ 67%]
3:heartbeat ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Create the Heartbeat configuration file ha.cf and copy it to <code>/etc/ha.d/ha.cf</code> on both nodes. Details about this file can be found at http://www.linux-ha.org/ha.cf. Below is an example configuration which uses the two network connections and also a serial connection for heartbeat packets:
<pre>
# Heartbeat logging configuration
logfacility daemon

# Heartbeat cluster members
node ovz-node1
node ovz-node2

# Heartbeat communication timing
keepalive 1
warntime 10
deadtime 30
initdead 120

# Heartbeat communication paths
udpport 694
ucast eth1 192.168.255.1
ucast eth1 192.168.255.2
ucast eth0 192.168.1.201
ucast eth0 192.168.1.202
baud 19200
serial /dev/ttyS0

# Don't fail back automatically
auto_failback off

# Monitoring of network connection to default gateway
ping 192.168.1.1
respawn hacluster /usr/lib64/heartbeat/ipfail
</pre>
Create the Heartbeat configuration file authkeys and copy it to <code>/etc/ha.d/authkeys</code> on both nodes. Set the permissions of this file to 600. Details about this file can be found at http://www.linux-ha.org/authkeys. Below is an example:
<pre>
auth 1
1 sha1 PutYourSuperSecretKeyHere
</pre>
Create the Heartbeat configuration file haresources and copy it to <code>/etc/ha.d/haresources</code> on both nodes. Details about this file can be found at http://www.linux-ha.org/haresources. Note that it is not necessary to configure IPs for gratuitous arp here. The gratuitous arp is done by OpenVZ itself, through <code>/etc/sysconfig/network-scripts/ifup-venet</code> and <code>/usr/lib/vzctl/scripts/vps-functions</code>. Below is an example for the haresources file:
<pre>
ovz-node1 drbddisk::r0 Filesystem::/dev/drbd0::/vz::ext3 vz MailTo::youremail@yourdomain.tld
</pre>
Finally, you can now start heartbeat on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services:
[ OK ]
[root@ovz-node1 ~]#
</pre>

== Before going in production: testing, testing, testing, and ...hm... testing! ==

The installation of the cluster is finished at this point. Before putting the cluster in production it is very important to test the cluster. Because of all the possible different kinds of hardware that you may have, you may encounter problems when a failover is necessary. And as the cluster is about high availability, such problems must be found before the cluster is used for production.

Here is one example: The e1000 driver that is included in kernels < 2.6.12 has a problem when a cable gets unplugged while broadcast packets are still being sent out on that interface. When using broadcast communication in Heartbeat on a crossover link, this fills up the transmit ring buffer on the adapter (the buffer is full after about 8 minutes after the cable got unplugged). Using unicast communication in Heartbeat fixes the problem for example. Details see: http://www.osdl.org/developer_bugzilla/show_bug.cgi?id=699#c22

Without testing you may not be aware of such problems and may face them when the cluster is in production and a failover would be necessary. So test your cluster carefully!

Possible tests can include:
* power outage test of active node
* power outage test of passive node
* network connection outage test of eth0 of active node
* network connection outage test of eth0 of passive node
* network connection outage test of crossover network connection
* ...

As mentioned above, some problems only arise after an outage lasts longer than some minutes. So do the tests also with a duration of >1h for example.

Before you start to test, build a test plan. Some valueable information on that can be found in chapter 3 "Testing a highly available Tivoli Storage Manager cluster environment" of the Redbook ''IBM Tivoli Storage Manager in a Clustered Environment'', see http://www.redbooks.ibm.com/abstracts/sg246679.html. In this chapter it is mentioned that the experience of the authoring team is that the testing phase must be at least two times the total implementation time for the cluster.

== Before installing kernel updates: testing again ==

New OpenVZ kernel often include driver updates. This kernel for examples includes an update of the e1000 module: http://openvz.org/news/updates/kernel-022stab078.21

To avoid to overlook problems with new components (such as a newer kernel), it is necessary to re-do the tests mentioned above. But as the cluster is already in production, a second cluster (test cluster) with the same hardware as the main cluster is needed. Use this test cluster to test updates of the kernel or main OS updates for the hardware node before putting them on the production cluster.

I know this is not an easy task, as it is time-consuming and needs additional hardware only for testing. But when really business-critical applications are running on the cluster, it is very good to now that the cluster works fine also with new updates installed on the hardware node. In many cases a dedicated test cluster and the time efford for the testing of updates may cause too much costs. If you cannot do such test of updates, keep in mind that over time (when you must install security updates of the OS or the kernel) you have a cluster that you have not tested in this configuration.

If you need a tested cluster (also with tested kernel updates), you may take a look on this Virtuozzo cluster: http://www.thomas-krenn.com/cluster

== How to do OpenVZ kernel updates when it contains a new DRBD version ==

As mentioned above, it is important to use the correct version of the DRBD userspace tools. When an OpenVZ kernel contains a new DRBD version, it is important that the DRBD API version of the userspace tools matches the API version of the DRBD module that is included in the OpenVZ kernel. The API versions can be found at http://svn.drbd.org/drbd/branches/drbd-0.7/ChangeLog. The best way is to always use the version of the DRBD userspace tools that matches the version of the DRBD module that is included in the OpenVZ kernel.

In this example the initial cluster installation contained OpenVZ kernel 2.6.8-022stab078.10, which contains the DRBD module 0.7.17. The steps below show the update procedure to OpenVZ kernel 2.6.8-022stab078.14, which contains the DRBD module 0.7.20.
In the first step build the DRBD userspace tools version 0.7.20 on your buildmachine. Then stop Heartbeat and DRBD on the passive node (hint: you can use 'cat /proc/drbd' to get a hint which node is active and which one is passive):
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services:
[ OK ]
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/drbd stop
Stopping all DRBD resources.
[root@ovz-node2 ~]# cat /proc/drbd
cat: /proc/drbd: No such file or directory
[root@ovz-node2 ~]#
</pre>
Then install the new kernel and the DRBD userspace tools on this node:
<pre>
[root@ovz-node2 ~]# rpm -ihv ovzkernel-2.6.8-022stab078.14.i686.rpm
warning: ovzkernel-2.6.8-022stab078.14.i686.rpm: V3 DSA signature: NOKEY, key ID a7a1d4b6
Preparing... ########################################### [100%]
1:ovzkernel ########################################### [100%]
[root@ovz-node2 ~]# rpm -Uhv drbd-0.7.20-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
/sbin/service
Stopping all DRBD resources.
[root@ovz-node2 ~]#
</pre>
Now set the new kernel as default kernel in /etc/grub.conf and then reboot this node.

After the reboot, the new DRBD version is visible:
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.20 (api:79/proto:74)
SVN Revision: 2260 build by phil@mescal, 2006-07-04 15:18:57
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:0 nr:28 dw:28 dr:0 al:0 bm:2 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]#
</pre>

To update the other node, switch-over the services to make the current active node the passive node. Execute the following on the still active node (it could be that the hb_standby command is located in /usr/lib/heartbeat):
<pre>
[root@ovz-node1 ~]# /usr/lib64/heartbeat/hb_standby
2006/08/03_21:09:41 Going standby [all].
[root@ovz-node1 ~]#
</pre>

Now do the same steps on the new passive node to update it: stop Heartbeat and DRBD, install the new kernel and the new DRBD userspace tools, set the new kernel as default kernel in /etc/grub.conf and reboot the node.

== How to do updates of vzctl, vzctl-lib, and vzquota ==

Ensure after every update of OpenVZ tools that OpenVZ is not started on system boot. To disable starting of OpenVZ on system boot execute on both nodes:
<pre>
[root@ovz-node1 ~]# chkconfig vz off
[root@ovz-node1 ~]#
</pre>

== Live-Switchover with the help of checkpointing ==

With the help of [[Checkpointing_and_live_migration|checkpointing]] it is possible to do live switchovers.

Important: although this HOWTO currently describes the use of DRBD 0.7, it is necessary to use DRBD 8 to be able to use this live-switchover feature reliable. Some hints on using OpenVZ Kernel 2.6.18 with DRBD 8 can be found in [http://forum.openvz.org/index.php?t=msg&th=3213&start=0& this thread in the forum].

The following scripts are written by Thomas Kappelmueller. They should be placed at /root/live-switchover/ on both nodes. To activate the scripts execute the following commands on both nodes:
<pre>
[root@ovz-node1 ~]# ln -s /root/live-switchover/openvz /etc/init.d/
[root@ovz-node1 ~]# ln -s /root/live-switchover/live_switchover.sh /root/bin/
[root@ovz-node1 ~]#
</pre>

It is also necessary to replace <code>vz</code> by an adjusted initscript (<code>openvz</code> in this example). So /etc/ha.d/haresources has the following content on both nodes:
<pre>
ovz-node1 drbddisk::r0 Filesystem::/dev/drbd0::/vz::ext3 openvz MailTo::youremail@yourdomain.tld
</pre>

=== Script cluster_freeze.sh ===
<pre>
#!/bin/bash
#Script by Thomas Kappelmueller
#Version 1.0
LIVESWITCH_PATH='/vz/cluster/liveswitch'

if [ -f $LIVESWITCH_PATH ]
then
rm -f $LIVESWITCH_PATH
fi

RUNNING_VE=$(vzlist -1)

for I in $RUNNING_VE
do
BOOTLINE=$(cat /etc/sysconfig/vz-scripts/$I.conf | grep -i "^onboot")
if [ $I != 1 -a "$BOOTLINE" = "ONBOOT=\"yes\"" ]
then
vzctl chkpnt $I

if [ $? -eq 0 ]
then
vzctl set $I --onboot no --save
echo $I >> $LIVESWITCH_PATH
fi
fi
done

exit 0
</pre>

=== Script cluster_unfreeze.sh ===
<pre>
#!/bin/bash
#Script by Thomas Kappelmueller
#Version 1.0

LIVESWITCH_PATH='/vz/cluster/liveswitch'

if [ -f $LIVESWITCH_PATH ]
then
FROZEN_VE=$(cat $LIVESWITCH_PATH)
else
exit 1
fi

for I in $FROZEN_VE
do
vzctl restore $I

if [ $? != 0 ]
then
vzctl start $I
fi

vzctl set $I --onboot yes --save
done

rm -f $LIVESWITCH_PATH

exit 0
</pre>

=== Script live_switchover.sh ===
<pre>
#!/bin/bash
#Script by Thomas Kappelmueller
#Version 1.0

ps -eaf | grep 'vzctl enter' | grep -v 'grep' > /dev/null
if [ $? -eq 0 ]
then
echo 'vzctl enter is active. please finish before live switchover.'
exit 1
fi
ps -eaf | grep 'vzctl exec' | grep -v 'grep' > /dev/null
if [ $? -eq 0 ]
then
echo 'vzctl exec is active. please finish before live switchover.'
exit 1
fi
echo "Freezing VEs..."
/root/live-switchover/cluster_freeze.sh
echo "Starting Switchover..."
/usr/lib64/heartbeat/hb_standby
</pre>

=== Script openvz ===
<pre>
#!/bin/bash
#
# openvz Startup script for OpenVZ
#

start() {
/etc/init.d/vz start > /dev/null 2>&1
RETVAL=$?
/root/live-switchover/cluster_unfreeze.sh
return $RETVAL
}
stop() {
/etc/init.d/vz stop > /dev/null 2>&1
RETVAL=$?
return $RETVAL
}
status() {
/etc/init.d/vz status > /dev/null 2>&1
RETVAL=$?
return $RETVAL
}

# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status
;;
*)
echo $"Usage: openvz {start|stop|status}"
exit 1
esac

exit $RETVAL
</pre>

[[Category: HOWTO]]

Partners

2007-12-14T12:17:14Z

Wfischer: added Thomas-Krenn.AG

Here are some companies that are working together with OpenVZ project in one or another way. Feel free to add your company profile here ('''in alphabetical order''').

== Computer Tyme ==
[http://ctyme.com Computer Tyme] is using OpenVZ for their spam filtering service [http://junkemailfilter.com Junk Email Filter dot com]. They are active proponents of OpenVZ; Marc Perkel worked at the OpenVZ booth during LinuxWorld Expo 2007.

== LastSpam ==
[http://www.lastspam.com LastSpam] offers managed e-mail security services (anti-spam and anti-virus/bad content) and uses OpenVZ to deliver excellent service and support to its customers. Ugo Bellavance, LastSpam's head server architect and manager is an active member of the OpenVZ community: helping on the forum, making suggestions, and submitting patches.

== netVOICE communications ==
[http://www.netvoice.ca/ netVOICE communications] is an Internet Telephony Service Provider (ITSP) that uses OpenVZ as the basis for their [http://www.vpas.ca/ Virtual Private Asterisk Server (VPAS)] offering. They also offer consulting on running Asterisk (the leading Open Source telephony platform) on OpenVZ. netVOICE is a Digium Approved Reseller and has a Digium Certified Asterisk Professional [http://www.digium.com/en/training/certifications/ (dCAP)] on staff.

== Proxmox ==

[http://proxmox.com Proxmox] is [http://www.proxmox.com/cms_proxmox/en/virtualization/openvz/ using OpenVZ] for their spam filtering appliance (see [[Proxmox Mail Gateway in VE]]). Proxmox is also the author of [http://www.proxmox.com/cms_proxmox/en/virtualization/openvz/vzdump/ vzdump] utility.

== SpiderTools.com ==
[http://spidertools.com SpiderTools] provides training for OpenVZ servers. Students work on live servers to gain skills on how to implement OpenVZ. 20% of all sales go back to OpenVZ for development. Students will get 6 weeks of live instruction and support.

== Thomas-Krenn.AG ==
[http://www.thomas-krenn.com Thomas-Krenn.AG] is a server specialist, selling server systems and solutions. One of the solution products is a pre-installed [http://www.thomas-krenn.com/en/system-solutions/ha-linux-cluster.html cluster system], built with Virtuozzo. They published how to build such a cluster with OpenVZ at [[HA_cluster_with_DRBD_and_Heartbeat]].

== See also ==

* [[Control panels]]
* [[2006 contributions]]

HA cluster with DRBD and Heartbeat

2007-09-06T12:02:50Z

Wfischer: updated URL to german cluster documentation

This article shows how to setup a OpenVZ high availability (HA) cluster using the data replication software DRBD and the cluster manager Heartbeat. In this example the two machines building the cluster run on CentOS 4.3. The article also shows how to do kernel updates in the cluster, including necessary steps like recompiling of new DRBD userspace tools. For this purpose, kernel 2.6.8-022stab078.10 (containing DRBD module 0.7.17) is used as initial kernel version, and kernel 2.6.8-022stab078.14 (containing DRBD module 0.7.20) as updated kernel version.

Update: this howto currently does not describe details on OpenVZ Kernel 2.6.18, which contains DRBD version 8.*. Meanwhile, some hints on using OpenVZ Kernel 2.6.18 with DRBD 8 can be found in [http://forum.openvz.org/index.php?t=msg&th=3213&start=0& this thread in the forum].

Additional information about clustering of virtual machines can be found in the following paper: http://www.linuxtag.org/2006/fileadmin/linuxtag/dvd/12080-paper.pdf

Some other additional information can be found in the documentation of the Thomas-Krenn.AG cluster (The author of this howto is working in the cluster development there, that is the reason why he was able to write this howto :-). The full documentation with interesting illustrations is currently only available in German: http://www.thomas-krenn.com/en/service-support/knowledge-center/cluster/documentation.html

== Prerequisites ==
The OpenVZ kernel already includes the DRBD module. The DRBD userspace tools and the cluster manager Heartbeat must be provided seperately. As the API version of the DRBD userspace tools must exactly match the API version of the module, compile them yourself. Also compile Heartbeat yourself, as at the time of this writing the CentOS extras repository only contained an old CVS version of Heartbeat.

On a hardware node for production use there should not be any application that is not really needed for running OpenVZ (any things which are not needed by OpenVZ should run in a VE for security reasons). As a result, compile DRBD and Heartbeat on another machine running CentOS 4.3 (in this example I used a virtual machine on a VMware Server).

=== Compiling Heartbeat ===
Heartbeat version 1.2.* has successfully been used in a lot of two-node-clusters around the world. As the codebase used in version 1.2.* is in production use for many years now, the code is very stable. At the time of writing, Heartbeat version 1.2.4 is the current version of the 1.2.* branch.

Get the tar.gz of the current version of the 1.2.* branch from http://linux-ha.org/download/index.html, at the time of this writing this is http://linux-ha.org/download/heartbeat-1.2.4.tar.gz. Use rpmbuild to build the package:
<pre>rpmbuild -ta heartbeat-1.2.4.tar.gz</pre>
After that, you find four rpm packes in /usr/src/redhat/RPMS/i386 (heartbeat-1.2.4-1.i386.rpm, heartbeat-ldirectord-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, heartbeat-stonith-1.2.4-1.i386.rpm). In this example only heartbeat-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, and heartbeat-stonith-1.2.4-1.i386.rpm are needed.

=== Compiling DRBD userspace tools ===
When compiling the DRBD userspace tools, you have to take care to take the version that matches the DRBD version that is included in the OpenVZ kernel you want to use. If you are unsure about the version, do the following steps while running the OpenVZ kernel that you want to use on a test machine (I used another virtual machine on a VMware server to try this):
<pre>
[root@testmachine ~]# cat /proc/version
Linux version 2.6.8-022stab078.10 (root@rhel4-32) (gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)) #1 Wed Jun 21 12:01:20 MSD 2006
[root@testmachine ~]# modprobe drbd
[root@testmachine ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Unconfigured
1: cs:Unconfigured
[root@testmachine ~]# rmmod drbd
[root@testmachine ~]#
</pre>
Here the version of the DRBD module is 0.7.17. So the userspace tools for 0.7.17 are neccessary.

Back on the buildmachine, do the following to create the rpm:
<pre>
[root@buildmachine ~]# yum install kernel-devel gcc bison flex
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Nothing to do
[root@buildmachine ~]# tar xfz drbd-0.7.17.tar.gz
[root@buildmachine ~]# cd drbd-0.7.17
[root@buildmachine drbd-0.7.17]# make rpm
[...]
You have now:
-rw-r--r-- 1 root root 288728 Jul 30 10:40 dist/RPMS/i386/drbd-0.7.17-1.i386.rpm
-rw-r--r-- 1 root root 518369 Jul 30 10:40 dist/RPMS/i386/drbd-km-2.6.9_34.0.2.EL-0.7.17-1.i386.rpm
[root@buildmachine drbd-0.7.17]#
</pre>
Note that in this way the kernel-devel from CentOS is used, but this does not matter as the created drbd-km rpm will not be used (the DRBD kernel module is already included in OpenVZ kernel). If the kernel-devel package is not the same version as the kernel package that is currently running, it is possible to execute 'make rpm KDIR=/usr/src/kernels/2.6.9-34.0.2.EL-i686/' to directly point to the kernel sources.

== Installing the two nodes ==
Install the two machines in the same way as you would install them for a normal OpenVZ installation, but do not create a filesystem for the /vz. This filesystem will be installed later on on top of DRBD.
{| class="wikitable"
|+'''Example installation configuration'''
! Parameter !! node1 !! node2
|-
! hostname
| ovz-node1
| ovz-node2
|-
! / filesystem
| hda1, 10 GB
| hda1, 10 GB
|-
! swap space
| hda2, 2048 MB
| hda2, 2048 MB
|-
! public LAN
| eth0, 192.168.1.201
| eth0, 192.168.1.202
|-
! private LAN
| eth1, 192.168.255.1 (Gbit Ethernet)
| eth1, 192.168.255.2 (Gbit Ethernet)
|-
! other install options
| no firewall, no SELinux
| no firewall, no SELinux
|-
! package groups
| deactivated everything, only kept vim-enhanced
| deactivated everything, only kept vim-enhanced
|}

== Installing OpenVZ ==

Get the OpenVZ kernel and utilities and install them on both nodes, as described in [[quick installation]]. Update grub configuration to use the OpenVZ kernel by default. Disable starting of OpenVZ on system boot on both nodes (OpenVZ will be started and stopped by Heartbeat):
<pre>
[root@ovz-node1 ~]# chkconfig vz off
[root@ovz-node1 ~]#
</pre>
Then reboot both machines.

== Setting up DRBD ==

On each of the two nodes create a partition that acts as underlying DRBD device. The partitions should have exactly the same size (I created a 10 GB partition hda3 using fdisk on each node for this example). Note that it might be necessary to reboot the machines to re-read the partition table.

Install the rpm of the DRBD userspace tools on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv drbd-0.7.17-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Then create the drbd.conf configuration file and copy it to /etc/drbd.conf on both nodes. Below is the example configuration file that is used in this article:
<pre>
resource r0 {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";

startup {
degr-wfc-timeout 120;
}

net {
on-disconnect reconnect;
}

disk {
on-io-error detach;
}

syncer {
rate 30M;
group 1;
al-extents 257;
}

on ovz-node1 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.1:7788;
meta-disk internal;
}

on ovz-node2 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.2:7788;
meta-disk internal;
}

}
</pre>
Start DRBD on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/drbd start
Starting DRBD resources: [ d0 s0 n0 ].
[root@ovz-node1 ~]#
</pre>
Then check the status of /proc/drbd:
<pre>
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Secondary ld:Inconsistent
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
[root@ovz-node1 ~]#
</pre>
Both nodes are now Secondary and Inconsistent. The latter is because the underlying storage is not yet in-sync, and DRBD has no way to know whether you want the initial sync from ovz-node1 to ovz-node2, or ovz-node2 to ovz-node1. As there is no data below it yet, it does not matter.

To start the sync from ovz-node1 to ovz-node2, do the following on ovz-node1:
<pre>
[root@ovz-node1 ~]# drbdadm -- --do-what-I-say primary all
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:SyncSource st:Primary/Secondary ld:Consistent
ns:627252 nr:0 dw:0 dr:629812 al:0 bm:38 lo:640 pe:0 ua:640 ap:0
[=>..................] sync'ed: 6.6% (8805/9418)M
finish: 0:04:51 speed: 30,888 (27,268) K/sec
[root@ovz-node1 ~]#
</pre>
As you see, DRBD syncs with about 30 MB per second, as we told it so in /etc/drbd.conf. On the SyncSource (ovz-node1 in this case) the DRBD device is already useable (although it is syncing in the background).

So you can immediately create the filesystem:
<pre>
[root@ovz-node1 ~]# mkfs.ext3 /dev/drbd0
[...]
[root@ovz-node1 ~]#
</pre>

=== Copy necessary OpenVZ files to DRBD device ===

Move the original /vz directory to /vz.orig and recreate the /vz directory to have it as a mount point '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /vz /vz.orig
[root@ovz-node1 ~]# mkdir /vz
[root@ovz-node1 ~]#
</pre>

Afterwards move the necessary OpenVZ directories (/etc/vz, /etc/sysconfig/vz-scripts, /var/vzquota) and replace them with symbolic links '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /etc/vz /etc/vz.orig
[root@ovz-node1 ~]# mv /etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts.orig
[root@ovz-node1 ~]# mv /var/vzquota /var/vzquota.orig
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/vz /etc/vz
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts
[root@ovz-node1 ~]# ln -s /vz/cluster/var/vzquota /var/vzquota
[root@ovz-node1 ~]#
</pre>

Currently, ovz-node1 is still Primary of /dev/drbd0. You can now mount it and copy the necessary files to it '''(only on ovz-node1!)''':
<pre>
[root@ovz-node1 ~]# mount /dev/drbd0 /vz
[root@ovz-node1 ~]# cp -a /vz.orig/* /vz/
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc/sysconfig
[root@ovz-node1 ~]# mkdir -p /vz/cluster/var
[root@ovz-node1 ~]# cp -a /etc/vz /vz/cluster/etc/
[root@ovz-node1 ~]# cp -a /etc/sysconfig/vz-scripts /vz/cluster/etc/sysconfig/
[root@ovz-node1 ~]# cp -a /var/vzquota /vz/cluster/var/
[root@ovz-node1 ~]# umount /dev/drbd0
[root@ovz-node1 ~]#
</pre>

== Setting up Heartbeat ==

Install the neccessary Heartbeat rpms on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv heartbeat-1.2.4-1.i386.rpm heartbeat-pils-1.2.4-1.i386.rpm heartbeat-stonith-1.2.4-1.i386.rpm
Preparing... ########################################### [100%]
1:heartbeat-pils ########################################### [ 33%]
2:heartbeat-stonith ########################################### [ 67%]
3:heartbeat ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Create the Heartbeat configuration file ha.cf and copy it to <code>/etc/ha.d/ha.cf</code> on both nodes. Details about this file can be found at http://www.linux-ha.org/ha.cf. Below is an example configuration which uses the two network connections and also a serial connection for heartbeat packets:
<pre>
# Heartbeat logging configuration
logfacility daemon

# Heartbeat cluster members
node ovz-node1
node ovz-node2

# Heartbeat communication timing
keepalive 1
warntime 10
deadtime 30
initdead 120

# Heartbeat communication paths
udpport 694
ucast eth1 192.168.255.1
ucast eth1 192.168.255.2
ucast eth0 192.168.1.201
ucast eth0 192.168.1.202
baud 19200
serial /dev/ttyS0

# Don't fail back automatically
auto_failback off

# Monitoring of network connection to default gateway
ping 192.168.1.1
respawn hacluster /usr/lib64/heartbeat/ipfail
</pre>
Create the Heartbeat configuration file authkeys and copy it to <code>/etc/ha.d/authkeys</code> on both nodes. Set the permissions of this file to 600. Details about this file can be found at http://www.linux-ha.org/authkeys. Below is an example:
<pre>
auth 1
1 sha1 PutYourSuperSecretKeyHere
</pre>
Create the Heartbeat configuration file haresources and copy it to <code>/etc/ha.d/haresources</code> on both nodes. Details about this file can be found at http://www.linux-ha.org/haresources. Note that it is not necessary to configure IPs for gratuitous arp here. The gratuitous arp is done by OpenVZ itself, through <code>/etc/sysconfig/network-scripts/ifup-venet</code> and <code>/usr/lib/vzctl/scripts/vps-functions</code>. Below is an example for the haresources file:
<pre>
ovz-node1 drbddisk::r0 Filesystem::/dev/drbd0::/vz::ext3 vz MailTo::youremail@yourdomain.tld
</pre>
Finally, you can now start heartbeat on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services:
[ OK ]
[root@ovz-node1 ~]#
</pre>

== Before going in production: testing, testing, testing, and ...hm... testing! ==

The installation of the cluster is finished at this point. Before putting the cluster in production it is very important to test the cluster. Because of all the possible different kinds of hardware that you may have, you may encounter problems when a failover is necessary. And as the cluster is about high availability, such problems must be found before the cluster is used for production.

Here is one example: The e1000 driver that is included in kernels < 2.6.12 has a problem when a cable gets unplugged while broadcast packets are still being sent out on that interface. When using broadcast communication in Heartbeat on a crossover link, this fills up the transmit ring buffer on the adapter (the buffer is full after about 8 minutes after the cable got unplugged). Using unicast communication in Heartbeat fixes the problem for example. Details see: http://www.osdl.org/developer_bugzilla/show_bug.cgi?id=699#c22

Without testing you may not be aware of such problems and may face them when the cluster is in production and a failover would be necessary. So test your cluster carefully!

Possible tests can include:
* power outage test of active node
* power outage test of passive node
* network connection outage test of eth0 of active node
* network connection outage test of eth0 of passive node
* network connection outage test of crossover network connection
* ...

As mentioned above, some problems only arise after an outage lasts longer than some minutes. So do the tests also with a duration of >1h for example.

Before you start to test, build a test plan. Some valueable information on that can be found in chapter 3 "Testing a highly available Tivoli Storage Manager cluster environment" of the Redbook ''IBM Tivoli Storage Manager in a Clustered Environment'', see http://www.redbooks.ibm.com/abstracts/sg246679.html. In this chapter it is mentioned that the experience of the authoring team is that the testing phase must be at least two times the total implementation time for the cluster.

== Before installing kernel updates: testing again ==

New OpenVZ kernel often include driver updates. This kernel for examples includes an update of the e1000 module: http://openvz.org/news/updates/kernel-022stab078.21

To avoid to overlook problems with new components (such as a newer kernel), it is necessary to re-do the tests mentioned above. But as the cluster is already in production, a second cluster (test cluster) with the same hardware as the main cluster is needed. Use this test cluster to test updates of the kernel or main OS updates for the hardware node before putting them on the production cluster.

I know this is not an easy task, as it is time-consuming and needs additional hardware only for testing. But when really business-critical applications are running on the cluster, it is very good to now that the cluster works fine also with new updates installed on the hardware node. In many cases a dedicated test cluster and the time efford for the testing of updates may cause too much costs. If you cannot do such test of updates, keep in mind that over time (when you must install security updates of the OS or the kernel) you have a cluster that you have not tested in this configuration.

If you need a tested cluster (also with tested kernel updates), you may take a look on this Virtuozzo cluster: http://www.thomas-krenn.com/cluster

== How to do OpenVZ kernel updates when it contains a new DRBD version ==

As mentioned above, it is important to use the correct version of the DRBD userspace tools. When an OpenVZ kernel contains a new DRBD version, it is important that the DRBD API version of the userspace tools matches the API version of the DRBD module that is included in the OpenVZ kernel. The API versions can be found at http://svn.drbd.org/drbd/branches/drbd-0.7/ChangeLog. The best way is to always use the version of the DRBD userspace tools that matches the version of the DRBD module that is included in the OpenVZ kernel.

In this example the initial cluster installation contained OpenVZ kernel 2.6.8-022stab078.10, which contains the DRBD module 0.7.17. The steps below show the update procedure to OpenVZ kernel 2.6.8-022stab078.14, which contains the DRBD module 0.7.20.
In the first step build the DRBD userspace tools version 0.7.20 on your buildmachine. Then stop Heartbeat and DRBD on the passive node (hint: you can use 'cat /proc/drbd' to get a hint which node is active and which one is passive):
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services:
[ OK ]
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/drbd stop
Stopping all DRBD resources.
[root@ovz-node2 ~]# cat /proc/drbd
cat: /proc/drbd: No such file or directory
[root@ovz-node2 ~]#
</pre>
Then install the new kernel and the DRBD userspace tools on this node:
<pre>
[root@ovz-node2 ~]# rpm -ihv ovzkernel-2.6.8-022stab078.14.i686.rpm
warning: ovzkernel-2.6.8-022stab078.14.i686.rpm: V3 DSA signature: NOKEY, key ID a7a1d4b6
Preparing... ########################################### [100%]
1:ovzkernel ########################################### [100%]
[root@ovz-node2 ~]# rpm -Uhv drbd-0.7.20-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
/sbin/service
Stopping all DRBD resources.
[root@ovz-node2 ~]#
</pre>
Now set the new kernel as default kernel in /etc/grub.conf and then reboot this node.

After the reboot, the new DRBD version is visible:
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.20 (api:79/proto:74)
SVN Revision: 2260 build by phil@mescal, 2006-07-04 15:18:57
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:0 nr:28 dw:28 dr:0 al:0 bm:2 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]#
</pre>

To update the other node, switch-over the services to make the current active node the passive node. Execute the following on the still active node (it could be that the hb_standby command is located in /usr/lib/heartbeat):
<pre>
[root@ovz-node1 ~]# /usr/lib64/heartbeat/hb_standby
2006/08/03_21:09:41 Going standby [all].
[root@ovz-node1 ~]#
</pre>

Now do the same steps on the new passive node to update it: stop Heartbeat and DRBD, install the new kernel and the new DRBD userspace tools, set the new kernel as default kernel in /etc/grub.conf and reboot the node.

== How to do updates of vzctl, vzctl-lib, and vzquota ==

Ensure after every update of OpenVZ tools that OpenVZ is not started on system boot. To disable starting of OpenVZ on system boot execute on both nodes:
<pre>
[root@ovz-node1 ~]# chkconfig vz off
[root@ovz-node1 ~]#
</pre>

[[Category: HOWTO]]

HA cluster with DRBD and Heartbeat

2007-09-06T09:26:10Z

Wfischer: added a hint on DRBD 8 which comes with OpenVZ 2.6.18 (added a link to a thread in the forum)

This article shows how to setup a OpenVZ high availability (HA) cluster using the data replication software DRBD and the cluster manager Heartbeat. In this example the two machines building the cluster run on CentOS 4.3. The article also shows how to do kernel updates in the cluster, including necessary steps like recompiling of new DRBD userspace tools. For this purpose, kernel 2.6.8-022stab078.10 (containing DRBD module 0.7.17) is used as initial kernel version, and kernel 2.6.8-022stab078.14 (containing DRBD module 0.7.20) as updated kernel version.

Update: this howto currently does not describe details on OpenVZ Kernel 2.6.18, which contains DRBD version 8.*. Meanwhile, some hints on using OpenVZ Kernel 2.6.18 with DRBD 8 can be found in [http://forum.openvz.org/index.php?t=msg&th=3213&start=0& this thread in the forum].

Additional information about clustering of virtual machines can be found in the following paper: http://www.linuxtag.org/2006/fileadmin/linuxtag/dvd/12080-paper.pdf

Some other additional information can be found in the documentation of the Thomas-Krenn.AG cluster (The author of this howto is working in the cluster development there, that is the reason why he was able to write this howto :-). The full documentation with interesting illustrations is currently only available in German:
http://my.thomas-krenn.com/service_support/index.php/page.242

== Prerequisites ==
The OpenVZ kernel already includes the DRBD module. The DRBD userspace tools and the cluster manager Heartbeat must be provided seperately. As the API version of the DRBD userspace tools must exactly match the API version of the module, compile them yourself. Also compile Heartbeat yourself, as at the time of this writing the CentOS extras repository only contained an old CVS version of Heartbeat.

On a hardware node for production use there should not be any application that is not really needed for running OpenVZ (any things which are not needed by OpenVZ should run in a VE for security reasons). As a result, compile DRBD and Heartbeat on another machine running CentOS 4.3 (in this example I used a virtual machine on a VMware Server).

=== Compiling Heartbeat ===
Heartbeat version 1.2.* has successfully been used in a lot of two-node-clusters around the world. As the codebase used in version 1.2.* is in production use for many years now, the code is very stable. At the time of writing, Heartbeat version 1.2.4 is the current version of the 1.2.* branch.

Get the tar.gz of the current version of the 1.2.* branch from http://linux-ha.org/download/index.html, at the time of this writing this is http://linux-ha.org/download/heartbeat-1.2.4.tar.gz. Use rpmbuild to build the package:
<pre>rpmbuild -ta heartbeat-1.2.4.tar.gz</pre>
After that, you find four rpm packes in /usr/src/redhat/RPMS/i386 (heartbeat-1.2.4-1.i386.rpm, heartbeat-ldirectord-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, heartbeat-stonith-1.2.4-1.i386.rpm). In this example only heartbeat-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, and heartbeat-stonith-1.2.4-1.i386.rpm are needed.

=== Compiling DRBD userspace tools ===
When compiling the DRBD userspace tools, you have to take care to take the version that matches the DRBD version that is included in the OpenVZ kernel you want to use. If you are unsure about the version, do the following steps while running the OpenVZ kernel that you want to use on a test machine (I used another virtual machine on a VMware server to try this):
<pre>
[root@testmachine ~]# cat /proc/version
Linux version 2.6.8-022stab078.10 (root@rhel4-32) (gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)) #1 Wed Jun 21 12:01:20 MSD 2006
[root@testmachine ~]# modprobe drbd
[root@testmachine ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Unconfigured
1: cs:Unconfigured
[root@testmachine ~]# rmmod drbd
[root@testmachine ~]#
</pre>
Here the version of the DRBD module is 0.7.17. So the userspace tools for 0.7.17 are neccessary.

Back on the buildmachine, do the following to create the rpm:
<pre>
[root@buildmachine ~]# yum install kernel-devel gcc bison flex
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Nothing to do
[root@buildmachine ~]# tar xfz drbd-0.7.17.tar.gz
[root@buildmachine ~]# cd drbd-0.7.17
[root@buildmachine drbd-0.7.17]# make rpm
[...]
You have now:
-rw-r--r-- 1 root root 288728 Jul 30 10:40 dist/RPMS/i386/drbd-0.7.17-1.i386.rpm
-rw-r--r-- 1 root root 518369 Jul 30 10:40 dist/RPMS/i386/drbd-km-2.6.9_34.0.2.EL-0.7.17-1.i386.rpm
[root@buildmachine drbd-0.7.17]#
</pre>
Note that in this way the kernel-devel from CentOS is used, but this does not matter as the created drbd-km rpm will not be used (the DRBD kernel module is already included in OpenVZ kernel). If the kernel-devel package is not the same version as the kernel package that is currently running, it is possible to execute 'make rpm KDIR=/usr/src/kernels/2.6.9-34.0.2.EL-i686/' to directly point to the kernel sources.

== Installing the two nodes ==
Install the two machines in the same way as you would install them for a normal OpenVZ installation, but do not create a filesystem for the /vz. This filesystem will be installed later on on top of DRBD.
{| class="wikitable"
|+'''Example installation configuration'''
! Parameter !! node1 !! node2
|-
! hostname
| ovz-node1
| ovz-node2
|-
! / filesystem
| hda1, 10 GB
| hda1, 10 GB
|-
! swap space
| hda2, 2048 MB
| hda2, 2048 MB
|-
! public LAN
| eth0, 192.168.1.201
| eth0, 192.168.1.202
|-
! private LAN
| eth1, 192.168.255.1 (Gbit Ethernet)
| eth1, 192.168.255.2 (Gbit Ethernet)
|-
! other install options
| no firewall, no SELinux
| no firewall, no SELinux
|-
! package groups
| deactivated everything, only kept vim-enhanced
| deactivated everything, only kept vim-enhanced
|}

== Installing OpenVZ ==

Get the OpenVZ kernel and utilities and install them on both nodes, as described in [[quick installation]]. Update grub configuration to use the OpenVZ kernel by default. Disable starting of OpenVZ on system boot on both nodes (OpenVZ will be started and stopped by Heartbeat):
<pre>
[root@ovz-node1 ~]# chkconfig vz off
[root@ovz-node1 ~]#
</pre>
Then reboot both machines.

== Setting up DRBD ==

On each of the two nodes create a partition that acts as underlying DRBD device. The partitions should have exactly the same size (I created a 10 GB partition hda3 using fdisk on each node for this example). Note that it might be necessary to reboot the machines to re-read the partition table.

Install the rpm of the DRBD userspace tools on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv drbd-0.7.17-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Then create the drbd.conf configuration file and copy it to /etc/drbd.conf on both nodes. Below is the example configuration file that is used in this article:
<pre>
resource r0 {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";

startup {
degr-wfc-timeout 120;
}

net {
on-disconnect reconnect;
}

disk {
on-io-error detach;
}

syncer {
rate 30M;
group 1;
al-extents 257;
}

on ovz-node1 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.1:7788;
meta-disk internal;
}

on ovz-node2 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.2:7788;
meta-disk internal;
}

}
</pre>
Start DRBD on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/drbd start
Starting DRBD resources: [ d0 s0 n0 ].
[root@ovz-node1 ~]#
</pre>
Then check the status of /proc/drbd:
<pre>
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Secondary ld:Inconsistent
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
[root@ovz-node1 ~]#
</pre>
Both nodes are now Secondary and Inconsistent. The latter is because the underlying storage is not yet in-sync, and DRBD has no way to know whether you want the initial sync from ovz-node1 to ovz-node2, or ovz-node2 to ovz-node1. As there is no data below it yet, it does not matter.

To start the sync from ovz-node1 to ovz-node2, do the following on ovz-node1:
<pre>
[root@ovz-node1 ~]# drbdadm -- --do-what-I-say primary all
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:SyncSource st:Primary/Secondary ld:Consistent
ns:627252 nr:0 dw:0 dr:629812 al:0 bm:38 lo:640 pe:0 ua:640 ap:0
[=>..................] sync'ed: 6.6% (8805/9418)M
finish: 0:04:51 speed: 30,888 (27,268) K/sec
[root@ovz-node1 ~]#
</pre>
As you see, DRBD syncs with about 30 MB per second, as we told it so in /etc/drbd.conf. On the SyncSource (ovz-node1 in this case) the DRBD device is already useable (although it is syncing in the background).

So you can immediately create the filesystem:
<pre>
[root@ovz-node1 ~]# mkfs.ext3 /dev/drbd0
[...]
[root@ovz-node1 ~]#
</pre>

=== Copy necessary OpenVZ files to DRBD device ===

Move the original /vz directory to /vz.orig and recreate the /vz directory to have it as a mount point '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /vz /vz.orig
[root@ovz-node1 ~]# mkdir /vz
[root@ovz-node1 ~]#
</pre>

Afterwards move the necessary OpenVZ directories (/etc/vz, /etc/sysconfig/vz-scripts, /var/vzquota) and replace them with symbolic links '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /etc/vz /etc/vz.orig
[root@ovz-node1 ~]# mv /etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts.orig
[root@ovz-node1 ~]# mv /var/vzquota /var/vzquota.orig
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/vz /etc/vz
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts
[root@ovz-node1 ~]# ln -s /vz/cluster/var/vzquota /var/vzquota
[root@ovz-node1 ~]#
</pre>

Currently, ovz-node1 is still Primary of /dev/drbd0. You can now mount it and copy the necessary files to it '''(only on ovz-node1!)''':
<pre>
[root@ovz-node1 ~]# mount /dev/drbd0 /vz
[root@ovz-node1 ~]# cp -a /vz.orig/* /vz/
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc/sysconfig
[root@ovz-node1 ~]# mkdir -p /vz/cluster/var
[root@ovz-node1 ~]# cp -a /etc/vz /vz/cluster/etc/
[root@ovz-node1 ~]# cp -a /etc/sysconfig/vz-scripts /vz/cluster/etc/sysconfig/
[root@ovz-node1 ~]# cp -a /var/vzquota /vz/cluster/var/
[root@ovz-node1 ~]# umount /dev/drbd0
[root@ovz-node1 ~]#
</pre>

== Setting up Heartbeat ==

Install the neccessary Heartbeat rpms on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv heartbeat-1.2.4-1.i386.rpm heartbeat-pils-1.2.4-1.i386.rpm heartbeat-stonith-1.2.4-1.i386.rpm
Preparing... ########################################### [100%]
1:heartbeat-pils ########################################### [ 33%]
2:heartbeat-stonith ########################################### [ 67%]
3:heartbeat ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Create the Heartbeat configuration file ha.cf and copy it to <code>/etc/ha.d/ha.cf</code> on both nodes. Details about this file can be found at http://www.linux-ha.org/ha.cf. Below is an example configuration which uses the two network connections and also a serial connection for heartbeat packets:
<pre>
# Heartbeat logging configuration
logfacility daemon

# Heartbeat cluster members
node ovz-node1
node ovz-node2

# Heartbeat communication timing
keepalive 1
warntime 10
deadtime 30
initdead 120

# Heartbeat communication paths
udpport 694
ucast eth1 192.168.255.1
ucast eth1 192.168.255.2
ucast eth0 192.168.1.201
ucast eth0 192.168.1.202
baud 19200
serial /dev/ttyS0

# Don't fail back automatically
auto_failback off

# Monitoring of network connection to default gateway
ping 192.168.1.1
respawn hacluster /usr/lib64/heartbeat/ipfail
</pre>
Create the Heartbeat configuration file authkeys and copy it to <code>/etc/ha.d/authkeys</code> on both nodes. Set the permissions of this file to 600. Details about this file can be found at http://www.linux-ha.org/authkeys. Below is an example:
<pre>
auth 1
1 sha1 PutYourSuperSecretKeyHere
</pre>
Create the Heartbeat configuration file haresources and copy it to <code>/etc/ha.d/haresources</code> on both nodes. Details about this file can be found at http://www.linux-ha.org/haresources. Note that it is not necessary to configure IPs for gratuitous arp here. The gratuitous arp is done by OpenVZ itself, through <code>/etc/sysconfig/network-scripts/ifup-venet</code> and <code>/usr/lib/vzctl/scripts/vps-functions</code>. Below is an example for the haresources file:
<pre>
ovz-node1 drbddisk::r0 Filesystem::/dev/drbd0::/vz::ext3 vz MailTo::youremail@yourdomain.tld
</pre>
Finally, you can now start heartbeat on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services:
[ OK ]
[root@ovz-node1 ~]#
</pre>

== Before going in production: testing, testing, testing, and ...hm... testing! ==

The installation of the cluster is finished at this point. Before putting the cluster in production it is very important to test the cluster. Because of all the possible different kinds of hardware that you may have, you may encounter problems when a failover is necessary. And as the cluster is about high availability, such problems must be found before the cluster is used for production.

Here is one example: The e1000 driver that is included in kernels < 2.6.12 has a problem when a cable gets unplugged while broadcast packets are still being sent out on that interface. When using broadcast communication in Heartbeat on a crossover link, this fills up the transmit ring buffer on the adapter (the buffer is full after about 8 minutes after the cable got unplugged). Using unicast communication in Heartbeat fixes the problem for example. Details see: http://www.osdl.org/developer_bugzilla/show_bug.cgi?id=699#c22

Without testing you may not be aware of such problems and may face them when the cluster is in production and a failover would be necessary. So test your cluster carefully!

Possible tests can include:
* power outage test of active node
* power outage test of passive node
* network connection outage test of eth0 of active node
* network connection outage test of eth0 of passive node
* network connection outage test of crossover network connection
* ...

As mentioned above, some problems only arise after an outage lasts longer than some minutes. So do the tests also with a duration of >1h for example.

Before you start to test, build a test plan. Some valueable information on that can be found in chapter 3 "Testing a highly available Tivoli Storage Manager cluster environment" of the Redbook ''IBM Tivoli Storage Manager in a Clustered Environment'', see http://www.redbooks.ibm.com/abstracts/sg246679.html. In this chapter it is mentioned that the experience of the authoring team is that the testing phase must be at least two times the total implementation time for the cluster.

== Before installing kernel updates: testing again ==

New OpenVZ kernel often include driver updates. This kernel for examples includes an update of the e1000 module: http://openvz.org/news/updates/kernel-022stab078.21

To avoid to overlook problems with new components (such as a newer kernel), it is necessary to re-do the tests mentioned above. But as the cluster is already in production, a second cluster (test cluster) with the same hardware as the main cluster is needed. Use this test cluster to test updates of the kernel or main OS updates for the hardware node before putting them on the production cluster.

I know this is not an easy task, as it is time-consuming and needs additional hardware only for testing. But when really business-critical applications are running on the cluster, it is very good to now that the cluster works fine also with new updates installed on the hardware node. In many cases a dedicated test cluster and the time efford for the testing of updates may cause too much costs. If you cannot do such test of updates, keep in mind that over time (when you must install security updates of the OS or the kernel) you have a cluster that you have not tested in this configuration.

If you need a tested cluster (also with tested kernel updates), you may take a look on this Virtuozzo cluster: http://www.thomas-krenn.com/cluster

== How to do OpenVZ kernel updates when it contains a new DRBD version ==

As mentioned above, it is important to use the correct version of the DRBD userspace tools. When an OpenVZ kernel contains a new DRBD version, it is important that the DRBD API version of the userspace tools matches the API version of the DRBD module that is included in the OpenVZ kernel. The API versions can be found at http://svn.drbd.org/drbd/branches/drbd-0.7/ChangeLog. The best way is to always use the version of the DRBD userspace tools that matches the version of the DRBD module that is included in the OpenVZ kernel.

In this example the initial cluster installation contained OpenVZ kernel 2.6.8-022stab078.10, which contains the DRBD module 0.7.17. The steps below show the update procedure to OpenVZ kernel 2.6.8-022stab078.14, which contains the DRBD module 0.7.20.
In the first step build the DRBD userspace tools version 0.7.20 on your buildmachine. Then stop Heartbeat and DRBD on the passive node (hint: you can use 'cat /proc/drbd' to get a hint which node is active and which one is passive):
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services:
[ OK ]
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/drbd stop
Stopping all DRBD resources.
[root@ovz-node2 ~]# cat /proc/drbd
cat: /proc/drbd: No such file or directory
[root@ovz-node2 ~]#
</pre>
Then install the new kernel and the DRBD userspace tools on this node:
<pre>
[root@ovz-node2 ~]# rpm -ihv ovzkernel-2.6.8-022stab078.14.i686.rpm
warning: ovzkernel-2.6.8-022stab078.14.i686.rpm: V3 DSA signature: NOKEY, key ID a7a1d4b6
Preparing... ########################################### [100%]
1:ovzkernel ########################################### [100%]
[root@ovz-node2 ~]# rpm -Uhv drbd-0.7.20-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
/sbin/service
Stopping all DRBD resources.
[root@ovz-node2 ~]#
</pre>
Now set the new kernel as default kernel in /etc/grub.conf and then reboot this node.

After the reboot, the new DRBD version is visible:
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.20 (api:79/proto:74)
SVN Revision: 2260 build by phil@mescal, 2006-07-04 15:18:57
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:0 nr:28 dw:28 dr:0 al:0 bm:2 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]#
</pre>

To update the other node, switch-over the services to make the current active node the passive node. Execute the following on the still active node (it could be that the hb_standby command is located in /usr/lib/heartbeat):
<pre>
[root@ovz-node1 ~]# /usr/lib64/heartbeat/hb_standby
2006/08/03_21:09:41 Going standby [all].
[root@ovz-node1 ~]#
</pre>

Now do the same steps on the new passive node to update it: stop Heartbeat and DRBD, install the new kernel and the new DRBD userspace tools, set the new kernel as default kernel in /etc/grub.conf and reboot the node.

== How to do updates of vzctl, vzctl-lib, and vzquota ==

Ensure after every update of OpenVZ tools that OpenVZ is not started on system boot. To disable starting of OpenVZ on system boot execute on both nodes:
<pre>
[root@ovz-node1 ~]# chkconfig vz off
[root@ovz-node1 ~]#
</pre>

[[Category: HOWTO]]

Kernel build

2007-02-19T13:45:59Z

Wfischer: fixed info on modifying kernel configs, corrected ksubrelease (may not contain "-" character)

This FAQ will help you in case you want to apply some patches to the kernel on your own or rebuild it from sources.
On RPM based distros such as RedHat Enterprise Linux/CentOS, Fedora Core or SUSE one can simpy rebuild kernel from SRPM.
For other distros it is required to install sources, build and install kernel manually. The below are given the details for both cases.

== Rebuilding kernel from SRPM ==

'''Note''': some of the paths below include $TOPDIR, which is distribution-dependent and can be further redefined by user. To find out the proper location on your system, issue this command:

<pre>rpm --eval "%{_topdir}"</pre>

=== Download ===
Source RPMS for different OpenVZ kernel branches can be downloaded from http://openvz.org/download/kernel/. You can also access http://download.openvz.org/kernel/ directly, or use one of the [[download mirrors|mirrors]].

=== Installation ===
Install the downloaded SRC RPM with the following command:
<pre>
# rpm -ihv ovzkernel-2.6.16-026test018.1.src.rpm
</pre>

After successfull installation, you can usually find kernel sources in <code>$TOPDIR/SOURCES/</code>
and kernel spec file (<code>kernel-ovz.spec</code>) in <code>$TOPDIR/SPECS</code>.

=== Adding your own patches ===
To modify the kernel one needs just to add specific patches to the kernel spec file and put this patch into <code>$TOPDIR/SOURCES</code> directory.

Put your patch into SOURCES directory with the following command:
<pre>
# cp <patch> $TOPDIR/SOURCES/
</pre>

Then open spec file <code>$TOPDIR/SPECS/kernel-ovz.spec</code> in the editor and add the following lines:
<pre>
Patch10000: <patch-name>
</pre>
and
<pre>
%patch10000 -p1
</pre>
in appropriate places where similar text lines are.

=== Adjust kernel version ===
Before rebuilding the kernel make sure that you adjusted the kernel version in <code>kernel-ovz.spec</code>.
This will help you to distinguish binaries then from already existing kernels
(or from the official OpenVZ kernels). To do so, edit the <code>$TOPDIR/SPECS/kernel-ovz.spec</code> file and replace the following line:
<pre>
%define ksubrelease 1
</pre>
with something like
<pre>
%define ksubrelease 1my.kernel.v1
</pre>

=== Modifying configs ===
If you want to modify the kernel config, you need to do the following before you continue with the next step "Building RPMs".
<pre>
# cd $TOPDIR/SPECS
# rpmbuild -bp kernel-ovz.spec
# cd $TOPDIR/BUILD/kernel-2.6.16/linux-2.6.16
</pre>
There you will find the configuration files in the subdirectory <code>config/*.config</code>. Copy the one you want to modifiy to <code>$TOPDIR/BUILD/kernel-2.6.16/linux-2.6.16/.config</code>. Then you can do a make menuconfig or something similar to adjust the kernel configuration. Afterwards copy
<code>$TOPDIR/BUILD/kernel-2.6.16/linux-2.6.16/.config</code> to the <code>$TOPDIR/SOURCES</code> directory, but use the corresponding file name in the target directory.
Some background information on this procedure can be found in the following thread: http://www.arcknowledge.com/gmane.comp.audio.planetccrma.general/2004-11/msg00018.html

=== Building RPMs ===
To rebuild the kernel, type the following commands:
<pre>
# cd $TOPDIR/SPECS
# rpmbuild -ba --target=i686 kernel-ovz.spec
</pre>

After successfull kernel compilation binary RPMs can be found at <code>$TOPDIR/RPMS/i686</code>.

== Rebuilding kernel from sources ==

=== Download ===
To compile OpenVZ linux kernel one need to download the original linux kernel sources and OpenVZ patches for it.

Linux kernel can be found at [http://www.kernel.org/ kernel.org], e.g. 2.6.16 kernel can be downloaded from [http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.tar.bz2 linux-2.6.16.tar.bz2].

Appropriate OpenVZ patches for this kernel version can be found at http://openvz.org/download/kernel/<version>/patches/. For example, at the moment there is a patch [http://download.openvz.org/kernel/devel/026test018.1/patches/patch-026test018-combined.gz patch-026test018-combined.gz] available.

Kernel configs are also available at OpenVZ download site. Most frequently SMP config is used, so let's download [http://download.openvz.org/kernel/devel/026test018.1/configs/kernel-2.6.16-026test018-i686-smp.config.ovz kernel-2.6.16-026test018-i686-smp.config.ovz]
for this example.

=== Building ===
First, extract the kernel sources from archive:
<pre>
# tar vjxf linux-2.6.16.tar.bz2
# cd linux-2.6.16
</pre>

Apply OpenVZ patches to the kernel:
<pre>
# gzip -dc patch-026test018-combined.gz | patch -p1
</pre>

Now we need to place the config and build the kernel:
<pre>
# cp kernel-2.6.16-026test018-i686-smp.config.ovz .config
# make oldconfig
# make
# make modules
</pre>

=== Installation ===
After successfull build of kernel it can be installed on the machine with the following commands run under '''root''' user:
<pre>
# make install
# make modules_install
</pre>

Also you need to edit your GRUB or LILO config to make your kernel available for boot.

[[Category:HOWTO]]
[[Category:Kernel]]

Kernel build

2007-02-19T11:27:37Z

Wfischer: added info on modifying kernel configs when using source-RPM

This FAQ will help you in case you want to apply some patches to the kernel on your own or rebuild it from sources.
On RPM based distros such as RedHat Enterprise Linux/CentOS, Fedora Core or SUSE one can simpy rebuild kernel from SRPM.
For other distros it is required to install sources, build and install kernel manually. The below are given the details for both cases.

== Rebuilding kernel from SRPM ==

'''Note''': some of the paths below include $TOPDIR, which is distribution-dependent and can be further redefined by user. To find out the proper location on your system, issue this command:

<pre>rpm --eval "%{_topdir}"</pre>

=== Download ===
Source RPMS for different OpenVZ kernel branches can be downloaded from http://openvz.org/download/kernel/. You can also access http://download.openvz.org/kernel/ directly, or use one of the [[download mirrors|mirrors]].

=== Installation ===
Install the downloaded SRC RPM with the following command:
<pre>
# rpm -ihv ovzkernel-2.6.16-026test018.1.src.rpm
</pre>

After successfull installation, you can usually find kernel sources in <code>$TOPDIR/SOURCES/</code>
and kernel spec file (<code>kernel-ovz.spec</code>) in <code>$TOPDIR/SPECS</code>.

=== Adding your own patches ===
To modify the kernel one needs just to add specific patches to the kernel spec file and put this patch into <code>$TOPDIR/SOURCES</code> directory.

Put your patch into SOURCES directory with the following command:
<pre>
# cp <patch> $TOPDIR/SOURCES/
</pre>

Then open spec file <code>$TOPDIR/SPECS/kernel-ovz.spec</code> in the editor and add the following lines:
<pre>
Patch10000: <patch-name>
</pre>
and
<pre>
%patch10000 -p1
</pre>
in appropriate places where similar text lines are.

=== Modifying configs ===
If you want to modify kernel configs, you need to do the changes via the kernel spec file. Insert your modifications after the following part of the spec file:
<pre>
# now run oldconfig over all the config files
for i in *.config*
do
mv $i .config
</pre>
You can add kernel settings for example in the following way:
<pre>
echo "YOUR_KERNEL_OPTION=m" >> .config
</pre>
If you try to modify the config files directly (not through the spec file), running rpmbuild will fail with the following error:
<pre>
make[1]: *** [nonint_oldconfig] Error 1
make: *** [nonint_oldconfig] Error 2
</pre>
Some information on this can be found in the following thread: http://www.redhat.com/archives/fedora-list/2005-October/msg00921.html

=== Building RPMs ===
Before rebuilding the kernel make sure that you adjusted the kernel version in <code>kernel-ovz.spec</code>.
This will help you to distinguish binaries then from already existing kernels
(or from the official OpenVZ kernels). To do so, edit the <code>$TOPDIR/SPECS/kernel-ovz.spec</code> file and replace the following line:
<pre>
%define ksubrelease 1
</pre>
with something like
<pre>
%define ksubrelease 1-my.kernel.v1
</pre>

To rebuild the kernel, type the following commands:
<pre>
# cd $TOPDIR/SPECS
# rpmbuild -ba --target=i686 kernel-ovz.spec
</pre>

After successfull kernel compilation binary RPMs can be found at <code>$TOPDIR/RPMS/i686</code>.

== Rebuilding kernel from sources ==

=== Download ===
To compile OpenVZ linux kernel one need to download the original linux kernel sources and OpenVZ patches for it.

Linux kernel can be found at [http://www.kernel.org/ kernel.org], e.g. 2.6.16 kernel can be downloaded from [http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.tar.bz2 linux-2.6.16.tar.bz2].

Appropriate OpenVZ patches for this kernel version can be found at http://openvz.org/download/kernel/<version>/patches/. For example, at the moment there is a patch [http://download.openvz.org/kernel/devel/026test018.1/patches/patch-026test018-combined.gz patch-026test018-combined.gz] available.

Kernel configs are also available at OpenVZ download site. Most frequently SMP config is used, so let's download [http://download.openvz.org/kernel/devel/026test018.1/configs/kernel-2.6.16-026test018-i686-smp.config.ovz kernel-2.6.16-026test018-i686-smp.config.ovz]
for this example.

=== Building ===
First, extract the kernel sources from archive:
<pre>
# tar vjxf linux-2.6.16.tar.bz2
# cd linux-2.6.16
</pre>

Apply OpenVZ patches to the kernel:
<pre>
# gzip -dc patch-026test018-combined.gz | patch -p1
</pre>

Now we need to place the config and build the kernel:
<pre>
# cp kernel-2.6.16-026test018-i686-smp.config.ovz .config
# make oldconfig
# make
# make modules
</pre>

=== Installation ===
After successfull build of kernel it can be installed on the machine with the following commands run under '''root''' user:
<pre>
# make install
# make modules_install
</pre>

Also you need to edit your GRUB or LILO config to make your kernel available for boot.

[[Category:HOWTO]]
[[Category:Kernel]]

HA cluster with DRBD and Heartbeat

2006-12-13T09:59:29Z

Wfischer: added a hint about gratuitous arp /* Setting up Heartbeat */

This article shows how to setup a OpenVZ high availability (HA) cluster using the data replication software DRBD and the cluster manager Heartbeat. In this example the two machines builing the cluster run on CentOS 4.3. The article also shows how to do kernel updates in the cluster, including necessary steps like recompiling of new DRBD userspace tools. For this purpose, kernel 2.6.8-022stab078.10 (containing DRBD module 0.7.17) is used as initial kernel version, and kernel 2.6.8-022stab078.14 (containing DRBD module 0.7.20) as updated kernel version.

Additional information about clustering of virtual machines can be found in the following paper: http://www.linuxtag.org/2006/fileadmin/linuxtag/dvd/12080-paper.pdf

Some other additional information can be found in the documentation of the Thomas-Krenn.AG cluster (The author of this howto is working in the cluster development there, that is the reason why he was able to write this howto :-). The full documentation with interesting illustrations is currently only available in German:
http://my.thomas-krenn.com/service_support/index.php/page.242

== Prerequisites ==
The OpenVZ kernel already includes the DRBD module. The DRBD userspace tools and the cluster manager Heartbeat must be provided seperately. As the API version of the DRBD userspace tools must exactly match the API version of the module, compile them yourself. Also compile Heartbeat yourself, as at the time of this writing the CentOS extras repository only contained an old CVS version of Heartbeat.

On a hardware node for production use there should not be any application that is not really needed for running OpenVZ (any things which are not needed by OpenVZ should run in a VE for security reasons). As a result, compile DRBD and Heartbeat on another machine running CentOS 4.3 (in this example I used a virtual machine on a VMware Server).

=== Compiling Heartbeat ===
Heartbeat version 1.2.* has successfully been used in a lot of two-node-clusters around the world. As the codebase used in version 1.2.* is in production use for many years now, the code is very stable. At the time of writing, Heartbeat version 1.2.4 is the current version of the 1.2.* branch.

Get the tar.gz of the current version of the 1.2.* branch from http://linux-ha.org/download/index.html, at the time of this writing this is http://linux-ha.org/download/heartbeat-1.2.4.tar.gz. Use rpmbuild to build the package:
<pre>rpmbuild -ta heartbeat-1.2.4.tar.gz</pre>
After that, you find four rpm packes in /usr/src/redhat/RPMS/i386 (heartbeat-1.2.4-1.i386.rpm, heartbeat-ldirectord-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, heartbeat-stonith-1.2.4-1.i386.rpm). In this example only heartbeat-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, and heartbeat-stonith-1.2.4-1.i386.rpm are needed.

=== Compiling DRBD userspace tools ===
When compiling the DRBD userspace tools, you have to take care to take the version that matches the DRBD version that is included in the OpenVZ kernel you want to use. If you are unsure about the version, do the following steps while running the OpenVZ kernel that you want to use on a test machine (I used another virtual machine on a VMware server to try this):
<pre>
[root@testmachine ~]# cat /proc/version
Linux version 2.6.8-022stab078.10 (root@rhel4-32) (gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)) #1 Wed Jun 21 12:01:20 MSD 2006
[root@testmachine ~]# modprobe drbd
[root@testmachine ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Unconfigured
1: cs:Unconfigured
[root@testmachine ~]# rmmod drbd
[root@testmachine ~]#
</pre>
Here the version of the DRBD module is 0.7.17. So the userspace tools for 0.7.17 are neccessary.

Back on the buildmachine, do the following to create the rpm:
<pre>
[root@buildmachine ~]# yum install kernel-devel gcc bison flex
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Nothing to do
[root@buildmachine ~]# tar xfz drbd-0.7.17.tar.gz
[root@buildmachine ~]# cd drbd-0.7.17
[root@buildmachine drbd-0.7.17]# make rpm
[...]
You have now:
-rw-r--r-- 1 root root 288728 Jul 30 10:40 dist/RPMS/i386/drbd-0.7.17-1.i386.rpm
-rw-r--r-- 1 root root 518369 Jul 30 10:40 dist/RPMS/i386/drbd-km-2.6.9_34.0.2.EL-0.7.17-1.i386.rpm
[root@buildmachine drbd-0.7.17]#
</pre>
Note that in this way the kernel-devel from CentOS is used, but this does not matter as the created drbd-km rpm will not be used (the DRBD kernel module is already included in OpenVZ kernel). If the kernel-devel package is not the same version as the kernel package that is currently running, it is possible to execute 'make rpm KDIR=/usr/src/kernels/2.6.9-34.0.2.EL-i686/' to directly point to the kernel sources.

== Installing the two nodes ==
Install the two machines in the same way as you would install them for a normal OpenVZ installation, but do not create a filesystem for the /vz. This filesystem will be installed later on on top of DRBD.
{| class="wikitable"
|+'''Example installation configuration'''
! Parameter !! node1 !! node2
|-
! hostname
| ovz-node1
| ovz-node2
|-
! / filesystem
| hda1, 10 GB
| hda1, 10 GB
|-
! swap space
| hda2, 2048 MB
| hda2, 2048 MB
|-
! public LAN
| eth0, 192.168.1.201
| eth0, 192.168.1.202
|-
! private LAN
| eth1, 192.168.255.1 (Gbit Ethernet)
| eth1, 192.168.255.2 (Gbit Ethernet)
|-
! other install options
| no firewall, no SELinux
| no firewall, no SELinux
|-
! package groups
| deactivated everything, only kept vim-enhanced
| deactivated everything, only kept vim-enhanced
|}

== Installing OpenVZ ==

Get the OpenVZ kernel and utilities and install them on both nodes, as described in [[quick installation]]. Update grub configuration to use the OpenVZ kernel by default. Disable starting of OpenVZ on system boot on both nodes (OpenVZ will be started and stopped by Heartbeat):
<pre>
[root@ovz-node1 ~]# chkconfig --del vz
[root@ovz-node1 ~]#
</pre>
Then reboot both machines.

== Setting up DRBD ==

On each of the two nodes create a partition that acts as underlying DRBD device. The partitions should have exactly the same size (I created a 10 GB partition hda3 using fdisk on each node for this example). Note that it might be necessary to reboot the machines to re-read the partition table.

Install the rpm of the DRBD userspace tools on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv drbd-0.7.17-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Then create the drbd.conf configuration file and copy it to /etc/drbd.conf on both nodes. Below is the example configuration file that is used in this article:
<pre>
resource r0 {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";

startup {
degr-wfc-timeout 120;
}

net {
on-disconnect reconnect;
}

disk {
on-io-error detach;
}

syncer {
rate 30M;
group 1;
al-extents 257;
}

on ovz-node1 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.1:7788;
meta-disk internal;
}

on ovz-node2 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.2:7788;
meta-disk internal;
}

}
</pre>
Start DRBD on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/drbd start
Starting DRBD resources: [ d0 s0 n0 ].
[root@ovz-node1 ~]#
</pre>
Then check the status of /proc/drbd:
<pre>
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Secondary ld:Inconsistent
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
[root@ovz-node1 ~]#
</pre>
Both nodes are now Secondary and Inconsistent. The latter is because the underlying storage is not yet in-sync, and DRBD has no way to know whether you want the initial sync from ovz-node1 to ovz-node2, or ovz-node2 to ovz-node1. As there is no data below it yet, it does not matter.

To start the sync from ovz-node1 to ovz-node2, do the following on ovz-node1:
<pre>
[root@ovz-node1 ~]# drbdadm -- --do-what-I-say primary all
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:SyncSource st:Primary/Secondary ld:Consistent
ns:627252 nr:0 dw:0 dr:629812 al:0 bm:38 lo:640 pe:0 ua:640 ap:0
[=>..................] sync'ed: 6.6% (8805/9418)M
finish: 0:04:51 speed: 30,888 (27,268) K/sec
[root@ovz-node1 ~]#
</pre>
As you see, DRBD syncs with about 30 MB per second, as we told it so in /etc/drbd.conf. On the SyncSource (ovz-node1 in this case) the DRBD device is already useable (although it is syncing in the background).

So you can immediately create the filesystem:
<pre>
[root@ovz-node1 ~]# mkfs.ext3 /dev/drbd0
[...]
[root@ovz-node1 ~]#
</pre>

=== Copy necessary OpenVZ files to DRBD device ===

Move the original /vz directory to /vz.orig and recreate the /vz directory to have it as a mount point '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /vz /vz.orig
[root@ovz-node1 ~]# mkdir /vz
[root@ovz-node1 ~]#
</pre>

Afterwards move the necessary OpenVZ directories (/etc/vz, /etc/sysconfig/vz-scripts, /var/vzquota) and replace them with symbolic links '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /etc/vz /etc/vz.orig
[root@ovz-node1 ~]# mv /etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts.orig
[root@ovz-node1 ~]# mv /var/vzquota /var/vzquota.orig
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/vz /etc/vz
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts
[root@ovz-node1 ~]# ln -s /vz/cluster/var/vzquota /var/vzquota
[root@ovz-node1 ~]#
</pre>

Currently, ovz-node1 is still Primary of /dev/drbd0. You can now mount it and copy the necessary files to it '''(only on ovz-node1!)''':
<pre>
[root@ovz-node1 ~]# mount /dev/drbd0 /vz
[root@ovz-node1 ~]# cp -a /vz.orig/* /vz/
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc/sysconfig
[root@ovz-node1 ~]# mkdir -p /vz/cluster/var
[root@ovz-node1 ~]# cp -a /etc/vz /vz/cluster/etc/
[root@ovz-node1 ~]# cp -a /etc/sysconfig/vz-scripts /vz/cluster/etc/sysconfig/
[root@ovz-node1 ~]# cp -a /var/vzquota /vz/cluster/var/
[root@ovz-node1 ~]# umount /dev/drbd0
[root@ovz-node1 ~]#
</pre>

== Setting up Heartbeat ==

Install the neccessary Heartbeat rpms on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv heartbeat-1.2.4-1.i386.rpm heartbeat-pils-1.2.4-1.i386.rpm heartbeat-stonith-1.2.4-1.i386.rpm
Preparing... ########################################### [100%]
1:heartbeat-pils ########################################### [ 33%]
2:heartbeat-stonith ########################################### [ 67%]
3:heartbeat ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Create the Heartbeat configuration file ha.cf and copy it to <code>/etc/ha.d/ha.cf</code> on both nodes. Details about this file can be found at http://www.linux-ha.org/ha.cf. Below is an example configuration which uses the two network connections and also a serial connection for heartbeat packets:
<pre>
# Heartbeat logging configuration
logfacility daemon

# Heartbeat cluster members
node ovz-node1
node ovz-node2

# Heartbeat communication timing
keepalive 1
warntime 10
deadtime 30
initdead 120

# Heartbeat communication paths
udpport 694
ucast eth1 192.168.255.1
ucast eth1 192.168.255.2
ucast eth0 192.168.1.201
ucast eth0 192.168.1.202
baud 19200
serial /dev/ttyS0

# Don't fail back automatically
auto_failback off

# Monitoring of network connection to default gateway
ping 192.168.1.1
respawn hacluster /usr/lib64/heartbeat/ipfail
</pre>
Create the Heartbeat configuration file authkeys and copy it to <code>/etc/ha.d/authkeys</code> on both nodes. Set the permissions of this file to 600. Details about this file can be found at http://www.linux-ha.org/authkeys. Below is an example:
<pre>
auth 1
1 sha1 PutYourSuperSecretKeyHere
</pre>
Create the Heartbeat configuration file haresources and copy it to <code>/etc/ha.d/haresources</code> on both nodes. Details about this file can be found at http://www.linux-ha.org/haresources. Note that it is not necessary to configure IPs for gratuitous arp here. The gratuitous arp is done by OpenVZ itself, through <code>/etc/sysconfig/network-scripts/ifup-venet</code> and <code>/usr/lib/vzctl/scripts/vps-functions</code>. Below is an example for the haresources file:
<pre>
ovz-node1 drbddisk::r0 Filesystem::/dev/drbd0::/vz::ext3 vz MailTo::youremail@yourdomain.tld
</pre>
Finally, you can now start heartbeat on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services:
[ OK ]
[root@ovz-node1 ~]#
</pre>

== Before going in production: testing, testing, testing, and ...hm... testing! ==

The installation of the cluster is finished at this point. Before putting the cluster in production it is very important to test the cluster. Because of all the possible different kinds of hardware that you may have, you may encounter problems when a failover is necessary. And as the cluster is about high availability, such problems must be found before the cluster is used for production.

Here is one example: The e1000 driver that is included in kernels < 2.6.12 has a problem when a cable gets unplugged while broadcast packets are still being sent out on that interface. When using broadcast communication in Heartbeat on a crossover link, this fills up the transmit ring buffer on the adapter (the buffer is full after about 8 minutes after the cable got unplugged). Using unicast communication in Heartbeat fixes the problem for example. Details see: http://www.osdl.org/developer_bugzilla/show_bug.cgi?id=699#c22

Without testing you may not be aware of such problems and may face them when the cluster is in production and a failover would be necessary. So test your cluster carefully!

Possible tests can include:
* power outage test of active node
* power outage test of passive node
* network connection outage test of eth0 of active node
* network connection outage test of eth0 of passive node
* network connection outage test of crossover network connection
* ...

As mentioned above, some problems only arise after an outage lasts longer than some minutes. So do the tests also with a duration of >1h for example.

Before you start to test, build a test plan. Some valueable information on that can be found in chapter 3 "Testing a highly available Tivoli Storage Manager cluster environment" of the Redbook ''IBM Tivoli Storage Manager in a Clustered Environment'', see http://www.redbooks.ibm.com/abstracts/sg246679.html. In this chapter it is mentioned that the experience of the authoring team is that the testing phase must be at least two times the total implementation time for the cluster.

== Before installing kernel updates: testing again ==

New OpenVZ kernel often include driver updates. This kernel for examples includes an update of the e1000 module: http://openvz.org/news/updates/kernel-022stab078.21

To avoid to overlook problems with new components (such as a newer kernel), it is necessary to re-do the tests mentioned above. But as the cluster is already in production, a second cluster (test cluster) with the same hardware as the main cluster is needed. Use this test cluster to test updates of the kernel or main OS updates for the hardware node before putting them on the production cluster.

I know this is not an easy task, as it is time-consuming and needs additional hardware only for testing. But when really business-critical applications are running on the cluster, it is very good to now that the cluster works fine also with new updates installed on the hardware node. In many cases a dedicated test cluster and the time efford for the testing of updates may cause too much costs. If you cannot do such test of updates, keep in mind that over time (when you must install security updates of the OS or the kernel) you have a cluster that you have not tested in this configuration.

If you need a tested cluster (also with tested kernel updates), you may take a look on this Virtuozzo cluster: http://www.thomas-krenn.com/cluster

== How to do OpenVZ kernel updates when it contains a new DRBD version ==

As mentioned above, it is important to use the correct version of the DRBD userspace tools. When an OpenVZ kernel contains a new DRBD version, it is important that the DRBD API version of the userspace tools matches the API version of the DRBD module that is included in the OpenVZ kernel. The API versions can be found at http://svn.drbd.org/drbd/branches/drbd-0.7/ChangeLog. The best way is to always use the version of the DRBD userspace tools that matches the version of the DRBD module that is included in the OpenVZ kernel.

In this example the initial cluster installation contained OpenVZ kernel 2.6.8-022stab078.10, which contains the DRBD module 0.7.17. The steps below show the update procedure to OpenVZ kernel 2.6.8-022stab078.14, which contains the DRBD module 0.7.20.
In the first step build the DRBD userspace tools version 0.7.20 on your buildmachine. Then stop Heartbeat and DRBD on the passive node (hint: you can use 'cat /proc/drbd' to get a hint which node is active and which one is passive):
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services:
[ OK ]
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/drbd stop
Stopping all DRBD resources.
[root@ovz-node2 ~]# cat /proc/drbd
cat: /proc/drbd: No such file or directory
[root@ovz-node2 ~]#
</pre>
Then install the new kernel and the DRBD userspace tools on this node:
<pre>
[root@ovz-node2 ~]# rpm -ihv ovzkernel-2.6.8-022stab078.14.i686.rpm
warning: ovzkernel-2.6.8-022stab078.14.i686.rpm: V3 DSA signature: NOKEY, key ID a7a1d4b6
Preparing... ########################################### [100%]
1:ovzkernel ########################################### [100%]
[root@ovz-node2 ~]# rpm -Uhv drbd-0.7.20-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
/sbin/service
Stopping all DRBD resources.
[root@ovz-node2 ~]#
</pre>
Now set the new kernel as default kernel in /etc/grub.conf and then reboot this node.

After the reboot, the new DRBD version is visible:
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.20 (api:79/proto:74)
SVN Revision: 2260 build by phil@mescal, 2006-07-04 15:18:57
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:0 nr:28 dw:28 dr:0 al:0 bm:2 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]#
</pre>

To update the other node, switch-over the services to make the current active node the passive node. Execute the following on the still active node (it could be that the hb_standby command is located in /usr/lib/heartbeat):
<pre>
[root@ovz-node1 ~]# /usr/lib64/heartbeat/hb_standby
2006/08/03_21:09:41 Going standby [all].
[root@ovz-node1 ~]#
</pre>

Now do the same steps on the new passive node to update it: stop Heartbeat and DRBD, install the new kernel and the new DRBD userspace tools, set the new kernel as default kernel in /etc/grub.conf and reboot the node.

== How to do updates of vzctl, vzctl-lib, and vzquota ==

Ensure after every update of OpenVZ tools that OpenVZ is not started on system boot. To disable starting of OpenVZ on system boot execute on both nodes:
<pre>
[root@ovz-node1 ~]# chkconfig --del vz
[root@ovz-node1 ~]#
</pre>

[[Category: HOWTO]]

FAQ

2006-12-13T09:25:13Z

Wfischer: updated link /* What hardware is supported by OpenVZ kernel? */

== General ==
===== What is a Virtual Environment (Virtual Private Server, VPS, VE)? =====
:See [[VE]]

===== Who needs OpenVZ? How it can be used? =====
:See [[Use cases]]

===== How is OpenVZ different from other technologies? =====
:See [[Introduction to virtualization]]

===== How is OpenVZ updated and why it is secure? =====
:See [[Security]]

===== I want to show my appreciation to OpenVZ and put some logo to my site. Where to get it? =====
:See [[Artwork]]

== Installation and upgrade ==

===== What hardware is supported by OpenVZ kernel? =====
:See [http://www.swsoft.com/en/products/virtuozzo/hcl/ Virtuozzo HCL].

===== Why there are different kernel flavours available and what do they mean? =====
:See [[Different kernel flavors (UP, SMP, ENTERPRISE, ENTNOSPLIT)]]

===== How do I rebuild the kernel? =====
:See [[Kernel build]]

===== What does 021stab018 in OpenVZ kernel version mean? =====
:See [[Kernel versioning]]

===== How can I check package signatures? =====
:See [[Package signatures]]

===== Is it possible to run x86 VPS on a x86_64 arch? =====
:Sure :) We actually did some work on that to enable migration of x86 VE from x86 to x86_64 and back, and to enable using 32-bit iptables in 32bit VE on an x86_64 system.

===== What filesystems should I choose for saving my VEs? =====
The safest choice is ext2/3 to save your VEs. Remember that ReiserFS is less stable than ext2/3.
If you choose to use XFS, think about, that there is no support for disk quota inside VEs.

== Networking ==

===== How do I set up VPN for a VE? =====
:See [[VPN via the TUN/TAP device]]

===== What is veth and how do I use it? =====
:See [[Virtual Ethernet device]]

== User Beancounters ==

===== What are those User Beancounters? =====
See [[UBC]].

===== What units are UBC parameters measured in? =====
See [[UBC parameter units]]

===== How do I set up a VE which is able to get X Mb of RAM? ======
See [[Setting UBC parameters]].

===== I can not start a program in VE: it reports out of memory. What do I do? =====
See [[Resource_shortage]].

===== How can I reset <code>failcnt</code> in <code>/proc/user_beancounters</code>? =====
See [[UBC failcnt reset]].

== Troubleshooting ==
===== My kernel crashed. What should I do? =====
:See [[When you have an oops]]

===== I see a lot of processes in D state. What does that mean? =====
:See [[Processes in D state]]

HA cluster with DRBD and Heartbeat

2006-10-11T15:34:26Z

Wfischer: corrected a broken footnote /* Bofore going in production: testing, testing, testing, and ...hm... testing! */

This article shows how to setup a OpenVZ high availability (HA) cluster using the data replication software DRBD and the cluster manager Heartbeat. In this example the two machines builing the cluster run on CentOS 4.3. The article also shows how to do kernel updates in the cluster, including necessary steps like recompiling of new DRBD userspace tools. For this purpose, kernel 2.6.8-022stab078.10 (containing DRBD module 0.7.17) is used as initial kernel version, and kernel 2.6.8-022stab078.14 (containing DRBD module 0.7.20) as updated kernel version.

Additional information about clustering of virtual machines can be found in the following paper: http://www.linuxtag.org/2006/fileadmin/linuxtag/dvd/12080-paper.pdf

Some other additional information can be found in the documentation of the Thomas-Krenn.AG cluster (The author of this howto is working in the cluster development there, that is the reason why he was able to write this howto :-). The full documentation with interesting illustrations is currently only available in German:
http://my.thomas-krenn.com/service_support/index.php/page.242

== Prerequisites ==
The OpenVZ kernel already includes the DRBD module. The DRBD userspace tools and the cluster manager Heartbeat must be provided seperately. As the API version of the DRBD userspace tools must exactly match the API version of the module, compile them yourself. Also compile Heartbeat yourself, as at the time of this writing the CentOS extras repository only contained an old CVS version of Heartbeat.

On a hardware node for production use there should not be any application that is not really needed for running OpenVZ (any things which are not needed by OpenVZ should run in a VE for security reasons). As a result, compile DRBD and Heartbeat on another machine running CentOS 4.3 (in this example I used a virtual machine on a VMware Server).

=== Compiling Heartbeat ===
Heartbeat version 1.2.* has successfully been used in a lot of two-node-clusters around the world. As the codebase used in version 1.2.* is in production use for many years now, the code is very stable. At the time of writing, Heartbeat version 1.2.4 is the current version of the 1.2.* branch.

Get the tar.gz of the current version of the 1.2.* branch from http://linux-ha.org/download/index.html, at the time of this writing this is http://linux-ha.org/download/heartbeat-1.2.4.tar.gz. Use rpmbuild to build the package:
<pre>rpmbuild -ta heartbeat-1.2.4.tar.gz</pre>
After that, you find four rpm packes in /usr/src/redhat/RPMS/i386 (heartbeat-1.2.4-1.i386.rpm, heartbeat-ldirectord-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, heartbeat-stonith-1.2.4-1.i386.rpm). In this example only heartbeat-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, and heartbeat-stonith-1.2.4-1.i386.rpm are needed.

=== Compiling DRBD userspace tools ===
When compiling the DRBD userspace tools, you have to take care to take the version that matches the DRBD version that is included in the OpenVZ kernel you want to use. If you are unsure about the version, do the following steps while running the OpenVZ kernel that you want to use on a test machine (I used another virtual machine on a VMware server to try this):
<pre>
[root@testmachine ~]# cat /proc/version
Linux version 2.6.8-022stab078.10 (root@rhel4-32) (gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)) #1 Wed Jun 21 12:01:20 MSD 2006
[root@testmachine ~]# modprobe drbd
[root@testmachine ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Unconfigured
1: cs:Unconfigured
[root@testmachine ~]# rmmod drbd
[root@testmachine ~]#
</pre>
Here the version of the DRBD module is 0.7.17. So the userspace tools for 0.7.17 are neccessary.

Back on the buildmachine, do the following to create the rpm:
<pre>
[root@buildmachine ~]# yum install kernel-devel gcc bison flex
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Nothing to do
[root@buildmachine ~]# tar xfz drbd-0.7.17.tar.gz
[root@buildmachine ~]# cd drbd-0.7.17
[root@buildmachine drbd-0.7.17]# make rpm
[...]
You have now:
-rw-r--r-- 1 root root 288728 Jul 30 10:40 dist/RPMS/i386/drbd-0.7.17-1.i386.rpm
-rw-r--r-- 1 root root 518369 Jul 30 10:40 dist/RPMS/i386/drbd-km-2.6.9_34.0.2.EL-0.7.17-1.i386.rpm
[root@buildmachine drbd-0.7.17]#
</pre>
Note that in this way the kernel-devel from CentOS is used, but this does not matter as the created drbd-km rpm will not be used (the DRBD kernel module is already included in OpenVZ kernel). If the kernel-devel package is not the same version as the kernel package that is currently running, it is possible to execute 'make rpm KDIR=/usr/src/kernels/2.6.9-34.0.2.EL-i686/' to directly point to the kernel sources.

== Installing the two nodes ==
Install the two machines in the same way as you would install them for a normal OpenVZ installation, but do not create a filesystem for the /vz. This filesystem will be installed later on on top of DRBD.
{| class="wikitable"
|+'''Example installation configuration'''
! Parameter !! node1 !! node2
|-
! hostname
| ovz-node1
| ovz-node2
|-
! / filesystem
| hda1, 10 GB
| hda1, 10 GB
|-
! swap space
| hda2, 2048 MB
| hda2, 2048 MB
|-
! public LAN
| eth0, 192.168.1.201
| eth0, 192.168.1.202
|-
! private LAN
| eth1, 192.168.255.1 (Gbit Ethernet)
| eth1, 192.168.255.2 (Gbit Ethernet)
|-
! other install options
| no firewall, no SELinux
| no firewall, no SELinux
|-
! package groups
| deactivated everything, only kept vim-enhanced
| deactivated everything, only kept vim-enhanced
|}

== Installing OpenVZ ==

Get the OpenVZ kernel and utilities and install them on both nodes, as described in [[quick installation]]. Update grub configuration to use the OpenVZ kernel by default. Disable starting of OpenVZ on system boot on both nodes (OpenVZ will be started and stopped by Heartbeat):
<pre>
[root@ovz-node1 ~]# chkconfig --del vz
[root@ovz-node1 ~]#
</pre>
Then reboot both machines.

== Setting up DRBD ==

On each of the two nodes create a partition that acts as underlying DRBD device. The partitions should have exactly the same size (I created a 10 GB partition hda3 using fdisk on each node for this example). Note that it might be necessary to reboot the machines to re-read the partition table.

Install the rpm of the DRBD userspace tools on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv drbd-0.7.17-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Then create the drbd.conf configuration file and copy it to /etc/drbd.conf on both nodes. Below is the example configuration file that is used in this article:
<pre>
resource r0 {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";

startup {
degr-wfc-timeout 120;
}

net {
on-disconnect reconnect;
}

disk {
on-io-error detach;
}

syncer {
rate 30M;
group 1;
al-extents 257;
}

on ovz-node1 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.1:7788;
meta-disk internal;
}

on ovz-node2 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.2:7788;
meta-disk internal;
}

}
</pre>
Start DRBD on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/drbd start
Starting DRBD resources: [ d0 s0 n0 ].
[root@ovz-node1 ~]#
</pre>
Then check the status of /proc/drbd:
<pre>
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Secondary ld:Inconsistent
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
[root@ovz-node1 ~]#
</pre>
Both nodes are now Secondary and Inconsistent. The latter is because the underlying storage is not yet in-sync, and DRBD has no way to know whether you want the initial sync from ovz-node1 to ovz-node2, or ovz-node2 to ovz-node1. As there is no data below it yet, it does not matter.

To start the sync from ovz-node1 to ovz-node2, do the following on ovz-node1:
<pre>
[root@ovz-node1 ~]# drbdadm -- --do-what-I-say primary all
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:SyncSource st:Primary/Secondary ld:Consistent
ns:627252 nr:0 dw:0 dr:629812 al:0 bm:38 lo:640 pe:0 ua:640 ap:0
[=>..................] sync'ed: 6.6% (8805/9418)M
finish: 0:04:51 speed: 30,888 (27,268) K/sec
[root@ovz-node1 ~]#
</pre>
As you see, DRBD syncs with about 30 MB per second, as we told it so in /etc/drbd.conf. On the SyncSource (ovz-node1 in this case) the DRBD device is already useable (although it is syncing in the background).

So you can immediately create the filesystem:
<pre>
[root@ovz-node1 ~]# mkfs.ext3 /dev/drbd0
[...]
[root@ovz-node1 ~]#
</pre>

=== Copy necessary OpenVZ files to DRBD device ===

Move the original /vz directory to /vz.orig and recreate the /vz directory to have it as a mount point '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /vz /vz.orig
[root@ovz-node1 ~]# mkdir /vz
[root@ovz-node1 ~]#
</pre>

Afterwards move the necessary OpenVZ directories (/etc/vz, /etc/sysconfig/vz-scripts, /var/vzquota) and replace them with symbolic links '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /etc/vz /etc/vz.orig
[root@ovz-node1 ~]# mv /etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts.orig
[root@ovz-node1 ~]# mv /var/vzquota /var/vzquota.orig
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/vz /etc/vz
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts
[root@ovz-node1 ~]# ln -s /vz/cluster/var/vzquota /var/vzquota
[root@ovz-node1 ~]#
</pre>

Currently, ovz-node1 is still Primary of /dev/drbd0. You can now mount it and copy the necessary files to it '''(only on ovz-node1!)''':
<pre>
[root@ovz-node1 ~]# mount /dev/drbd0 /vz
[root@ovz-node1 ~]# cp -a /vz.orig/* /vz/
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc/sysconfig
[root@ovz-node1 ~]# mkdir -p /vz/cluster/var
[root@ovz-node1 ~]# cp -a /etc/vz /vz/cluster/etc/
[root@ovz-node1 ~]# cp -a /etc/sysconfig/vz-scripts /vz/cluster/etc/sysconfig/
[root@ovz-node1 ~]# cp -a /var/vzquota /vz/cluster/var/
[root@ovz-node1 ~]# umount /dev/drbd0
[root@ovz-node1 ~]#
</pre>

== Setting up Heartbeat ==

Install the neccessary Heartbeat rpms on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv heartbeat-1.2.4-1.i386.rpm heartbeat-pils-1.2.4-1.i386.rpm heartbeat-stonith-1.2.4-1.i386.rpm
Preparing... ########################################### [100%]
1:heartbeat-pils ########################################### [ 33%]
2:heartbeat-stonith ########################################### [ 67%]
3:heartbeat ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Create the Heartbeat configuration file ha.cf and copy it to /etc/ha.d/ha.cf on both nodes. Details about this file can be found at http://www.linux-ha.org/ha.cf. Below is an example configuration which uses the two network connections and also a serial connection for heartbeat packets:
<pre>
# Heartbeat logging configuration
logfacility daemon

# Heartbeat cluster members
node ovz-node1
node ovz-node2

# Heartbeat communication timing
keepalive 1
warntime 10
deadtime 30
initdead 120

# Heartbeat communication paths
udpport 694
ucast eth1 192.168.255.1
ucast eth1 192.168.255.2
ucast eth0 192.168.1.201
ucast eth0 192.168.1.202
baud 19200
serial /dev/ttyS0

# Don't fail back automatically
auto_failback off

# Monitoring of network connection to default gateway
ping 192.168.1.1
respawn hacluster /usr/lib64/heartbeat/ipfail
</pre>
Create the Heartbeat configuration file authkeys and copy it to /etc/ha.d/authkeys on both nodes. Set the permissions of this file to 600. Details about this file can be found at http://www.linux-ha.org/authkeys. Below is an example:
<pre>
auth 1
1 sha1 PutYourSuperSecretKeyHere
</pre>
Create the Heartbeat configuration file haresources and copy it to /etc/ha.d/haresources on both nodes. Details about this file can be found at http://www.linux-ha.org/haresources. Below is an example:
<pre>
ovz-node1 drbddisk::r0 Filesystem::/dev/drbd0::/vz::ext3 vz MailTo::youremail@yourdomain.tld
</pre>
Finally, you can now start heartbeat on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services:
[ OK ]
[root@ovz-node1 ~]#
</pre>

== Bofore going in production: testing, testing, testing, and ...hm... testing! ==

The installation of the cluster is finished at this point. Before putting the cluster in production it is very important to test the cluster. Because of all the possible different kinds of hardware that you may have, you may encounter problems when a failover is necessary. And as the cluster is about high availability, such problems must be found before the cluster is used for production.

Here is one example: The e1000 driver that is included in kernels < 2.6.12 has a problem when a cable gets unplugged while broadcast packets are still being sent out on that interface. When using broadcast communication in Heartbeat on a crossover link, this fills up the transmit ring buffer on the adapter (the buffer is full after about 8 minutes after the cable got unplugged). Using unicast communication in Heartbeat fixes the problem for example. Details see: http://www.osdl.org/developer_bugzilla/show_bug.cgi?id=699#c22

Without testing you may not be aware of such problems and may face them when the cluster is in production and a failover would be necessary. So test your cluster carefully!

Possible tests can include:
* power outage test of active node
* power outage test of passive node
* network connection outage test of eth0 of active node
* network connection outage test of eth0 of passive node
* network connection outage test of crossover network connection
* ...

As mentioned above, some problems only arise after an outage lasts longer than some minutes. So do the tests also with a duration of >1h for example.

Before you start to test, build a test plan. Some valueable information on that can be found in chapter 3 "Testing a highly available Tivoli Storage Manager cluster environment" of the Redbook ''IBM Tivoli Storage Manager in a Clustered Environment'', see http://www.redbooks.ibm.com/abstracts/sg246679.html. In this chapter it is mentioned that the experience of the authoring team is that the testing phase must be at least two times the total implementation time for the cluster.

== Bofore installing kernel updates: testing again ==

New OpenVZ kernel often include driver updates. This kernel for examples includes an update of the e1000 module: http://openvz.org/news/updates/kernel-022stab078.21

To avoid to overlook problems with new components (such as a newer kernel), it is necessary to re-do the tests mentioned above. But as the cluster is already in production, a second cluster (test cluster) with the same hardware as the main cluster is needed. Use this test cluster to test updates of the kernel or main OS updates for the hardware node before putting them on the production cluster.

I know this is not an easy task, as it is time-consuming and needs additional hardware only for testing. But when really business-critical applications are running on the cluster, it is very good to now that the cluster works fine also with new updates installed on the hardware node. In many cases a dedicated test cluster and the time efford for the testing of updates may cause too much costs. If you cannot do such test of updates, keep in mind that over time (when you must install security updates of the OS or the kernel) you have a cluster that you have not tested in this configuration.

If you need a tested cluster (also with tested kernel updates), you may take a look on this Virtuozzo cluster: http://www.thomas-krenn.com/cluster

== How to do OpenVZ kernel updates when it contains a new DRBD version ==

As mentioned above, it is important to use the correct version of the DRBD userspace tools. When an OpenVZ kernel contains a new DRBD version, it is important that the DRBD API version of the userspace tools matches the API version of the DRBD module that is included in the OpenVZ kernel. The API versions can be found at http://svn.drbd.org/drbd/branches/drbd-0.7/ChangeLog. The best way is to always use the version of the DRBD userspace tools that matches the version of the DRBD module that is included in the OpenVZ kernel.

In this example the initial cluster installation contained OpenVZ kernel 2.6.8-022stab078.10, which contains the DRBD module 0.7.17. The steps below show the update procedure to OpenVZ kernel 2.6.8-022stab078.14, which contains the DRBD module 0.7.20.
In the first step build the DRBD userspace tools version 0.7.20 on your buildmachine. Then stop Heartbeat and DRBD on the passive node (hint: you can use 'cat /proc/drbd' to get a hint which node is active and which one is passive):
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services:
[ OK ]
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/drbd stop
Stopping all DRBD resources.
[root@ovz-node2 ~]# cat /proc/drbd
cat: /proc/drbd: No such file or directory
[root@ovz-node2 ~]#
</pre>
Then install the new kernel and the DRBD userspace tools on this node:
<pre>
[root@ovz-node2 ~]# rpm -ihv ovzkernel-2.6.8-022stab078.14.i686.rpm
warning: ovzkernel-2.6.8-022stab078.14.i686.rpm: V3 DSA signature: NOKEY, key ID a7a1d4b6
Preparing... ########################################### [100%]
1:ovzkernel ########################################### [100%]
[root@ovz-node2 ~]# rpm -Uhv drbd-0.7.20-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
/sbin/service
Stopping all DRBD resources.
[root@ovz-node2 ~]#
</pre>
Now set the new kernel as default kernel in /etc/grub.conf and then reboot this node.

After the reboot, the new DRBD version is visible:
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.20 (api:79/proto:74)
SVN Revision: 2260 build by phil@mescal, 2006-07-04 15:18:57
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:0 nr:28 dw:28 dr:0 al:0 bm:2 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]#
</pre>

To update the other node, switch-over the services to make the current active node the passive node. Execute the following on the still active node (it could be that the hb_standby command is located in /usr/lib/heartbeat):
<pre>
[root@ovz-node1 ~]# /usr/lib64/heartbeat/hb_standby
2006/08/03_21:09:41 Going standby [all].
[root@ovz-node1 ~]#
</pre>

Now do the same steps on the new passive node to update it: stop Heartbeat and DRBD, install the new kernel and the new DRBD userspace tools, set the new kernel as default kernel in /etc/grub.conf and reboot the node.

== How to do updates of vzctl, vzctl-lib, and vzquota ==

Ensure after every update of OpenVZ tools that OpenVZ is not started on system boot. To disable starting of OpenVZ on system boot execute on both nodes:
<pre>
[root@ovz-node1 ~]# chkconfig --del vz
[root@ovz-node1 ~]#
</pre>

[[Category: HOWTO]]

HA cluster with DRBD and Heartbeat

2006-10-11T15:31:15Z

Wfischer: added 2 chapters on testing

This article shows how to setup a OpenVZ high availability (HA) cluster using the data replication software DRBD and the cluster manager Heartbeat. In this example the two machines builing the cluster run on CentOS 4.3. The article also shows how to do kernel updates in the cluster, including necessary steps like recompiling of new DRBD userspace tools. For this purpose, kernel 2.6.8-022stab078.10 (containing DRBD module 0.7.17) is used as initial kernel version, and kernel 2.6.8-022stab078.14 (containing DRBD module 0.7.20) as updated kernel version.

Additional information about clustering of virtual machines can be found in the following paper: http://www.linuxtag.org/2006/fileadmin/linuxtag/dvd/12080-paper.pdf

Some other additional information can be found in the documentation of the Thomas-Krenn.AG cluster (The author of this howto is working in the cluster development there, that is the reason why he was able to write this howto :-). The full documentation with interesting illustrations is currently only available in German:
http://my.thomas-krenn.com/service_support/index.php/page.242

== Prerequisites ==
The OpenVZ kernel already includes the DRBD module. The DRBD userspace tools and the cluster manager Heartbeat must be provided seperately. As the API version of the DRBD userspace tools must exactly match the API version of the module, compile them yourself. Also compile Heartbeat yourself, as at the time of this writing the CentOS extras repository only contained an old CVS version of Heartbeat.

On a hardware node for production use there should not be any application that is not really needed for running OpenVZ (any things which are not needed by OpenVZ should run in a VE for security reasons). As a result, compile DRBD and Heartbeat on another machine running CentOS 4.3 (in this example I used a virtual machine on a VMware Server).

=== Compiling Heartbeat ===
Heartbeat version 1.2.* has successfully been used in a lot of two-node-clusters around the world. As the codebase used in version 1.2.* is in production use for many years now, the code is very stable. At the time of writing, Heartbeat version 1.2.4 is the current version of the 1.2.* branch.

Get the tar.gz of the current version of the 1.2.* branch from http://linux-ha.org/download/index.html, at the time of this writing this is http://linux-ha.org/download/heartbeat-1.2.4.tar.gz. Use rpmbuild to build the package:
<pre>rpmbuild -ta heartbeat-1.2.4.tar.gz</pre>
After that, you find four rpm packes in /usr/src/redhat/RPMS/i386 (heartbeat-1.2.4-1.i386.rpm, heartbeat-ldirectord-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, heartbeat-stonith-1.2.4-1.i386.rpm). In this example only heartbeat-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, and heartbeat-stonith-1.2.4-1.i386.rpm are needed.

=== Compiling DRBD userspace tools ===
When compiling the DRBD userspace tools, you have to take care to take the version that matches the DRBD version that is included in the OpenVZ kernel you want to use. If you are unsure about the version, do the following steps while running the OpenVZ kernel that you want to use on a test machine (I used another virtual machine on a VMware server to try this):
<pre>
[root@testmachine ~]# cat /proc/version
Linux version 2.6.8-022stab078.10 (root@rhel4-32) (gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)) #1 Wed Jun 21 12:01:20 MSD 2006
[root@testmachine ~]# modprobe drbd
[root@testmachine ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Unconfigured
1: cs:Unconfigured
[root@testmachine ~]# rmmod drbd
[root@testmachine ~]#
</pre>
Here the version of the DRBD module is 0.7.17. So the userspace tools for 0.7.17 are neccessary.

Back on the buildmachine, do the following to create the rpm:
<pre>
[root@buildmachine ~]# yum install kernel-devel gcc bison flex
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Nothing to do
[root@buildmachine ~]# tar xfz drbd-0.7.17.tar.gz
[root@buildmachine ~]# cd drbd-0.7.17
[root@buildmachine drbd-0.7.17]# make rpm
[...]
You have now:
-rw-r--r-- 1 root root 288728 Jul 30 10:40 dist/RPMS/i386/drbd-0.7.17-1.i386.rpm
-rw-r--r-- 1 root root 518369 Jul 30 10:40 dist/RPMS/i386/drbd-km-2.6.9_34.0.2.EL-0.7.17-1.i386.rpm
[root@buildmachine drbd-0.7.17]#
</pre>
Note that in this way the kernel-devel from CentOS is used, but this does not matter as the created drbd-km rpm will not be used (the DRBD kernel module is already included in OpenVZ kernel). If the kernel-devel package is not the same version as the kernel package that is currently running, it is possible to execute 'make rpm KDIR=/usr/src/kernels/2.6.9-34.0.2.EL-i686/' to directly point to the kernel sources.

== Installing the two nodes ==
Install the two machines in the same way as you would install them for a normal OpenVZ installation, but do not create a filesystem for the /vz. This filesystem will be installed later on on top of DRBD.
{| class="wikitable"
|+'''Example installation configuration'''
! Parameter !! node1 !! node2
|-
! hostname
| ovz-node1
| ovz-node2
|-
! / filesystem
| hda1, 10 GB
| hda1, 10 GB
|-
! swap space
| hda2, 2048 MB
| hda2, 2048 MB
|-
! public LAN
| eth0, 192.168.1.201
| eth0, 192.168.1.202
|-
! private LAN
| eth1, 192.168.255.1 (Gbit Ethernet)
| eth1, 192.168.255.2 (Gbit Ethernet)
|-
! other install options
| no firewall, no SELinux
| no firewall, no SELinux
|-
! package groups
| deactivated everything, only kept vim-enhanced
| deactivated everything, only kept vim-enhanced
|}

== Installing OpenVZ ==

Get the OpenVZ kernel and utilities and install them on both nodes, as described in [[quick installation]]. Update grub configuration to use the OpenVZ kernel by default. Disable starting of OpenVZ on system boot on both nodes (OpenVZ will be started and stopped by Heartbeat):
<pre>
[root@ovz-node1 ~]# chkconfig --del vz
[root@ovz-node1 ~]#
</pre>
Then reboot both machines.

== Setting up DRBD ==

On each of the two nodes create a partition that acts as underlying DRBD device. The partitions should have exactly the same size (I created a 10 GB partition hda3 using fdisk on each node for this example). Note that it might be necessary to reboot the machines to re-read the partition table.

Install the rpm of the DRBD userspace tools on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv drbd-0.7.17-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Then create the drbd.conf configuration file and copy it to /etc/drbd.conf on both nodes. Below is the example configuration file that is used in this article:
<pre>
resource r0 {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";

startup {
degr-wfc-timeout 120;
}

net {
on-disconnect reconnect;
}

disk {
on-io-error detach;
}

syncer {
rate 30M;
group 1;
al-extents 257;
}

on ovz-node1 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.1:7788;
meta-disk internal;
}

on ovz-node2 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.255.2:7788;
meta-disk internal;
}

}
</pre>
Start DRBD on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/drbd start
Starting DRBD resources: [ d0 s0 n0 ].
[root@ovz-node1 ~]#
</pre>
Then check the status of /proc/drbd:
<pre>
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Secondary ld:Inconsistent
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
[root@ovz-node1 ~]#
</pre>
Both nodes are now Secondary and Inconsistent. The latter is because the underlying storage is not yet in-sync, and DRBD has no way to know whether you want the initial sync from ovz-node1 to ovz-node2, or ovz-node2 to ovz-node1. As there is no data below it yet, it does not matter.

To start the sync from ovz-node1 to ovz-node2, do the following on ovz-node1:
<pre>
[root@ovz-node1 ~]# drbdadm -- --do-what-I-say primary all
[root@ovz-node1 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:SyncSource st:Primary/Secondary ld:Consistent
ns:627252 nr:0 dw:0 dr:629812 al:0 bm:38 lo:640 pe:0 ua:640 ap:0
[=>..................] sync'ed: 6.6% (8805/9418)M
finish: 0:04:51 speed: 30,888 (27,268) K/sec
[root@ovz-node1 ~]#
</pre>
As you see, DRBD syncs with about 30 MB per second, as we told it so in /etc/drbd.conf. On the SyncSource (ovz-node1 in this case) the DRBD device is already useable (although it is syncing in the background).

So you can immediately create the filesystem:
<pre>
[root@ovz-node1 ~]# mkfs.ext3 /dev/drbd0
[...]
[root@ovz-node1 ~]#
</pre>

=== Copy necessary OpenVZ files to DRBD device ===

Move the original /vz directory to /vz.orig and recreate the /vz directory to have it as a mount point '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /vz /vz.orig
[root@ovz-node1 ~]# mkdir /vz
[root@ovz-node1 ~]#
</pre>

Afterwards move the necessary OpenVZ directories (/etc/vz, /etc/sysconfig/vz-scripts, /var/vzquota) and replace them with symbolic links '''(do this on both nodes)''':
<pre>
[root@ovz-node1 ~]# mv /etc/vz /etc/vz.orig
[root@ovz-node1 ~]# mv /etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts.orig
[root@ovz-node1 ~]# mv /var/vzquota /var/vzquota.orig
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/vz /etc/vz
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts
[root@ovz-node1 ~]# ln -s /vz/cluster/var/vzquota /var/vzquota
[root@ovz-node1 ~]#
</pre>

Currently, ovz-node1 is still Primary of /dev/drbd0. You can now mount it and copy the necessary files to it '''(only on ovz-node1!)''':
<pre>
[root@ovz-node1 ~]# mount /dev/drbd0 /vz
[root@ovz-node1 ~]# cp -a /vz.orig/* /vz/
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc
[root@ovz-node1 ~]# mkdir -p /vz/cluster/etc/sysconfig
[root@ovz-node1 ~]# mkdir -p /vz/cluster/var
[root@ovz-node1 ~]# cp -a /etc/vz /vz/cluster/etc/
[root@ovz-node1 ~]# cp -a /etc/sysconfig/vz-scripts /vz/cluster/etc/sysconfig/
[root@ovz-node1 ~]# cp -a /var/vzquota /vz/cluster/var/
[root@ovz-node1 ~]# umount /dev/drbd0
[root@ovz-node1 ~]#
</pre>

== Setting up Heartbeat ==

Install the neccessary Heartbeat rpms on both nodes:
<pre>
[root@ovz-node1 ~]# rpm -ihv heartbeat-1.2.4-1.i386.rpm heartbeat-pils-1.2.4-1.i386.rpm heartbeat-stonith-1.2.4-1.i386.rpm
Preparing... ########################################### [100%]
1:heartbeat-pils ########################################### [ 33%]
2:heartbeat-stonith ########################################### [ 67%]
3:heartbeat ########################################### [100%]
[root@ovz-node1 ~]#
</pre>
Create the Heartbeat configuration file ha.cf and copy it to /etc/ha.d/ha.cf on both nodes. Details about this file can be found at http://www.linux-ha.org/ha.cf. Below is an example configuration which uses the two network connections and also a serial connection for heartbeat packets:
<pre>
# Heartbeat logging configuration
logfacility daemon

# Heartbeat cluster members
node ovz-node1
node ovz-node2

# Heartbeat communication timing
keepalive 1
warntime 10
deadtime 30
initdead 120

# Heartbeat communication paths
udpport 694
ucast eth1 192.168.255.1
ucast eth1 192.168.255.2
ucast eth0 192.168.1.201
ucast eth0 192.168.1.202
baud 19200
serial /dev/ttyS0

# Don't fail back automatically
auto_failback off

# Monitoring of network connection to default gateway
ping 192.168.1.1
respawn hacluster /usr/lib64/heartbeat/ipfail
</pre>
Create the Heartbeat configuration file authkeys and copy it to /etc/ha.d/authkeys on both nodes. Set the permissions of this file to 600. Details about this file can be found at http://www.linux-ha.org/authkeys. Below is an example:
<pre>
auth 1
1 sha1 PutYourSuperSecretKeyHere
</pre>
Create the Heartbeat configuration file haresources and copy it to /etc/ha.d/haresources on both nodes. Details about this file can be found at http://www.linux-ha.org/haresources. Below is an example:
<pre>
ovz-node1 drbddisk::r0 Filesystem::/dev/drbd0::/vz::ext3 vz MailTo::youremail@yourdomain.tld
</pre>
Finally, you can now start heartbeat on both nodes:
<pre>
[root@ovz-node1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services:
[ OK ]
[root@ovz-node1 ~]#
</pre>

== Bofore going in production: testing, testing, testing, and ...hm... testing! ==

The installation of the cluster is finished at this point. Before putting the cluster in production it is very important to test the cluster. Because of all the possible different kinds of hardware that you may have, you may encounter problems when a failover is necessary. And as the cluster is about high availability, such problems must be found before the cluster is used for production.

Here is one example: The e1000 driver that is included in kernels < 2.6.12 has a problem when a cable gets unplugged while broadcast packets are still being sent out on that interface. When using broadcast communication in Heartbeat on a crossover link, this fills up the transmit ring buffer on the adapter (the buffer is full after about 8 minutes after the cable got unplugged). Using unicast communication in Heartbeat fixes the problem for example. Details see: http://www.osdl.org/developer_bugzilla/show_bug.cgi?id=699#c22

Without testing you may not be aware of such problems and may face them when the cluster is in production and a failover would be necessary. So test your cluster carefully!

Possible tests can include:
* power outage test of active node
* power outage test of passive node
* network connection outage test of eth0 of active node
* network connection outage test of eth0 of passive node
* network connection outage test of crossover network connection
* ...

As mentioned above, some problems only arise after an outage lasts longer than some minutes. So do the tests also with a duration of >1h for example.

Before you start to test, build a test plan. Some valueable information on that can be found in chapter 3 "Testing a highly available Tivoli Storage Manager cluster environment" of the Redbook ''IBM Tivoli Storage Manager in a Clustered Environment''<ref>http://www.redbooks.ibm.com/abstracts/sg246679.html</ref>. In this chapter it is mentioned that the experience of the authoring team is that the testing phase must be at least two times the total implementation time for the cluster.

== Bofore installing kernel updates: testing again ==

New OpenVZ kernel often include driver updates. This kernel for examples includes an update of the e1000 module: http://openvz.org/news/updates/kernel-022stab078.21

To avoid to overlook problems with new components (such as a newer kernel), it is necessary to re-do the tests mentioned above. But as the cluster is already in production, a second cluster (test cluster) with the same hardware as the main cluster is needed. Use this test cluster to test updates of the kernel or main OS updates for the hardware node before putting them on the production cluster.

I know this is not an easy task, as it is time-consuming and needs additional hardware only for testing. But when really business-critical applications are running on the cluster, it is very good to now that the cluster works fine also with new updates installed on the hardware node. In many cases a dedicated test cluster and the time efford for the testing of updates may cause too much costs. If you cannot do such test of updates, keep in mind that over time (when you must install security updates of the OS or the kernel) you have a cluster that you have not tested in this configuration.

If you need a tested cluster (also with tested kernel updates), you may take a look on this Virtuozzo cluster: http://www.thomas-krenn.com/cluster

== How to do OpenVZ kernel updates when it contains a new DRBD version ==

As mentioned above, it is important to use the correct version of the DRBD userspace tools. When an OpenVZ kernel contains a new DRBD version, it is important that the DRBD API version of the userspace tools matches the API version of the DRBD module that is included in the OpenVZ kernel. The API versions can be found at http://svn.drbd.org/drbd/branches/drbd-0.7/ChangeLog. The best way is to always use the version of the DRBD userspace tools that matches the version of the DRBD module that is included in the OpenVZ kernel.

In this example the initial cluster installation contained OpenVZ kernel 2.6.8-022stab078.10, which contains the DRBD module 0.7.17. The steps below show the update procedure to OpenVZ kernel 2.6.8-022stab078.14, which contains the DRBD module 0.7.20.
In the first step build the DRBD userspace tools version 0.7.20 on your buildmachine. Then stop Heartbeat and DRBD on the passive node (hint: you can use 'cat /proc/drbd' to get a hint which node is active and which one is passive):
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services:
[ OK ]
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2093 build by phil@mescal, 2006-03-06 15:04:12
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:60 nr:136 dw:196 dr:97 al:3 bm:3 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]# /etc/init.d/drbd stop
Stopping all DRBD resources.
[root@ovz-node2 ~]# cat /proc/drbd
cat: /proc/drbd: No such file or directory
[root@ovz-node2 ~]#
</pre>
Then install the new kernel and the DRBD userspace tools on this node:
<pre>
[root@ovz-node2 ~]# rpm -ihv ovzkernel-2.6.8-022stab078.14.i686.rpm
warning: ovzkernel-2.6.8-022stab078.14.i686.rpm: V3 DSA signature: NOKEY, key ID a7a1d4b6
Preparing... ########################################### [100%]
1:ovzkernel ########################################### [100%]
[root@ovz-node2 ~]# rpm -Uhv drbd-0.7.20-1.i386.rpm
Preparing... ########################################### [100%]
1:drbd ########################################### [100%]
/sbin/service
Stopping all DRBD resources.
[root@ovz-node2 ~]#
</pre>
Now set the new kernel as default kernel in /etc/grub.conf and then reboot this node.

After the reboot, the new DRBD version is visible:
<pre>
[root@ovz-node2 ~]# cat /proc/drbd
version: 0.7.20 (api:79/proto:74)
SVN Revision: 2260 build by phil@mescal, 2006-07-04 15:18:57
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:0 nr:28 dw:28 dr:0 al:0 bm:2 lo:0 pe:0 ua:0 ap:0
[root@ovz-node2 ~]#
</pre>

To update the other node, switch-over the services to make the current active node the passive node. Execute the following on the still active node (it could be that the hb_standby command is located in /usr/lib/heartbeat):
<pre>
[root@ovz-node1 ~]# /usr/lib64/heartbeat/hb_standby
2006/08/03_21:09:41 Going standby [all].
[root@ovz-node1 ~]#
</pre>

Now do the same steps on the new passive node to update it: stop Heartbeat and DRBD, install the new kernel and the new DRBD userspace tools, set the new kernel as default kernel in /etc/grub.conf and reboot the node.

== How to do updates of vzctl, vzctl-lib, and vzquota ==

Ensure after every update of OpenVZ tools that OpenVZ is not started on system boot. To disable starting of OpenVZ on system boot execute on both nodes:
<pre>
[root@ovz-node1 ~]# chkconfig --del vz
[root@ovz-node1 ~]#
</pre>

[[Category: HOWTO]]

User:Wfischer

2006-09-27T14:09:00Z

Wfischer:

Hi, my name is Werner Fischer. I am working for Thomas-Krenn.AG in the development team for a bundled cluster solution based on Heartbeat, DRBD, and Virtuozzo.

Private website (includes publications and talks): [http://www.wefi.net www.wefi.net]

Company website: [http://www.thomas-krenn.com www.thomas-krenn.com]

Company website (cluster information): [http://www.thomas-krenn.com/cluster www.thomas-krenn.com/cluster]

HA cluster with DRBD and Heartbeat

2006-09-22T10:20:48Z

Wfischer: fixed two typos

HA cluster with DRBD and Heartbeat

2006-09-22T10:15:24Z

Wfischer: /* Copy necessary OpenVZ files to DRBD device */ fixed one link (/etc/vz) and reworded the steps more clearly

HA cluster with DRBD and Heartbeat

2006-09-19T12:50:14Z

Wfischer: added a hint about additional infos on virtuozzo clustering

HA cluster with DRBD and Heartbeat

2006-09-18T07:03:46Z

Wfischer: corrected minor typos and hints

HA cluster with DRBD and Heartbeat

2006-08-04T09:29:04Z

Wfischer: /* How to do updates of vzctl, vzctl-lib, and vzquota */

HA cluster with DRBD and Heartbeat

2006-08-04T09:24:39Z

Wfischer: Added info to disable starting of OpenVZ on system boot /* Installing OpenVZ */

HA cluster with DRBD and Heartbeat

2006-08-04T09:20:45Z

Wfischer: Added an info about kernel-devel version, corrected make rpm command /* Compiling DRBD userspace tools */

HA cluster with DRBD and Heartbeat

2006-08-04T08:56:36Z

Wfischer: /* How to do OpenVZ kernel updates when it contains a new DRBD version */

HA cluster with DRBD and Heartbeat

2006-08-04T08:54:14Z

Wfischer: /* Setting up Heartbeat */

HA cluster with DRBD and Heartbeat

2006-07-30T18:17:53Z

Wfischer: