Using private IPs for Hardware Nodes

This article describes how to assign public IPs to VEs running on OVZ Hardware Nodes in case you have a following network topology:

Prerequisites

This article assumes you have already installed OpenVZ, prepared the OS template cache(s) and have VE(s) created. If not, follow the links to perform the steps needed.

Note: don't assign an IP after VE creation.

(1) An OVZ Hardware Node has the only one ethernet interface

(assume eth0)

Hardware Node configuration

Create a bridge device

[HN]# brctl addbr br0

Remove an IP from eth0 interface

[HN]# ifconfig eth0 0

Add eth0 interface into the bridge

[HN]# brctl addif br0 eth0

Assign the IP to the bridge

(the same that was assigned on eth0 earlier)

[HN]# ifconfig br0 10.0.0.2/24

Resurrect the default routing

[HN]# ip route add default via 10.0.0.1 dev br0

Note: if you are configuring the node remotely you must prepare a script with the above commands and run it in background with the redirected output or you'll lose the access to the Node.

A script example

[HN]# cat /tmp/br_add 
#!/bin/bash

brctl addbr br0
ifconfig eth0 0 
brctl addif br0 eth0 
ifconfig br0 10.0.0.2/24 
ip route add default via 10.0.0.1 dev br0

[HN]# /tmp/br_add >/dev/null 2>&1 &

VE configuration

Start a VE

[HN]# vzctl start 101

Add a veth interface to the VE

[HN]# vzctl set 101 --netif_add eth0 --save

Set up an IP to the newly created VE's veth interface

[HN]# vzctl exec 101 ifconfig eth0 85.86.87.194/26

Set up the default route for the VE

[HN]# vzctl exec 101 ip route change default via 85.86.87.192 dev eth0

Add the VE's veth interface to the bridge

[HN]# brctl addif br0 veth101.0

(Optional) Make HN(s) to be accessible from a VE

The configuration above provides following connections available:

VE X <-> VE Y (where VE X and VE Y can locate on any OVZ HN)
VE   <-> Internet
HN    -> VE

If you really need a VE to have an access to the HN(s) add an additional route rule:

[HN]# vzctl exec 101 ip route add 10.0.0.0/24 dev eth0

The resulted OVZ Node configuration

Making the configuration persistent

Set up a bridge on a HN

This can be done by configuring ifcfg-* files located in /etc/sysconfig/network-scripts/.

Assuming you had a configuration file (e.g. ifcfg-eth0) like:

DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1

To make bridge br0 automatically created you can create ifcfg-br0:

DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1

and edit ifcfg-eth0 file to add eth0 interface into the bridge br0:

DEVICE=eth0
ONBOOT=yes
BRIDGE=br0

Edit the VE's configuration

Add some parameters to the /etc/vz/conf/$VEID.conf which will be used during the network configuration:

• Add/change CONFIG_CUSTOMIZED="yes" (indicates that a custom script should be run on a VE start)
• Add VETH_IP_ADDRESS="<VE IP>/<MASK>" (a VE can have multiple IPs separated by spaces)
• Add VE_DEFAULT_GATEWAY="<VE DEFAULT GATEWAY>"
• Add BRIDGEDEV="<BRIDGE NAME>" (a bridge name to which the VE veth interface should be added)

An example:

# Network customization section
CONFIG_CUSTOMIZED="yes"
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"

Create a custom network configuration script

which should be called each time a VE started (e.g. /usr/sbin/vznetcfg.custom):

#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a VE

GLOBALCONFIGFILE=/etc/vz/vz.conf
VECONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
ip=/sbin/ip
. $GLOBALCONFIGFILE
. $VECONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
        # getting 'ifname' parameter value
        if [[ "$str" =~ "^ifname=" ]]; then
                # remove the parameter name from the string (along with '=')
                VEIFNAME=${str#*=};
        fi
        # getting 'host_ifname' parameter value
        if [[ "$str" =~ "^host_ifname=" ]]; then
                # remove the parameter name from the string (along with '=')
                VZHOSTIF=${str#*=};
        fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
   echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
   exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
   echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
   exit 1
fi

if [ ! -n "$VEIFNAME" ]; then
   echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
   exit 1
fi

for IP in $VETH_IP_ADDRESS; do
   echo "Initializing interface $VZHOSTIF for VE$VEID."
   /sbin/ifconfig $VZHOSTIF 0
done

VEROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
   echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
   VEROUTEDEV=$BRIDGEDEV
   /usr/sbin/brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $VEIFNAME link in VE$VEID
$vzctl exec $VEID $ip link set $VEIFNAME up

for IP in $VETH_IP_ADDRESS; do
   echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
   $vzctl exec $VEID $ip address add $IP dev $VEIFNAME

   # removing the netmask
   IP_STRIP=${IP%%/*};

   echo "Adding a route from VE0 to VE$VEID."
   $ip route add $IP_STRIP dev $VEROUTEDEV
done

if [ -n "$VE0_IP" ]; then
   echo "Adding a route from VE$VEID to VE0."
   $vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
   echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
   $vzctl exec $VEID \
        $ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
fi

exit 0

Make the script to be run on a VE start

In order to run above script on a VE start create the following /etc/vz/vznet.conf file:

#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"

Note: both /etc/vz/vznet.conf and /usr/sbin/vznetcfg.custom should be executable files.

Setting the route VE -> HN

To set up a route from VE to HN the custom script has to get a HN IP (the $VE0_IP variable in the script). There can be different approaches to specify it:

1. Add an entry VE0_IP="VE0 IP" to the $VEID.conf
2. Add an entry VE0_IP="VE0 IP" to the /etc/vz/vz.conf (the global configuration config file)
3. Implement some smart algorithm to determine the VE0 IP right in the custom network configuration script

All the variants have their pros and cons, nevertheless for HN static IP configuration variant 2 seems acceptable (and the most simple).

(2) An OVZ Hardware Node has two ethernet interfaces (TODO)

(assume eth0 and eth1)