Editing Using private IPs for Hardware Nodes

This article describes how to assign public IPs to VEs running on OVZ Hardware Nodes in case you have a following network topology:

[[Image:PrivateIPs_fig1.gif|An initial network topology]]

== Prerequisites ==
This configuration was tested on a RHEL5 OpenVZ Hardware Node and a VE based on a Fedora Core 5 template.
Other host OSs and templates might require some configuration changes, please add corresponding OS specific changes if you've faced any.

This article assumes the presence of 'brctl', 'ip' and 'ifconfig' utils. You may need to install missing packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.

This article assumes you have already [[Quick installation|installed OpenVZ]], prepared the [[OS template cache]](s) and have [[Basic_operations_in_OpenVZ_environment|VE(s) created]]. If not, follow the links to perform the steps needed.
{{Note|don't assign an IP after VE creation.}}

== An OVZ Hardware Node has the only one Ethernet interface ==
(assume eth0)

=== Hardware Node configuration ===

==== Create a bridge device ====
 [HN]# brctl addbr br0

==== Remove an IP from eth0 interface ====
 [HN]# ifconfig eth0 0

==== Add eth0 interface into the bridge ====
 [HN]# brctl addif br0 eth0
 
==== Assign the IP to the bridge ====
(the same that was assigned on eth0 earlier)
 [HN]# ifconfig br0 10.0.0.2/24

==== Resurrect the default routing ====
 [HN]# ip route add default via 10.0.0.1 dev br0
 
{{Warning|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}

==== A script example ====
<pre>
[HN]# cat /tmp/br_add 
#!/bin/bash

brctl addbr br0
ifconfig eth0 0 
brctl addif br0 eth0 
ifconfig br0 10.0.0.2/24 
ip route add default via 10.0.0.1 dev br0
</pre>

 [HN]# /tmp/br_add >/dev/null 2>&1 &

=== VE configuration ===

==== Start a VE ====
 [HN]# vzctl start 101

==== Add a [[Virtual_Ethernet_device|veth interface]] to the VE ====
 [HN]# vzctl set 101 --netif_add eth0 --save

==== Set up an IP to the newly created VE's veth interface ====
 [HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26
 
==== Add the VE's veth interface to the bridge ====
 [HN]# brctl addif br0 veth101.0

{{Note|There will be a delay of about 15 seconds(default for 2.6.18 kernel) while the bridge software runs STP to detect loops and transitions the veth interface to the forwarding state.
<!-- /sys/class/net/$BR_NAME/bridge/forward_delay in SEC*USER_HZ -->}}

==== Set up the default route for the VE ====
 [HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0
 
==== (Optional) Add VE↔HN routes ====
The above configuration provides the following connections:
* VE X ↔ VE Y (where VE X and VE Y can locate on any OVZ HN)
* VE   ↔ Internet

Note that

* The accessability of the VE from the HN depends on the local gateway providing NAT(probably - yes)

* The accessability of the HN from the VE depends on the ISP gateway being aware of the local network(probably not)


So to provide VE ↔ HN accessibility despite the gateways' configuration you can add the following routes:

 [HN]# ip route add 85.86.87.195 dev br0
 [HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0

=== Resulting OpenVZ Node configuration ===
[[Image:PrivateIPs_fig2.gif|Resulting OpenVZ Node configuration]]

=== Making the configuration persistent ===

==== Set up a bridge on a HN ====
This can be done by configuring the <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.

Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
<pre>
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

To automatically create bridge <code>br0</code>  you can create <code>ifcfg-br0</code>:
<pre>
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

and edit <code>ifcfg-eth0</code> to add the <code>eth0</code> interface into the bridge <code>br0</code>:
<pre>
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
</pre>

==== Edit the VE's configuration ====
Add these parameters to the <code>/etc/vz/conf/$VEID.conf</code> file which will be used during the network configuration:
* Add/change <code>CONFIG_CUSTOMIZED="yes"</code> (indicates that a custom script should be run on a VE start)
* Add <code>VETH_IP_ADDRESS="VE IP/MASK"</code> (a VE can have multiple IPs separated by spaces)
* Add <code>VE_DEFAULT_GATEWAY="VE DEFAULT GATEWAY"</code>
* Add <code>BRIDGEDEV="BRIDGE NAME"</code> (a bridge name to which the VE veth interface should be added)

An example:
<pre>
# Network customization section
CONFIG_CUSTOMIZED="yes"
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"
</pre>

==== Create a custom network configuration script ====
which should be called each time a VE is started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
<pre>
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a VE

GLOBALCONFIGFILE=/etc/vz/vz.conf
VECONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
brctl=/usr/sbin/brctl
ip=/sbin/ip
ifconfig=/sbin/ifconfig
. $GLOBALCONFIGFILE
. $VECONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
        # getting 'ifname' parameter value
        if [[ "$str" =~ "^ifname=" ]]; then
                # remove the parameter name from the string (along with '=')
                VEIFNAME=${str#*=};
        fi
        # getting 'host_ifname' parameter value
        if [[ "$str" =~ "^host_ifname=" ]]; then
                # remove the parameter name from the string (along with '=')
                VZHOSTIF=${str#*=};
        fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
   echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
   exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
   echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
   exit 1
fi

if [ ! -n "$VEIFNAME" ]; then
   echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
   exit 1
fi

echo "Initializing interface $VZHOSTIF for VE$VEID."
$ifconfig $VZHOSTIF 0

VEROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
   echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
   VEROUTEDEV=$BRIDGEDEV
   $brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $VEIFNAME link in VE$VEID
$vzctl exec $VEID $ip link set $VEIFNAME up

for IP in $VETH_IP_ADDRESS; do
   echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
   $vzctl exec $VEID $ip address add $IP dev $VEIFNAME

   # removing the netmask
   IP_STRIP=${IP%%/*};

   echo "Adding a route from VE0 to VE$VEID."
   $ip route add $IP_STRIP dev $VEROUTEDEV
done

if [ -n "$VE0_IP" ]; then
   echo "Adding a route from VE$VEID to VE0."
   $vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
   echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
   $vzctl exec $VEID \
        $ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
fi

exit 0
</pre>

==== Make the script to be run on a VE start ====
In order to run above script on a VE start create the file <code>/etc/vz/vznet.conf</code> with the following contents:

 EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"

{{Note|<code>/usr/sbin/vznetcfg.custom</code> should be executable.(chmod +x /usr/sbin/vznetcfg.custom)}}

==== Setting the route VE → HN ====
To set up a route from the VE to the HN, the custom script has to get a HN IP (the $VE0_IP variable in the script). There are several ways to specify it:

# Add an entry VE0_IP="VE0 IP" to the <code>$VEID.conf</code>
# Add an entry VE0_IP="VE0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
# Implement some smart algorithm to determine the VE0 IP right in the custom network configuration script

Each variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).

== An OpenVZ Hardware Node has two Ethernet interfaces ==
Assuming you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from  external traffic.
Let's assign eth0 for the external traffic and eth1 for the local one.

If there is no need to make the VE accessible from the HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of the above configuration:
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]

It is nesessary to set a local IP for 'br0' to ensure VE ↔ HN connection availability.

== Putting VEs to different subnetworks ==
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the 
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_VE.27s_configuration|above configuration]].

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

[[Category: HOWTO]]
[[Category: Networking]]