Editing Using private IPs for Hardware Nodes

Jump to: navigation, search

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.
Latest revision Your text
Line 1: Line 1:
{{Legacy}}
+
This article describes how to assign public IPs to VEs running on OVZ Hardware Nodes in case you have a following network topology:
 
 
This article describes how to assign public IPs to containers running on OVZ Hardware Nodes in case you have a following network topology:
 
 
 
 
[[Image:PrivateIPs_fig1.gif|An initial network topology]]
 
[[Image:PrivateIPs_fig1.gif|An initial network topology]]
 
== Using a spare IP in the same range ==
 
If you have a spare IP to use, you could assign this as a subinterface and use this as nameserver:
 
 
<pre>[HN] ifconfig eth0:1 *.*.*.*
 
[HN] vzctl set 101 --nameserver *.*.*.*</pre>
 
  
 
== Prerequisites ==
 
== Prerequisites ==
This configuration was tested on a RHEL5 OpenVZ Hardware Node and a container based on a Fedora Core 5 template.
+
This configuration was tested on a RHEL5 OVZ Hardware Node and a VE based on a Fedora Core 5 template.
Other host OSs and templates might require some configuration changes, please add corresponding OS specific changes if you've faced any.
+
Other host OSes and templates might require some configuration changes, please, add corresponding OS specific changes if you've faced any.<br>
 +
This article assumes the presence of 'brctl', 'ip', 'ifconfig' utils thus might require installation of missed packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.
  
This article assumes the presence of 'brctl', 'ip' and 'ifconfig' utils. You may need to install missing packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.
+
This article assumes you have already [[Quick installation|installed OpenVZ]], prepared the [[OS template cache]](s) and have [[Basic_operations_in_OpenVZ_environment|VE(s) created]]. If not, follow the links to perform the steps needed.
 +
{{Note|don't assign an IP after VE creation.}}
 +
<br>
  
This article assumes you have already [[Quick installation|installed OpenVZ]],
+
== (1) An OVZ Hardware Node has the only one ethernet interface ==
prepared the [[OS template cache]](s) and have
 
[[Basic_operations_in_OpenVZ_environment|container(s) created]]. If not, follow the links to perform the steps needed.
 
{{Note|don't assign an IP after container creation.}}
 
 
 
== An OVZ Hardware Node has the only one Ethernet interface ==
 
 
(assume eth0)
 
(assume eth0)
  
=== Hardware Node configuration ===
+
=== <u>Hardware Node configuration</u> ===
 
 
{{Warning|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the below commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}
 
  
 
==== Create a bridge device ====
 
==== Create a bridge device ====
[HN]# brctl addbr br0
+
<pre>[HN]# brctl addbr br0</pre>
  
 
==== Remove an IP from eth0 interface ====
 
==== Remove an IP from eth0 interface ====
[HN]# ifconfig eth0 0
+
<pre>[HN]# ifconfig eth0 0</pre>
  
 
==== Add eth0 interface into the bridge ====
 
==== Add eth0 interface into the bridge ====
[HN]# brctl addif br0 eth0
+
<pre>[HN]# brctl addif br0 eth0</pre>
 
   
 
   
 
==== Assign the IP to the bridge ====
 
==== Assign the IP to the bridge ====
 
(the same that was assigned on eth0 earlier)
 
(the same that was assigned on eth0 earlier)
[HN]# ifconfig br0 10.0.0.2/24
+
<pre>[HN]# ifconfig br0 10.0.0.2/24</pre>
  
 
==== Resurrect the default routing ====
 
==== Resurrect the default routing ====
[HN]# ip route add default via 10.0.0.1 dev br0
+
<pre>[HN]# ip route add default via 10.0.0.1 dev br0</pre>
 
   
 
   
 
+
{{Note|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}
  
 
==== A script example ====
 
==== A script example ====
Line 59: Line 46:
 
</pre>
 
</pre>
  
[HN]# /tmp/br_add >/dev/null 2>&1 &
+
<pre>[HN]# /tmp/br_add >/dev/null 2>&1 &</pre>
 +
<br>
 +
=== <u>VE configuration</u> ===
  
=== Container configuration ===
+
==== Start a VE ====
 +
<pre>[HN]# vzctl start 101</pre>
  
==== Start a container ====
+
==== Add a [[Virtual_Ethernet_device|veth interface]] to the VE ====
[HN]# vzctl start 101
+
<pre>[HN]# vzctl set 101 --netif_add eth0 --save</pre>
  
==== Add a [[Virtual_Ethernet_device|veth interface]] to the container ====
+
==== Set up an IP to the newly created VE's veth interface ====
[HN]# vzctl set 101 --netif_add eth0 --save
+
<pre>[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26</pre>
 
 
==== Set up an IP to the newly created container's veth interface ====
 
[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26
 
 
   
 
   
==== Add the container's veth interface to the bridge ====
+
==== Add the VE's veth interface to the bridge ====
[HN]# brctl addif br0 veth101.0
+
<pre>[HN]# brctl addif br0 veth101.0</pre>
 
+
{{Note|veth interface will work as expected only after it is turned by the bridge into the forwarding state after some delay.
{{Note|There will be a delay of about 15 seconds(default for 2.6.18 kernel) while the bridge software runs STP to detect loops and transitions the veth interface to the forwarding state.
+
In 2.6.18 kernel it is 15 sec by default.
 
<!-- /sys/class/net/$BR_NAME/bridge/forward_delay in SEC*USER_HZ -->}}
 
<!-- /sys/class/net/$BR_NAME/bridge/forward_delay in SEC*USER_HZ -->}}
  
==== Set up the default route for the container ====
+
==== Set up the default route for the VE ====
[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0
+
<pre>[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0</pre>
 
   
 
   
==== (Optional) Add CT↔HN routes ====
+
==== (Optional) Add routes VE <-> HN ====
The above configuration provides the following connections:
+
The configuration above provides following connections available:
* CT X ↔ CT Y (where CT X and CT Y can locate on any OVZ HN)
+
<pre>
* CT   Internet
+
VE X <-> VE Y (where VE X and VE Y can locate on any OVZ HN)
 +
VE   <-> Internet
 +
</pre>
 +
* A VE accessibility from the HN depends on if the local gateway provides NAT or not (probably - yes).
 +
* A HN accessibility from a VE depends on if the ISP gateway is aware about the local network addresses (most probably - no).
  
Note that
+
So to provide VE <-> HN accessibility despite the gateways' configuration you can add following route rules:
 
+
<pre>
* The accessability of the CT from the HN depends on the local gateway providing NAT (probably - yes)
+
[HN]# ip route add 85.86.87.195 dev br0
 
+
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0
* The accessability of the HN from the CT depends on the ISP gateway being aware of the local network (probably not)
+
</pre>
 
 
So to provide CT ↔ HN accessibility despite the gateways' configuration you can add the following routes:
 
 
 
[HN]# ip route add 85.86.87.195 dev br0
 
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0
 
  
=== Resulting OpenVZ Node configuration ===
+
=== <u>The resulted OVZ Node configuration</u> ===
[[Image:PrivateIPs_fig2.gif|Resulting OpenVZ Node configuration]]
+
[[Image:PrivateIPs_fig2.gif|The resulted OVZ Node configuration]]
  
=== Making the configuration persistent ===
+
=== <u>Making the configuration persistent</u> ===
  
 
==== Set up a bridge on a HN ====
 
==== Set up a bridge on a HN ====
This can be done by configuring the <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.
+
This can be done by configuring <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.
  
 
Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
 
Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
Line 113: Line 99:
 
GATEWAY=10.0.0.1
 
GATEWAY=10.0.0.1
 
</pre>
 
</pre>
 
+
<br>
To automatically create bridge <code>br0</code> you can create <code>ifcfg-br0</code>:
+
To make bridge <code>br0</code> automatically created you can create <code>ifcfg-br0</code>:
 
<pre>
 
<pre>
 
DEVICE=br0
 
DEVICE=br0
Line 124: Line 110:
 
</pre>
 
</pre>
  
and edit <code>ifcfg-eth0</code> to add the <code>eth0</code> interface into the bridge <code>br0</code>:
+
and edit <code>ifcfg-eth0</code> file to add <code>eth0</code> interface into the bridge <code>br0</code>:
 
<pre>
 
<pre>
 
DEVICE=eth0
 
DEVICE=eth0
Line 131: Line 117:
 
</pre>
 
</pre>
  
==== Edit the container's configuration ====
+
==== Edit the VE's configuration ====
Add these parameters to the <code>/etc/vz/conf/$CTID.conf</code> file which will be used during the network configuration:
+
Add some parameters to the <code>/etc/vz/conf/$VEID.conf</code> which will be used during the network configuration:
* Add <code>VETH_IP_ADDRESS="IP/MASK"</code> (a container can have multiple IPs separated by spaces)
+
* Add/change CONFIG_CUSTOMIZED="yes" (indicates that a custom script should be run on a VE start)
* Add <code>VE_DEFAULT_GATEWAY="CT DEFAULT GATEWAY"</code>
+
* Add VETH_IP_ADDRESS="<VE IP>/<MASK>" (a VE can have multiple IPs separated by spaces)
* Add <code>BRIDGEDEV="BRIDGE NAME"</code> (a bridge name to which the container veth interface should be added)
+
* Add VE_DEFAULT_GATEWAY="<VE DEFAULT GATEWAY>"
 +
* Add BRIDGEDEV="<BRIDGE NAME>" (a bridge name to which the VE veth interface should be added)
  
 
An example:
 
An example:
 
<pre>
 
<pre>
 
# Network customization section
 
# Network customization section
 +
CONFIG_CUSTOMIZED="yes"
 
VETH_IP_ADDRESS="85.86.87.195/26"
 
VETH_IP_ADDRESS="85.86.87.195/26"
 
VE_DEFAULT_GATEWAY="85.86.87.193"
 
VE_DEFAULT_GATEWAY="85.86.87.193"
Line 146: Line 134:
  
 
==== Create a custom network configuration script ====
 
==== Create a custom network configuration script ====
which should be called each time a container is started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
+
which should be called each time a VE started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
 
<pre>
 
<pre>
 
#!/bin/bash
 
#!/bin/bash
 
# /usr/sbin/vznetcfg.custom
 
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a container
+
# a script to bring up bridged network interfaces (veth's) in a VE
  
 
GLOBALCONFIGFILE=/etc/vz/vz.conf
 
GLOBALCONFIGFILE=/etc/vz/vz.conf
CTCONFIGFILE=/etc/vz/conf/$VEID.conf
+
VECONFIGFILE=/etc/vz/conf/$VEID.conf
 
vzctl=/usr/sbin/vzctl
 
vzctl=/usr/sbin/vzctl
 
brctl=/usr/sbin/brctl
 
brctl=/usr/sbin/brctl
Line 159: Line 147:
 
ifconfig=/sbin/ifconfig
 
ifconfig=/sbin/ifconfig
 
. $GLOBALCONFIGFILE
 
. $GLOBALCONFIGFILE
. $CTCONFIGFILE
+
. $VECONFIGFILE
  
 
NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
 
NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
 
for str in $NETIF_OPTIONS; do \
 
for str in $NETIF_OPTIONS; do \
 
         # getting 'ifname' parameter value
 
         # getting 'ifname' parameter value
         if echo "$str" | grep -o "^ifname=" ; then
+
         if [[ "$str" =~ "^ifname=" ]]; then
 
                 # remove the parameter name from the string (along with '=')
 
                 # remove the parameter name from the string (along with '=')
                 CTIFNAME=${str#*=};
+
                 VEIFNAME=${str#*=};
 
         fi
 
         fi
 
         # getting 'host_ifname' parameter value
 
         # getting 'host_ifname' parameter value
         if echo "$str" | grep -o "^host_ifname=" ; then
+
         if [[ "$str" =~ "^host_ifname=" ]]; then
 
                 # remove the parameter name from the string (along with '=')
 
                 # remove the parameter name from the string (along with '=')
 
                 VZHOSTIF=${str#*=};
 
                 VZHOSTIF=${str#*=};
Line 176: Line 164:
  
 
if [ ! -n "$VETH_IP_ADDRESS" ]; then
 
if [ ! -n "$VETH_IP_ADDRESS" ]; then
   echo "According to $CONFIGFILE CT$VEID has no veth IPs configured."
+
   echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
 
   exit 1
 
   exit 1
 
fi
 
fi
  
 
if [ ! -n "$VZHOSTIF" ]; then
 
if [ ! -n "$VZHOSTIF" ]; then
   echo "According to $CONFIGFILE CT$VEID has no veth interface configured."
+
   echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
 
   exit 1
 
   exit 1
 
fi
 
fi
  
if [ ! -n "$CTIFNAME" ]; then
+
if [ ! -n "$VEIFNAME" ]; then
 
   echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
 
   echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
 
   exit 1
 
   exit 1
 
fi
 
fi
  
echo "Initializing interface $VZHOSTIF for CT$VEID."
+
echo "Initializing interface $VZHOSTIF for VE$VEID."
 
$ifconfig $VZHOSTIF 0
 
$ifconfig $VZHOSTIF 0
  
CTROUTEDEV=$VZHOSTIF
+
VEROUTEDEV=$VZHOSTIF
  
 
if [ -n "$BRIDGEDEV" ]; then
 
if [ -n "$BRIDGEDEV" ]; then
 
   echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
 
   echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
   CTROUTEDEV=$BRIDGEDEV
+
   VEROUTEDEV=$BRIDGEDEV
 
   $brctl addif $BRIDGEDEV $VZHOSTIF
 
   $brctl addif $BRIDGEDEV $VZHOSTIF
 
fi
 
fi
  
# Up the interface $CTIFNAME link in CT$VEID
+
# Up the interface $VEIFNAME link in VE$VEID
$vzctl exec $VEID $ip link set $CTIFNAME up
+
$vzctl exec $VEID $ip link set $VEIFNAME up
  
 
for IP in $VETH_IP_ADDRESS; do
 
for IP in $VETH_IP_ADDRESS; do
   echo "Adding an IP $IP to the $CTIFNAME for CT$VEID."
+
   echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
   $vzctl exec $VEID $ip address add $IP dev $CTIFNAME
+
   $vzctl exec $VEID $ip address add $IP dev $VEIFNAME
  
 
   # removing the netmask
 
   # removing the netmask
 
   IP_STRIP=${IP%%/*};
 
   IP_STRIP=${IP%%/*};
  
   echo "Adding a route from CT0 to CT$VEID using $IP_STRIP."
+
   echo "Adding a route from VE0 to VE$VEID."
   $ip route add $IP_STRIP dev $CTROUTEDEV
+
   $ip route add $IP_STRIP dev $VEROUTEDEV
 
done
 
done
  
if [ -n "$CT0_IP" ]; then
+
if [ -n "$VE0_IP" ]; then
   echo "Adding a route from CT$VEID to CT0."
+
   echo "Adding a route from VE$VEID to VE0."
   $vzctl exec $VEID $ip route add $CT0_IP dev $CTIFNAME
+
   $vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
 
fi
 
fi
  
 
if [ -n "$VE_DEFAULT_GATEWAY" ]; then
 
if [ -n "$VE_DEFAULT_GATEWAY" ]; then
   echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for CT$VEID."
+
   echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
 
   $vzctl exec $VEID \
 
   $vzctl exec $VEID \
         $ip route add default via $VE_DEFAULT_GATEWAY dev $CTIFNAME
+
         $ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
 
fi
 
fi
  
 
exit 0
 
exit 0
 
</pre>
 
</pre>
<p><small>Note: this script can be easily extended to work for multiple triples &lt;bridge, ip address, veth device&gt;, see http://sysadmin-ivanov.blogspot.com/2008/02/2-veth-with-2-bridges-on-openvz-at.html </small></p>
 
 
==== Make the script to be run on a container start ====
 
In order to run above script on a container start create the file
 
<code>/etc/vz/vznet.conf</code> with the following contents:
 
 
EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"
 
 
{{Note|<code>/usr/sbin/vznetcfg.custom</code> should be executable (chmod +x /usr/sbin/vznetcfg.custom)}}
 
 
{{Note|When CT is stoped there are HW → CT route(s) still present in route table. We can use On-umount script for solve this.}}
 
 
==== Create On-umount script for remove HW → CT route(s) ====
 
which should be called each time a container with VEID (<code>/etc/vz/conf/$VEID.umount</code>), or any container (<code>/etc/vz/conf/vps.umount</code>) is stopped.
 
  
 +
==== Make the script to be run on a VE start ====
 +
In order to run above script on a VE start create the following <code>/etc/vz/vznet.conf</code> file:
 
<pre>
 
<pre>
#!/bin/bash
+
EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"
# /etc/vz/conf/$VEID.umount or /etc/vz/conf/vps.umount
 
# a script to remove routes to container with veth-bridge from bridge
 
 
 
CTCONFIGFILE=/etc/vz/conf/$VEID.conf
 
ip=/sbin/ip
 
. $CTCONFIGFILE
 
 
 
if [ ! -n "$VETH_IP_ADDRESS" ]; then
 
  exit 0
 
fi
 
 
 
if [ ! -n "$BRIDGEDEV" ]; then
 
  exit 0
 
fi
 
 
 
for IP in $VETH_IP_ADDRESS; do
 
  # removing the netmask
 
  IP_STRIP=${IP%%/*};
 
 
 
  echo "Remove a route from CT0 to CT$VEID using $IP_STRIP."
 
  $ip route del $IP_STRIP dev $BRIDGEDEV
 
done
 
 
 
exit 0
 
 
</pre>
 
</pre>
 +
{{Note|<code>/usr/sbin/vznetcfg.custom</code> should be executable.}}
  
{{Note|The script should be executable (chmod +x /etc/vz/conf/vps.umount)}}
+
==== Setting the route VE -> HN ====
 
+
To set up a route from VE to HN the custom script has to get a HN IP (the $VE0_IP variable in the script). There can be different approaches to specify it:
==== Setting the route CT → HN ====
+
# Add an entry VE0_IP="VE0 IP" to the <code>$VEID.conf</code>
To set up a route from the CT to the HN, the custom script has to get a HN IP (the $CT0_IP variable in the script). There are several ways to specify it:
+
# Add an entry VE0_IP="VE0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
 
+
# Implement some smart algorithm to determine the VE0 IP right in the custom network configuration script
# Add an entry CT0_IP="CT0 IP" to the <code>$VEID.conf</code>
+
Every variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).
# Add an entry CT0_IP="CT0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
 
# Implement some smart algorithm to determine the CT0 IP right in the custom network configuration script
 
 
 
Each variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).
 
  
== An OpenVZ Hardware Node has two Ethernet interfaces ==
+
== (2) An OVZ Hardware Node has two ethernet interfaces ==
Assuming you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from external traffic.
+
Assume you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from the external traffic.
 
Let's assign eth0 for the external traffic and eth1 for the local one.
 
Let's assign eth0 for the external traffic and eth1 for the local one.
  
If there is no need to make the container accessible from the HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of the above configuration:
+
If there is no aim to make VE accessible from HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of above configuration:
* Hardware Node configuration [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
+
* Hardware Node configuration -> [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]
+
* Hardware Node configuration -> [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]
  
It is nesessary to set a local IP for 'br0' to ensure CT ↔ HN connection availability.
+
For the VE <-> HN connections availability it's nesessary to set an IP (local) to the 'br0'.
  
== Putting containers to different subnetworks ==
+
== (3) Putting VEs to different subnetworks ==
 
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the  
 
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the  
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_container.27s_configuration|above configuration]].
+
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_VE.27s_configuration|above configuration]].
  
 
== See also ==
 
== See also ==

Please note that all contributions to OpenVZ Virtuozzo Containers Wiki may be edited, altered, or removed by other contributors. If you don't want your writing to be edited mercilessly, then don't submit it here.
If you are going to add external links to an article, read the External links policy first!

To edit this page, please answer the question that appears below (more info):

Cancel Editing help (opens in new window)

Templates used on this page: