OpenVZ Virtuozzo Containers Wiki - User contributions [en]

Using private IPs for Hardware Nodes

2012-01-04T15:18:54Z

SergeyIvanov: fix link. - Sergey Ivanov

This article describes how to assign public IPs to containers running on OVZ Hardware Nodes in case you have a following network topology:

[[Image:PrivateIPs_fig1.gif|An initial network topology]]

== Using a spare IP in the same range ==
If you have a spare IP to use, you could assign this as a subinterface and use this as nameserver:

<pre>[HN] ifconfig eth0:1 *.*.*.*
[HN] vzctl set 101 --nameserver *.*.*.*</pre>

== Prerequisites ==
This configuration was tested on a RHEL5 OpenVZ Hardware Node and a container based on a Fedora Core 5 template.
Other host OSs and templates might require some configuration changes, please add corresponding OS specific changes if you've faced any.

This article assumes the presence of 'brctl', 'ip' and 'ifconfig' utils. You may need to install missing packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.

This article assumes you have already [[Quick installation|installed OpenVZ]],
prepared the [[OS template cache]](s) and have
[[Basic_operations_in_OpenVZ_environment|container(s) created]]. If not, follow the links to perform the steps needed.
{{Note|don't assign an IP after container creation.}}

== An OVZ Hardware Node has the only one Ethernet interface ==
(assume eth0)

=== Hardware Node configuration ===

==== Create a bridge device ====
[HN]# brctl addbr br0

==== Remove an IP from eth0 interface ====
[HN]# ifconfig eth0 0

==== Add eth0 interface into the bridge ====
[HN]# brctl addif br0 eth0

==== Assign the IP to the bridge ====
(the same that was assigned on eth0 earlier)
[HN]# ifconfig br0 10.0.0.2/24

==== Resurrect the default routing ====
[HN]# ip route add default via 10.0.0.1 dev br0

{{Warning|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}

==== A script example ====
<pre>
[HN]# cat /tmp/br_add
#!/bin/bash

brctl addbr br0
ifconfig eth0 0
brctl addif br0 eth0
ifconfig br0 10.0.0.2/24
ip route add default via 10.0.0.1 dev br0
</pre>

[HN]# /tmp/br_add >/dev/null 2>&1 &

=== Container configuration ===

==== Start a container ====
[HN]# vzctl start 101

==== Add a [[Virtual_Ethernet_device|veth interface]] to the container ====
[HN]# vzctl set 101 --netif_add eth0 --save

==== Set up an IP to the newly created container's veth interface ====
[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26

==== Add the container's veth interface to the bridge ====
[HN]# brctl addif br0 veth101.0

{{Note|There will be a delay of about 15 seconds(default for 2.6.18 kernel) while the bridge software runs STP to detect loops and transitions the veth interface to the forwarding state.
}}

==== Set up the default route for the container ====
[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0

==== (Optional) Add CT↔HN routes ====
The above configuration provides the following connections:
* CT X ↔ CT Y (where CT X and CT Y can locate on any OVZ HN)
* CT ↔ Internet

Note that

* The accessability of the CT from the HN depends on the local gateway providing NAT (probably - yes)

* The accessability of the HN from the CT depends on the ISP gateway being aware of the local network (probably not)

So to provide CT ↔ HN accessibility despite the gateways' configuration you can add the following routes:

[HN]# ip route add 85.86.87.195 dev br0
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0

=== Resulting OpenVZ Node configuration ===
[[Image:PrivateIPs_fig2.gif|Resulting OpenVZ Node configuration]]

=== Making the configuration persistent ===

==== Set up a bridge on a HN ====
This can be done by configuring the <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.

Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
<pre>
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

To automatically create bridge <code>br0</code> you can create <code>ifcfg-br0</code>:
<pre>
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

and edit <code>ifcfg-eth0</code> to add the <code>eth0</code> interface into the bridge <code>br0</code>:
<pre>
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
</pre>

==== Edit the container's configuration ====
Add these parameters to the <code>/etc/vz/conf/$CTID.conf</code> file which will be used during the network configuration:
* Add <code>VETH_IP_ADDRESS="IP/MASK"</code> (a container can have multiple IPs separated by spaces)
* Add <code>VE_DEFAULT_GATEWAY="CT DEFAULT GATEWAY"</code>
* Add <code>BRIDGEDEV="BRIDGE NAME"</code> (a bridge name to which the container veth interface should be added)

An example:
<pre>
# Network customization section
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"
</pre>

==== Create a custom network configuration script ====
which should be called each time a container is started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
<pre>
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a container

GLOBALCONFIGFILE=/etc/vz/vz.conf
CTCONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
brctl=/usr/sbin/brctl
ip=/sbin/ip
ifconfig=/sbin/ifconfig
. $GLOBALCONFIGFILE
. $CTCONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
# getting 'ifname' parameter value
if echo "$str" | grep -o "^ifname=" ; then
# remove the parameter name from the string (along with '=')
CTIFNAME=${str#*=};
fi
# getting 'host_ifname' parameter value
if echo "$str" | grep -o "^host_ifname=" ; then
# remove the parameter name from the string (along with '=')
VZHOSTIF=${str#*=};
fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
echo "According to $CONFIGFILE CT$VEID has no veth IPs configured."
exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE CT$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$CTIFNAME" ]; then
echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
exit 1
fi

echo "Initializing interface $VZHOSTIF for CT$VEID."
$ifconfig $VZHOSTIF 0

CTROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
CTROUTEDEV=$BRIDGEDEV
$brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $CTIFNAME link in CT$VEID
$vzctl exec $VEID $ip link set $CTIFNAME up

for IP in $VETH_IP_ADDRESS; do
echo "Adding an IP $IP to the $CTIFNAME for CT$VEID."
$vzctl exec $VEID $ip address add $IP dev $CTIFNAME

# removing the netmask
IP_STRIP=${IP%%/*};

echo "Adding a route from CT0 to CT$VEID using $IP_STRIP."
$ip route add $IP_STRIP dev $CTROUTEDEV
done

if [ -n "$CT0_IP" ]; then
echo "Adding a route from CT$VEID to CT0."
$vzctl exec $VEID $ip route add $CT0_IP dev $CTIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for CT$VEID."
$vzctl exec $VEID \
$ip route add default via $VE_DEFAULT_GATEWAY dev $CTIFNAME
fi

exit 0
</pre>
Note: this script can be easily extended to work for multiple triples <bridge, ip address, veth device>, see http://sysadmin-ivanov.blogspot.com/2008/02/2-veth-with-2-bridges-on-openvz-at.html 

==== Make the script to be run on a container start ====
In order to run above script on a container start create the file
<code>/etc/vz/vznet.conf</code> with the following contents:

EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"

{{Note|<code>/usr/sbin/vznetcfg.custom</code> should be executable (chmod +x /usr/sbin/vznetcfg.custom)}}

{{Note|When CT is stoped there are HW → CT route(s) still present in route table. We can use On-umount script for solve this.}}

==== Create On-umount script for remove HW → CT route(s) ====
which should be called each time a container with VEID (<code>/etc/vz/conf/$VEID.umount</code>), or any container (<code>/etc/vz/conf/vps.umount</code>) is stop.

<pre>
#!/bin/bash
# /etc/vz/conf/$VEID.umount or /etc/vz/conf/vps.umount
# a script to remove routes to container with veth-bridge from bridge

CTCONFIGFILE=/etc/vz/conf/$VEID.conf
ip=/sbin/ip
. $CTCONFIGFILE

if [ ! -n "$VETH_IP_ADDRESS" ]; then
exit 0
fi

if [ ! -n "$BRIDGEDEV" ]; then
exit 0
fi

for IP in $VETH_IP_ADDRESS; do
# removing the netmask
IP_STRIP=${IP%%/*};

echo "Remove a route from CT0 to CT$VEID using $IP_STRIP."
$ip route del $IP_STRIP dev $BRIDGEDEV
done

exit 0
</pre>

{{Note|The script should be executable (chmod +x /etc/vz/conf/vps.umount)}}

==== Setting the route CT → HN ====
To set up a route from the CT to the HN, the custom script has to get a HN IP (the $CT0_IP variable in the script). There are several ways to specify it:

# Add an entry CT0_IP="CT0 IP" to the <code>$VEID.conf</code>
# Add an entry CT0_IP="CT0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
# Implement some smart algorithm to determine the CT0 IP right in the custom network configuration script

Each variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).

== An OpenVZ Hardware Node has two Ethernet interfaces ==
Assuming you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from external traffic.
Let's assign eth0 for the external traffic and eth1 for the local one.

If there is no need to make the container accessible from the HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of the above configuration:
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]

It is nesessary to set a local IP for 'br0' to ensure CT ↔ HN connection availability.

== Putting containers to different subnetworks ==
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_container.27s_configuration|above configuration]].

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

[[Category: HOWTO]]
[[Category: Networking]]

Using private IPs for Hardware Nodes

2012-01-04T15:17:38Z

SergeyIvanov: I have moved it to another blog. - Sergey Ivanov

This article describes how to assign public IPs to containers running on OVZ Hardware Nodes in case you have a following network topology:

[[Image:PrivateIPs_fig1.gif|An initial network topology]]

== Using a spare IP in the same range ==
If you have a spare IP to use, you could assign this as a subinterface and use this as nameserver:

<pre>[HN] ifconfig eth0:1 *.*.*.*
[HN] vzctl set 101 --nameserver *.*.*.*</pre>

== Prerequisites ==
This configuration was tested on a RHEL5 OpenVZ Hardware Node and a container based on a Fedora Core 5 template.
Other host OSs and templates might require some configuration changes, please add corresponding OS specific changes if you've faced any.

This article assumes the presence of 'brctl', 'ip' and 'ifconfig' utils. You may need to install missing packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.

This article assumes you have already [[Quick installation|installed OpenVZ]],
prepared the [[OS template cache]](s) and have
[[Basic_operations_in_OpenVZ_environment|container(s) created]]. If not, follow the links to perform the steps needed.
{{Note|don't assign an IP after container creation.}}

== An OVZ Hardware Node has the only one Ethernet interface ==
(assume eth0)

=== Hardware Node configuration ===

==== Create a bridge device ====
[HN]# brctl addbr br0

==== Remove an IP from eth0 interface ====
[HN]# ifconfig eth0 0

==== Add eth0 interface into the bridge ====
[HN]# brctl addif br0 eth0

==== Assign the IP to the bridge ====
(the same that was assigned on eth0 earlier)
[HN]# ifconfig br0 10.0.0.2/24

==== Resurrect the default routing ====
[HN]# ip route add default via 10.0.0.1 dev br0

{{Warning|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}

==== A script example ====
<pre>
[HN]# cat /tmp/br_add
#!/bin/bash

brctl addbr br0
ifconfig eth0 0
brctl addif br0 eth0
ifconfig br0 10.0.0.2/24
ip route add default via 10.0.0.1 dev br0
</pre>

[HN]# /tmp/br_add >/dev/null 2>&1 &

=== Container configuration ===

==== Start a container ====
[HN]# vzctl start 101

==== Add a [[Virtual_Ethernet_device|veth interface]] to the container ====
[HN]# vzctl set 101 --netif_add eth0 --save

==== Set up an IP to the newly created container's veth interface ====
[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26

==== Add the container's veth interface to the bridge ====
[HN]# brctl addif br0 veth101.0

{{Note|There will be a delay of about 15 seconds(default for 2.6.18 kernel) while the bridge software runs STP to detect loops and transitions the veth interface to the forwarding state.
}}

==== Set up the default route for the container ====
[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0

==== (Optional) Add CT↔HN routes ====
The above configuration provides the following connections:
* CT X ↔ CT Y (where CT X and CT Y can locate on any OVZ HN)
* CT ↔ Internet

Note that

* The accessability of the CT from the HN depends on the local gateway providing NAT (probably - yes)

* The accessability of the HN from the CT depends on the ISP gateway being aware of the local network (probably not)

So to provide CT ↔ HN accessibility despite the gateways' configuration you can add the following routes:

[HN]# ip route add 85.86.87.195 dev br0
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0

=== Resulting OpenVZ Node configuration ===
[[Image:PrivateIPs_fig2.gif|Resulting OpenVZ Node configuration]]

=== Making the configuration persistent ===

==== Set up a bridge on a HN ====
This can be done by configuring the <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.

Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
<pre>
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

To automatically create bridge <code>br0</code> you can create <code>ifcfg-br0</code>:
<pre>
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

and edit <code>ifcfg-eth0</code> to add the <code>eth0</code> interface into the bridge <code>br0</code>:
<pre>
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
</pre>

==== Edit the container's configuration ====
Add these parameters to the <code>/etc/vz/conf/$CTID.conf</code> file which will be used during the network configuration:
* Add <code>VETH_IP_ADDRESS="IP/MASK"</code> (a container can have multiple IPs separated by spaces)
* Add <code>VE_DEFAULT_GATEWAY="CT DEFAULT GATEWAY"</code>
* Add <code>BRIDGEDEV="BRIDGE NAME"</code> (a bridge name to which the container veth interface should be added)

An example:
<pre>
# Network customization section
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"
</pre>

==== Create a custom network configuration script ====
which should be called each time a container is started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
<pre>
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a container

GLOBALCONFIGFILE=/etc/vz/vz.conf
CTCONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
brctl=/usr/sbin/brctl
ip=/sbin/ip
ifconfig=/sbin/ifconfig
. $GLOBALCONFIGFILE
. $CTCONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
# getting 'ifname' parameter value
if echo "$str" | grep -o "^ifname=" ; then
# remove the parameter name from the string (along with '=')
CTIFNAME=${str#*=};
fi
# getting 'host_ifname' parameter value
if echo "$str" | grep -o "^host_ifname=" ; then
# remove the parameter name from the string (along with '=')
VZHOSTIF=${str#*=};
fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
echo "According to $CONFIGFILE CT$VEID has no veth IPs configured."
exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE CT$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$CTIFNAME" ]; then
echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
exit 1
fi

echo "Initializing interface $VZHOSTIF for CT$VEID."
$ifconfig $VZHOSTIF 0

CTROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
CTROUTEDEV=$BRIDGEDEV
$brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $CTIFNAME link in CT$VEID
$vzctl exec $VEID $ip link set $CTIFNAME up

for IP in $VETH_IP_ADDRESS; do
echo "Adding an IP $IP to the $CTIFNAME for CT$VEID."
$vzctl exec $VEID $ip address add $IP dev $CTIFNAME

# removing the netmask
IP_STRIP=${IP%%/*};

echo "Adding a route from CT0 to CT$VEID using $IP_STRIP."
$ip route add $IP_STRIP dev $CTROUTEDEV
done

if [ -n "$CT0_IP" ]; then
echo "Adding a route from CT$VEID to CT0."
$vzctl exec $VEID $ip route add $CT0_IP dev $CTIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for CT$VEID."
$vzctl exec $VEID \
$ip route add default via $VE_DEFAULT_GATEWAY dev $CTIFNAME
fi

exit 0
</pre>
Note: this script can be easily extended to work for multiple triples <bridge, ip address, veth device>, see http://sysadmin-ivanov.blogspot.com/2009/11/autofonix.html 

==== Make the script to be run on a container start ====
In order to run above script on a container start create the file
<code>/etc/vz/vznet.conf</code> with the following contents:

EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"

{{Note|<code>/usr/sbin/vznetcfg.custom</code> should be executable (chmod +x /usr/sbin/vznetcfg.custom)}}

{{Note|When CT is stoped there are HW → CT route(s) still present in route table. We can use On-umount script for solve this.}}

==== Create On-umount script for remove HW → CT route(s) ====
which should be called each time a container with VEID (<code>/etc/vz/conf/$VEID.umount</code>), or any container (<code>/etc/vz/conf/vps.umount</code>) is stop.

<pre>
#!/bin/bash
# /etc/vz/conf/$VEID.umount or /etc/vz/conf/vps.umount
# a script to remove routes to container with veth-bridge from bridge

CTCONFIGFILE=/etc/vz/conf/$VEID.conf
ip=/sbin/ip
. $CTCONFIGFILE

if [ ! -n "$VETH_IP_ADDRESS" ]; then
exit 0
fi

if [ ! -n "$BRIDGEDEV" ]; then
exit 0
fi

for IP in $VETH_IP_ADDRESS; do
# removing the netmask
IP_STRIP=${IP%%/*};

echo "Remove a route from CT0 to CT$VEID using $IP_STRIP."
$ip route del $IP_STRIP dev $BRIDGEDEV
done

exit 0
</pre>

{{Note|The script should be executable (chmod +x /etc/vz/conf/vps.umount)}}

==== Setting the route CT → HN ====
To set up a route from the CT to the HN, the custom script has to get a HN IP (the $CT0_IP variable in the script). There are several ways to specify it:

# Add an entry CT0_IP="CT0 IP" to the <code>$VEID.conf</code>
# Add an entry CT0_IP="CT0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
# Implement some smart algorithm to determine the CT0 IP right in the custom network configuration script

Each variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).

== An OpenVZ Hardware Node has two Ethernet interfaces ==
Assuming you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from external traffic.
Let's assign eth0 for the external traffic and eth1 for the local one.

If there is no need to make the container accessible from the HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of the above configuration:
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]

It is nesessary to set a local IP for 'br0' to ensure CT ↔ HN connection availability.

== Putting containers to different subnetworks ==
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_container.27s_configuration|above configuration]].

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

[[Category: HOWTO]]
[[Category: Networking]]

Talk:Virtual Ethernet device

2011-12-07T22:14:43Z

SergeyIvanov:

Under Common Configurations -> Simple configuration -> Configure devices in VE0, the example shows

<pre>
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
</pre>

being run on the host node, VE0. I don't think this can be correct, because eth0 exists in VE 101, not VE0. {{unsigned|Andrex|01:11, 15 July 2006}}

: This is correct, because we need to enable forwarding on both network interfaces in VE0 (host-node eth0 and veth101.0) to allow the network packets on one network interface (host-node eth0) to be forwarded to another network interface (veth101.0). The same thing with proxy ARP, we need to enable it on host-node eth0 and veth101.0 network interfaces. --[[User:Major|Major]] 08:42, 17 July 2006 (EDT)

==Multiple persistent Veth interfaces==
The script for making a persistent bridge get added to a particular host-bridge on startup VPS/VE, doesn't seem to account for having multiple bridges. I have a bridge to a private lan for administrative functions and a bridge to the public network for inbound connections for services. How might one ensure that the vethVEID.x's are hooked up to the bridges on the host every time the VE is started? --[[User:Btrotter|Btrotter]] 05:34, 17 March 2008 (EDT)

: Please take a look at [[Using private IPs for Hardware Nodes]] and http://sysadmin-ivanov.blogspot.com/2008/02/2-veth-with-2-bridges-on-openvz-at.html. There you will find scripts which will help you to manage 2 bridges, and you can simply extend this script to use more then 2 bridges. --[[User:Major|Major]] 15:59, 27 March 2008 (MSK)

==Making a bridged veth-device persistent==

Is this also obsolete, like the section I just deleted? [[User:Mrjcleaver|Mrjcleaver]] 13:35, 31 May 2008 (EDT)

Please leave this in for now! I just needed it today for setup on a Debian HN. 
Also I noted that a crucial word/reference at the end of the following sentence is missing: "4. Of course, the CT's operating system will need to have ." Could someone with wiki-knowledge please restore this (if it was ever known)?
[[User:Laboa|Laboa]] 22:50, 12. Oct 2008 (CEST)

==Configure devices in CT0==
These devices under /proc didn't exist for me, but I left them in case they are still needed, even with the new vzctl. [[User:Mrjcleaver|Mrjcleaver]] 13:59, 31 May 2008 (EDT)
: These files did not exist because you executed a copy/paste, which is incorrect. You should replace 'veth101.0' with veth###.# where ###.# is your relevant VEID (for scripts this should do: 'veth${VEID}.0') according to your VE IDs, and for each VE ID you use veth. -- {{unsigned|79.107.116.50|21:29, 25 November 2009}}

== Creating veth without OpenVZ ==

How to create veth devices pair without of OpenVZ?
I want something like
<pre>modprobe veth
ifconfig veth0 192.168.0.1
ifconfig veth1 192.168.0.2</pre>

Answer: Enable CONFIG_VETH in your kernel, load the veth module and create a new pair with the 'ip' command from the [http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 iproute2 package].
<pre>modprobe veth
ip link add name veth0 type veth peer name veth1
ifconfig veth0 192.168.0.1
ifconfig veth1 192.168.0.2</pre>

You may also want to check out this [ftp://robur.slu.se/pub/Linux/bifrost/seminars/workshop-2010-01-27/jens/Namespaces.pdf presentation on namespaces in Linux].

== Using iproute toolset or net-tools toolset ==

Hello,

I propose to substitute all usage of

ifconfig <device> 0

with

ip link set <device> up

Since iproute seems to be a requirement, I think its a good idea to drop support for ifconfig.

Virtual Ethernet device

2011-12-07T22:12:15Z

SergeyIvanov:

'''Virtual Ethernet device''' is an Ethernet-like device which can be used
inside a [[container]]. Unlike [[venet]] network device, [[veth]] device
has a MAC address, therefore it can be used in configurations, when veth
is bridged to ethX or other device and container's user fully sets up
his networking himself, including IPs, gateways etc.

Virtual Ethernet device consist of two Ethernet devices --
the one in [[CT0]] and another one in CT. These devices are connected
to each other, so if a packet goes to one
device it will come out from the other device.

== Virtual Ethernet device usage ==

=== Kernel module ===
First of all, make sure the <code>vzethdev</code> module is loaded:
<pre>
# lsmod | grep vzeth
vzethdev 8224 0
vzmon 35164 5 vzethdev,vznetdev,vzrst,vzcpt
vzdev 3080 4 vzethdev,vznetdev,vzmon,vzdquota
</pre>

In case it is not loaded, load it:
<pre>
# modprobe vzethdev
</pre>

=== MAC addresses ===
In the below commands, you should use random MAC addresses. Do not use MAC addresses of real eth devices, because this can lead to collisions.

MAC addresses must be entered in XX:XX:XX:XX:XX:XX format.

YOU MAY NOT NEED TO GENERATE MAC ADDRESSES BY HAND BECAUSE vzctl --veth_add
MAY GENERATE THEM AUTOMATICALLY AS NECESSARY.

Nevertheless, there is a utility script available for generating MAC addresses: http://www.easyvmx.com/software/easymac.sh. It is to be used like this:

chmod +x easymac.sh
./easymac.sh -R

=== Adding veth to a CT ===

vzctl set <CTID> --netif_add <ifname>[,<mac>,<host_ifname>,<host_mac>,<bridge>]

Here
* <tt>ifname</tt> is the Ethernet device name in the CT
* <tt>mac</tt> is its MAC address in the CT
* <tt>host_ifname</tt> is the Ethernet device name on the host ([[CT0]])
* <tt>host_mac</tt> is its MAC address on the host ([[CT0]])
* <tt>bridge</tt> is an optional parameter which can be used in custom network start scripts to automatically add the interface to a bridge.

{{Note|All parameters except <code>ifname</code> are optional and are automatically generated if not specified.}}

Example:

vzctl set 101 --netif_add eth0 --save

Or, if you want to specify everything:

vzctl set 101 --netif_add eth0,00:12:34:56:78:9A,veth101.0,00:12:34:56:78:9B --save

Or, if you want to specify the bridge and leave the other values autogenerated:

vzctl set 101 --netif_add eth0,,,,vmbr1 --save

=== Removing veth from a CT ===

vzctl set <CTID> --netif_del <dev_name>|all

Here
* <code>dev_name</code> is the Ethernet device name in the [[CT]].

{{Note|If you want to remove all Ethernet devices in CT, use <code>all</code>.}}

Example:

vzctl set 101 --netif_del eth0 --save

== Common configurations with virtual Ethernet devices ==
Module <tt>vzethdev</tt> must be loaded to operate with veth devices.

=== Simple configuration with virtual Ethernet device ===

Assuming you have 192.168.0.0/24 on your LAN, you will learn how to integrate a container in this LAN using veth.

==== Start a CT ====

[host-node]# vzctl start 101

==== Add veth device to CT ====

[host-node]# vzctl set 101 --netif_add eth0 --save

This allocates a MAC address and associates it with the host eth0 port.

==== Configure devices in CT0 ====
<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/proxy_arp
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
</pre>

==== Configure device in CT ====
<pre>
[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0
[ve-101]# /sbin/ip addr add 192.168.0.101 dev eth0
[ve-101]# /sbin/ip route add default dev eth0
</pre>

Notes:
* Until you ifconfig eth0 it won't appear. When you do it will use the mac address netif_add added earlier
* 192.168.0.101 is chosen to be an [[unrouteable private ip address]]. Where 101 reminds you that it is node 101.
* The "ip route" tells all traffic to head to "device eth0"
* In theory you could [[use dhcpd with OpenVZ]] and dhclient to pick up an DHCP address from your router instead of hardwiring it
** http://openvz.org/pipermail/users/2005-November/000020.html

==== Add route in [[CT0]] ====

[host-node]# ip route add 192.168.0.101 dev veth101.0

=== Using a directly routed IPv4 with virtual Ethernet device ===

==== Situation ====
Hardware Node (HN/CT0) has 192.168.0.1/24 with router 192.168.0.254.

We also know that IPv4 10.0.0.1/32 is directly routed to 192.168.0.1 (this is called a ''fail-over IP'').

We want to give this directly routed IPv4 address to a container (CT).

==== Start container ====

[host-node]# vzctl start 101

==== Add veth device to CT ====

[host-node]# vzctl set 101 --netif_add eth0 --save

This allocates a MAC address and associates it with the host eth0 port.

==== Configure device and add route in CT0 ====

<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# ip route add 10.0.0.1 dev veth101.0
</pre>

You can automatize this at VPS creation by using a mount script <tt>$VEID.mount</tt>.

The problem here is that the ''veth'' interface appears in CT0 '''after''' VPS has started, therefore we cannot directly use the commands in the mount script. We launch a shell script (enclosed by { }) in background (operator '''&''') that waits for the interface to be ready and then adds the IP route.

Contents of the mount script <tt>/etc/vz/conf/101.mount</tt>:
<pre>
#!/bin/bash
# This script source VPS configuration files in the same order as vzctl does

# if one of these files does not exist then something is really broken
[ -f /etc/vz/vz.conf ] || exit 1
[ -f $VE_CONFFILE ] || exit 1

# source both files. Note the order, it is important
. /etc/vz/vz.conf
. $VE_CONFFILE

# Configure veth with IP after VPS has started
{
IP=X.Y.Z.T
DEV=veth101.0
while sleep 1; do
/sbin/ifconfig $DEV 0 >/dev/null 2>&1
if [ $? -eq 0 ]; then
/sbin/ip route add $IP dev $DEV
break
fi
done
} &
</pre>

==== Make sure IPv4 forwarding is enabled in CT0 ====

<pre>
[host-node]# echo 1 > /proc/sys/net/ipv4/ip_forward
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
</pre>
You can permanently set this by using <tt>/etc/sysctl.conf</tt>.

==== Configure device in CT ====

1. Configure IP address

2. Add gateway

3. Add default route

<pre>
[ve-101]# /sbin/ifconfig eth0 10.0.0.1 netmask 255.255.255.255
[ve-101]# /sbin/ip route add 192.168.0.1 dev eth0
[ve-101]# /sbin/ip route add default via 192.168.0.1
</pre>

In a Debian container, you can configure this permanently by using <tt>/etc/network/interfaces</tt>:
<pre>
auto eth0
iface eth0 inet static
address 10.0.0.1
netmask 255.255.255.255
up /sbin/ip route add 192.168.0.1 dev eth0
up /sbin/ip route add default via 192.168.0.1
</pre>

=== Virtual Ethernet device with IPv6 ===

See the [[VEs and HNs in same subnets]] article.

=== Virtual Ethernet devices can be joined in one bridge ===
Perform steps 1 - 4 from Simple configuration chapter for several containers and/or veth devices

==== Create bridge device ====
<pre>
[host-node]# brctl addbr vzbr0
</pre>

==== Add veth devices to bridge ====
<pre>
[host-node]# brctl addif vzbr0 veth101.0
...
[host-node]# brctl addif vzbr0 veth101.n
[host-node]# brctl addif vzbr0 veth102.0
...
...
[host-node]# brctl addif vzbr0 vethXXX.N
</pre>

==== Configure bridge device ====
<pre>
[host-node]# ifconfig vzbr0 0
</pre>

==== Add routes in [[CT0]] ====
<pre>
[host-node]# ip route add 192.168.101.1 dev vzbr0
...
[host-node]# ip route add 192.168.101.n dev vzbr0
[host-node]# ip route add 192.168.102.1 dev vzbr0
...
...
[host-node]# ip route add 192.168.XXX.N dev vzbr0
</pre>

Thus you'll have more convinient configuration, i.e. all routes to containers will be through this bridge and containers can communicate with each other even without these routes.

=== Making a veth-device persistent ===
According to http://bugzilla.openvz.org/show_bug.cgi?id=301 , a bug that stopped the veth device persistent was "Obsoleted now when --veth_add/--veth_del are introduced"

See http://wiki.openvz.org/w/index.php?title=Virtual_Ethernet_device&diff=5990&oldid=5989#Making_a_veth-device_persistent for a workaround that used to be described in this section.

That's it! At this point, when you restart the CT you should see a new line in the output, indicating that the interface is being configured and a new route being added. And you should be able to ping the host, and to enter the CT and use the network.

=== Making a bridged veth-device persistent ===

Like the above example, here it is how to add the veth device to a bridge in a persistent way.

vzctl include a 'vznetaddbr' script, which makes use of the ''bridge'' parameter of the --netif_add switch.

Just create /etc/vz/vznet.conf containing the following.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetaddbr"
</pre>

The script uses 'vmbr0' as default bridge name when no bridge is specified.

=== Virtual Ethernet devices + VLAN ===
This configuration can be done by adding vlan device to the previous configuration.

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]
* [[Using private IPs for Hardware Nodes]]
* Patch: [[Disable venet interface]]
* Troubleshooting: [[Bridge doesn't forward packets]]

== External links ==
* [http://www.tldp.org/HOWTO/Linux+IPv6-HOWTO/hints-daemons-radvd.html Linux IPv6 HOWTO, a chapter about radvd]
* [http://sysadmin-ivanov.blogspot.com/2008/02/2-veth-with-2-bridges-on-openvz-at.html 2 veth with 2 bridges setup]

[[Category: Networking]]
[[Category: HOWTO]]

Using private IPs for Hardware Nodes

2008-02-20T06:23:22Z

SergeyIvanov: /* Create a custom network configuration script */

This article describes how to assign public IPs to VEs running on OVZ Hardware Nodes in case you have a following network topology:

[[Image:PrivateIPs_fig1.gif|An initial network topology]]

== Prerequisites ==
This configuration was tested on a RHEL5 OpenVZ Hardware Node and a VE based on a Fedora Core 5 template.
Other host OSs and templates might require some configuration changes, please add corresponding OS specific changes if you've faced any.

This article assumes the presence of 'brctl', 'ip' and 'ifconfig' utils. You may need to install missing packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.

This article assumes you have already [[Quick installation|installed OpenVZ]], prepared the [[OS template cache]](s) and have [[Basic_operations_in_OpenVZ_environment|VE(s) created]]. If not, follow the links to perform the steps needed.
{{Note|don't assign an IP after VE creation.}}

== An OVZ Hardware Node has the only one Ethernet interface ==
(assume eth0)

=== Hardware Node configuration ===

==== Create a bridge device ====
[HN]# brctl addbr br0

==== Remove an IP from eth0 interface ====
[HN]# ifconfig eth0 0

==== Add eth0 interface into the bridge ====
[HN]# brctl addif br0 eth0

==== Assign the IP to the bridge ====
(the same that was assigned on eth0 earlier)
[HN]# ifconfig br0 10.0.0.2/24

==== Resurrect the default routing ====
[HN]# ip route add default via 10.0.0.1 dev br0

{{Warning|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}

==== A script example ====
<pre>
[HN]# cat /tmp/br_add
#!/bin/bash

brctl addbr br0
ifconfig eth0 0
brctl addif br0 eth0
ifconfig br0 10.0.0.2/24
ip route add default via 10.0.0.1 dev br0
</pre>

[HN]# /tmp/br_add >/dev/null 2>&1 &

=== VE configuration ===

==== Start a VE ====
[HN]# vzctl start 101

==== Add a [[Virtual_Ethernet_device|veth interface]] to the VE ====
[HN]# vzctl set 101 --netif_add eth0 --save

==== Set up an IP to the newly created VE's veth interface ====
[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26

==== Add the VE's veth interface to the bridge ====
[HN]# brctl addif br0 veth101.0

{{Note|There will be a delay of about 15 seconds(default for 2.6.18 kernel) while the bridge software runs STP to detect loops and transitions the veth interface to the forwarding state.
}}

==== Set up the default route for the VE ====
[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0

==== (Optional) Add VE↔HN routes ====
The above configuration provides the following connections:
* VE X ↔ VE Y (where VE X and VE Y can locate on any OVZ HN)
* VE ↔ Internet

Note that

* The accessability of the VE from the HN depends on the local gateway providing NAT(probably - yes)

* The accessability of the HN from the VE depends on the ISP gateway being aware of the local network(probably not)

So to provide VE ↔ HN accessibility despite the gateways' configuration you can add the following routes:

[HN]# ip route add 85.86.87.195 dev br0
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0

=== Resulting OpenVZ Node configuration ===
[[Image:PrivateIPs_fig2.gif|Resulting OpenVZ Node configuration]]

=== Making the configuration persistent ===

==== Set up a bridge on a HN ====
This can be done by configuring the <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.

Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
<pre>
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

To automatically create bridge <code>br0</code> you can create <code>ifcfg-br0</code>:
<pre>
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

and edit <code>ifcfg-eth0</code> to add the <code>eth0</code> interface into the bridge <code>br0</code>:
<pre>
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
</pre>

==== Edit the VE's configuration ====
Add these parameters to the <code>/etc/vz/conf/$VEID.conf</code> file which will be used during the network configuration:
* Add/change <code>CONFIG_CUSTOMIZED="yes"</code> (indicates that a custom script should be run on a VE start)
* Add <code>VETH_IP_ADDRESS="VE IP/MASK"</code> (a VE can have multiple IPs separated by spaces)
* Add <code>VE_DEFAULT_GATEWAY="VE DEFAULT GATEWAY"</code>
* Add <code>BRIDGEDEV="BRIDGE NAME"</code> (a bridge name to which the VE veth interface should be added)

An example:
<pre>
# Network customization section
CONFIG_CUSTOMIZED="yes"
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"
</pre>

==== Create a custom network configuration script ====
which should be called each time a VE is started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
<pre>
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a VE

GLOBALCONFIGFILE=/etc/vz/vz.conf
VECONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
brctl=/usr/sbin/brctl
ip=/sbin/ip
ifconfig=/sbin/ifconfig
. $GLOBALCONFIGFILE
. $VECONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
# getting 'ifname' parameter value
if [[ "$str" =~ "^ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VEIFNAME=${str#*=};
fi
# getting 'host_ifname' parameter value
if [[ "$str" =~ "^host_ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VZHOSTIF=${str#*=};
fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VEIFNAME" ]; then
echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
exit 1
fi

echo "Initializing interface $VZHOSTIF for VE$VEID."
$ifconfig $VZHOSTIF 0

VEROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
VEROUTEDEV=$BRIDGEDEV
$brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $VEIFNAME link in VE$VEID
$vzctl exec $VEID $ip link set $VEIFNAME up

for IP in $VETH_IP_ADDRESS; do
echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
$vzctl exec $VEID $ip address add $IP dev $VEIFNAME

# removing the netmask
IP_STRIP=${IP%%/*};

echo "Adding a route from VE0 to VE$VEID."
$ip route add $IP_STRIP dev $VEROUTEDEV
done

if [ -n "$VE0_IP" ]; then
echo "Adding a route from VE$VEID to VE0."
$vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
$vzctl exec $VEID \
$ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
fi

exit 0
</pre>
Note: this script can be easily extended to work for multiple triples <bridge, ip address, veth device>, see http://vireso.blogspot.com/2008/02/2-veth-with-2-brindges-on-openvz-at.html 

==== Make the script to be run on a VE start ====
In order to run above script on a VE start create the file <code>/etc/vz/vznet.conf</code> with the following contents:

EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"

{{Note|<code>/usr/sbin/vznetcfg.custom</code> should be executable.(chmod +x /usr/sbin/vznetcfg.custom)}}

==== Setting the route VE → HN ====
To set up a route from the VE to the HN, the custom script has to get a HN IP (the $VE0_IP variable in the script). There are several ways to specify it:

# Add an entry VE0_IP="VE0 IP" to the <code>$VEID.conf</code>
# Add an entry VE0_IP="VE0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
# Implement some smart algorithm to determine the VE0 IP right in the custom network configuration script

Each variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).

== An OpenVZ Hardware Node has two Ethernet interfaces ==
Assuming you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from external traffic.
Let's assign eth0 for the external traffic and eth1 for the local one.

If there is no need to make the VE accessible from the HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of the above configuration:
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]

It is nesessary to set a local IP for 'br0' to ensure VE ↔ HN connection availability.

== Putting VEs to different subnetworks ==
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_VE.27s_configuration|above configuration]].

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

[[Category: HOWTO]]
[[Category: Networking]]

Using private IPs for Hardware Nodes

2008-02-10T17:43:59Z

SergeyIvanov: /* An OpenVZ Hardware Node has two Ethernet interfaces */

This article describes how to assign public IPs to VEs running on OVZ Hardware Nodes in case you have a following network topology:

[[Image:PrivateIPs_fig1.gif|An initial network topology]]

== Prerequisites ==
This configuration was tested on a RHEL5 OpenVZ Hardware Node and a VE based on a Fedora Core 5 template.
Other host OSs and templates might require some configuration changes, please add corresponding OS specific changes if you've faced any.

This article assumes the presence of 'brctl', 'ip' and 'ifconfig' utils. You may need to install missing packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.

This article assumes you have already [[Quick installation|installed OpenVZ]], prepared the [[OS template cache]](s) and have [[Basic_operations_in_OpenVZ_environment|VE(s) created]]. If not, follow the links to perform the steps needed.
{{Note|don't assign an IP after VE creation.}}

== An OVZ Hardware Node has the only one Ethernet interface ==
(assume eth0)

=== Hardware Node configuration ===

==== Create a bridge device ====
[HN]# brctl addbr br0

==== Remove an IP from eth0 interface ====
[HN]# ifconfig eth0 0

==== Add eth0 interface into the bridge ====
[HN]# brctl addif br0 eth0

==== Assign the IP to the bridge ====
(the same that was assigned on eth0 earlier)
[HN]# ifconfig br0 10.0.0.2/24

==== Resurrect the default routing ====
[HN]# ip route add default via 10.0.0.1 dev br0

{{Warning|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}

==== A script example ====
<pre>
[HN]# cat /tmp/br_add
#!/bin/bash

brctl addbr br0
ifconfig eth0 0
brctl addif br0 eth0
ifconfig br0 10.0.0.2/24
ip route add default via 10.0.0.1 dev br0
</pre>

[HN]# /tmp/br_add >/dev/null 2>&1 &

=== VE configuration ===

==== Start a VE ====
[HN]# vzctl start 101

==== Add a [[Virtual_Ethernet_device|veth interface]] to the VE ====
[HN]# vzctl set 101 --netif_add eth0 --save

==== Set up an IP to the newly created VE's veth interface ====
[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26

==== Add the VE's veth interface to the bridge ====
[HN]# brctl addif br0 veth101.0

{{Note|There will be a delay of about 15 seconds(default for 2.6.18 kernel) while the bridge software runs STP to detect loops and transitions the veth interface to the forwarding state.
}}

==== Set up the default route for the VE ====
[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0

==== (Optional) Add VE↔HN routes ====
The above configuration provides the following connections:
* VE X ↔ VE Y (where VE X and VE Y can locate on any OVZ HN)
* VE ↔ Internet

Note that

* The accessability of the VE from the HN depends on the local gateway providing NAT(probably - yes)

* The accessability of the HN from the VE depends on the ISP gateway being aware of the local network(probably not)

So to provide VE ↔ HN accessibility despite the gateways' configuration you can add the following routes:

[HN]# ip route add 85.86.87.195 dev br0
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0

=== Resulting OpenVZ Node configuration ===
[[Image:PrivateIPs_fig2.gif|Resulting OpenVZ Node configuration]]

=== Making the configuration persistent ===

==== Set up a bridge on a HN ====
This can be done by configuring the <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.

Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
<pre>
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

To automatically create bridge <code>br0</code> you can create <code>ifcfg-br0</code>:
<pre>
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

and edit <code>ifcfg-eth0</code> to add the <code>eth0</code> interface into the bridge <code>br0</code>:
<pre>
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
</pre>

==== Edit the VE's configuration ====
Add these parameters to the <code>/etc/vz/conf/$VEID.conf</code> file which will be used during the network configuration:
* Add/change <code>CONFIG_CUSTOMIZED="yes"</code> (indicates that a custom script should be run on a VE start)
* Add <code>VETH_IP_ADDRESS="VE IP/MASK"</code> (a VE can have multiple IPs separated by spaces)
* Add <code>VE_DEFAULT_GATEWAY="VE DEFAULT GATEWAY"</code>
* Add <code>BRIDGEDEV="BRIDGE NAME"</code> (a bridge name to which the VE veth interface should be added)

An example:
<pre>
# Network customization section
CONFIG_CUSTOMIZED="yes"
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"
</pre>

==== Create a custom network configuration script ====
which should be called each time a VE is started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
<pre>
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a VE

GLOBALCONFIGFILE=/etc/vz/vz.conf
VECONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
brctl=/usr/sbin/brctl
ip=/sbin/ip
ifconfig=/sbin/ifconfig
. $GLOBALCONFIGFILE
. $VECONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
# getting 'ifname' parameter value
if [[ "$str" =~ "^ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VEIFNAME=${str#*=};
fi
# getting 'host_ifname' parameter value
if [[ "$str" =~ "^host_ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VZHOSTIF=${str#*=};
fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VEIFNAME" ]; then
echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
exit 1
fi

echo "Initializing interface $VZHOSTIF for VE$VEID."
$ifconfig $VZHOSTIF 0

VEROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
VEROUTEDEV=$BRIDGEDEV
$brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $VEIFNAME link in VE$VEID
$vzctl exec $VEID $ip link set $VEIFNAME up

for IP in $VETH_IP_ADDRESS; do
echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
$vzctl exec $VEID $ip address add $IP dev $VEIFNAME

# removing the netmask
IP_STRIP=${IP%%/*};

echo "Adding a route from VE0 to VE$VEID."
$ip route add $IP_STRIP dev $VEROUTEDEV
done

if [ -n "$VE0_IP" ]; then
echo "Adding a route from VE$VEID to VE0."
$vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
$vzctl exec $VEID \
$ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
fi

exit 0
</pre>

==== Make the script to be run on a VE start ====
In order to run above script on a VE start create the file <code>/etc/vz/vznet.conf</code> with the following contents:

EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"

{{Note|<code>/usr/sbin/vznetcfg.custom</code> should be executable.(chmod +x /usr/sbin/vznetcfg.custom)}}

==== Setting the route VE → HN ====
To set up a route from the VE to the HN, the custom script has to get a HN IP (the $VE0_IP variable in the script). There are several ways to specify it:

# Add an entry VE0_IP="VE0 IP" to the <code>$VEID.conf</code>
# Add an entry VE0_IP="VE0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
# Implement some smart algorithm to determine the VE0 IP right in the custom network configuration script

Each variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).

== An OpenVZ Hardware Node has two Ethernet interfaces ==
Assuming you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from external traffic.
Let's assign eth0 for the external traffic and eth1 for the local one.

If there is no need to make the VE accessible from the HN and vice versa, it's enough to replace 'br0' and 'eth0' with 'br1' and 'eth1' in the following steps of the above configuration:
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]

It is nesessary to set a local IP for 'br0' to ensure VE ↔ HN connection availability.

== Putting VEs to different subnetworks ==
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_VE.27s_configuration|above configuration]].

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

[[Category: HOWTO]]
[[Category: Networking]]

Using gdb

2007-04-13T11:30:47Z

SergeyIvanov: using gdb or valgrind for threaded programs: upgrade kernel

If you have problems with using gdb or valgrind with threaded programs (like that of http://bugzilla.openvz.org/show_bug.cgi?id=487), just upgrade to ovzkernel-2.6.18.

[[Category: Troubleshooting]]