Since Virtuozzo 7 kernel vzkernel-3.10.0-327.18.2.vz7.14.7 it is possible to run Docker inside containers. This article describes how.
(This page is applicable for Virtuozzo 7.)

== Prerequisites ==

* Kernel 3.10.0-327.18.2.vz7.14.7 or later version
* Kernel modules '''veth''' and '''overlay''' loaded on host

Note: if you use kernel >= 3.10.0-327.18.2.vz7.14.25, you need to allow using "overlayfs" inside a Virtuozzo Container:
echo 1 > /proc/sys/fs/experimental_fs_enable

== Container tuning ==

* Turn on bridge feature to allow docker creating bridged network:
vzctl set $veid --features bridge:on --save
* Setup Container veth-based network (Container must be '''veth'''-based, not '''venet'''-based):
vzctl set $veid --netif_add eth0 --save
* Allow all iptables modules to be used in containers:
vzctl set $veid --netfilter full --save

== Limitations ==

* Only '''overlay''' and '''vfs''' Docker graph drivers are currently supported
* [[Checkpointing and live migration]] of a container with Docker containers inside is not supported yet (to be done)

== See also ==
* [[Docker inside CT]] page for running Docker Containers inside OpenVZ 6 Containers

[[Category:HOWTO]]
[[Category: TRD]]

2016-02-06T09:55:31Z

Finist:

This article describes how to build an kernel module which is not included into the stock Virtuozzo kernel.
(This article applies to Virtuozzo 7.)

== Building a kernel module (*.ko) ==

Here is an example how to build "via-rhine" kernel module which is in the Virtuozzo kernel source tree, but not enabled in kernel config by default.

// You need to install some dev packages in advance (the list here may be incomplete).
'''# yum install rpm-build gcc xmlto asciidoc hmaccalc python-devel newt-devel pesign'''

// If you are going to build a kernel module against some kernel, you need kernel headers for that kernel.
// Assume you want to build a kernel module against currently running kernel.
'''# yum install vzkernel-devel.x86_64'''

// Get sources of the module you'd like to build,
// in this particular example the easiest way i believe is just to download the kernel src.rpm.
'''# cd /tmp'''
'''# wget https://download.openvz.org/virtuozzo/factory/source/SRPMS/v/vzkernel-3.10.0-327.3.1.vz7.10.10.src.rpm'''
'''# rpm -ihv vzkernel-3.10.0-327.3.1.vz7.10.10.src.rpm'''

// "Prepare" source tree, it's not enough just to take the archive stored in it,
// you need to apply additional patch(es), rpmbuild does this for us.
'''# rpmbuild -bp /root/rpmbuild/SPECS/kernel.spec --nodeps'''

// Go to the module source directory.
'''# cd /root/rpmbuild/BUILD/kernel-3.10.0-327.3.1.el7/linux-3.10.0-327.3.1.vz7.10.10/drivers/net/ethernet/via'''

// Edit the Makefile so you get the required kernel module compiled.
// In this particular example the via-rhine compiles in-kernel by default, so we need to force it to be built as a module.
'''# sed -ie 's/$(CONFIG_VIA_RHINE)/m/' Makefile'''

// Build and install the module.
'''# make -C /lib/modules/`uname -r`/build M=$PWD'''
'''# make -C /lib/modules/`uname -r`/build M=$PWD modules_install'''

// Check the module has been really copied and load it.
'''# find /lib/modules -name \*rhine\*'''
/lib/modules/3.10.0-327.3.1.vz7.10.10/extra/via-rhine.ko

'''# modprobe via-rhine'''
'''# lsmod |grep rhine'''
via_rhine 32501 0
mii 13934 1 via_rhine

Here you are!

{{Note|Your case is a bit more complicated? Read [https://www.kernel.org/doc/Documentation/kbuild/modules.txt Building External Modules]}}

== Building a kernel module using Dynamic Kernel Module Support (DKMS) ==
TBD, you are welcome to put the description here. :)

== Building a kernel module rpm package (kmod) ==
TBD, you are welcome to put the description here. :)

[[Category: HOWTO]]
[[Category: Kernel]]
[[Category: Installation]]

Building external kernel modules

2016-02-06T09:54:46Z

Finist: Created page with "This article describes how to build an kernel module which is not included into the stock Virtuozzo kernel. (This article applies to Virtuozzo 7.) == Building a kernel module..."

Kernel TODO

2016-02-04T16:34:44Z

Finist:

Download/kernel/rhel7-testing

2016-01-19T10:41:13Z

Finist: /* Plans */

{{Virtuozzo}}

We are currently porting the OpenVZ patchset to the RHEL7 kernel. Test kernel builds are available in [http://download.openvz.org/virtuozzo/releases/ our repositories].

You can monitor the development progress by looking into [https://src.openvz.org/projects/OVZ/repos/vzkernel/commits RHEL7 source repository] and/or the [http://lists.openvz.org/pipermail/devel/ devel@ mailing list archives].

== Plans ==

[[Image:Rhel7-kernel-plans.png|1000px]]

* [http://lists.openvz.org/pipermail/devel/2015-September/033081.html Virtuozzo 7 kernel branches and plans 20150908]
* [http://lists.openvz.org/pipermail/devel/2015-July/032692.html Virtuozzo 7 kernel branches and plans 20150731]
* [http://lists.openvz.org/pipermail/devel/2016-January/067684.html Virtuozzo 7 kernel branches and plans 20160119]

== Contribute ==

If you want to contribute to kernel development, see [[Kernel patches]] document.

== External links ==

* RHEL7 based kernel git repo: https://src.openvz.org/projects/OVZ/repos/vzkernel/commits
* devel@ mailing list subscription: http://lists.openvz.org/mailman/listinfo/devel/
* devel@ mailing list archives: http://lists.openvz.org/pipermail/devel/
* [[Packages|Virtuozzo kernel in Linux distributions]]

[[Category:Virtuozzo]]

File:Rhel7-kernel-plans.png

2016-01-19T10:28:52Z

Finist: Finist uploaded a new version of File:Rhel7-kernel-plans.png

Kernel TODO

2015-11-16T12:00:04Z

2014-12-25T13:26:27Z

Finist:

For IPsec to work inside a container:
* Kernel 042stab084.8 or later
* The following kernel modules must be loaded before container start:
: <code>af_key esp4 esp6 xfrm4_mode_tunnel xfrm6_mode_tunnel</code>
* Capability <code>net_admin</code> must be granted to a container

Tested with libreswan.

Limitations:
* online migration on a Container with IPsec inside - does not work

[[Category: HOWTO]]
[[Category: Networking]]

Using private IPs for Hardware Nodes

2008-02-11T04:54:17Z

Finist: Undo revision 4122 by SergeyIvanov (Talk)

This article describes how to assign public IPs to VEs running on OVZ Hardware Nodes in case you have a following network topology:

[[Image:PrivateIPs_fig1.gif|An initial network topology]]

== Prerequisites ==
This configuration was tested on a RHEL5 OpenVZ Hardware Node and a VE based on a Fedora Core 5 template.
Other host OSs and templates might require some configuration changes, please add corresponding OS specific changes if you've faced any.

This article assumes the presence of 'brctl', 'ip' and 'ifconfig' utils. You may need to install missing packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.

This article assumes you have already [[Quick installation|installed OpenVZ]], prepared the [[OS template cache]](s) and have [[Basic_operations_in_OpenVZ_environment|VE(s) created]]. If not, follow the links to perform the steps needed.
{{Note|don't assign an IP after VE creation.}}

== An OVZ Hardware Node has the only one Ethernet interface ==
(assume eth0)

=== Hardware Node configuration ===

==== Create a bridge device ====
[HN]# brctl addbr br0

==== Remove an IP from eth0 interface ====
[HN]# ifconfig eth0 0

==== Add eth0 interface into the bridge ====
[HN]# brctl addif br0 eth0

==== Assign the IP to the bridge ====
(the same that was assigned on eth0 earlier)
[HN]# ifconfig br0 10.0.0.2/24

==== Resurrect the default routing ====
[HN]# ip route add default via 10.0.0.1 dev br0

{{Warning|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}

==== A script example ====
<pre>
[HN]# cat /tmp/br_add
#!/bin/bash

brctl addbr br0
ifconfig eth0 0
brctl addif br0 eth0
ifconfig br0 10.0.0.2/24
ip route add default via 10.0.0.1 dev br0
</pre>

[HN]# /tmp/br_add >/dev/null 2>&1 &

=== VE configuration ===

==== Start a VE ====
[HN]# vzctl start 101

==== Add a [[Virtual_Ethernet_device|veth interface]] to the VE ====
[HN]# vzctl set 101 --netif_add eth0 --save

==== Set up an IP to the newly created VE's veth interface ====
[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26

==== Add the VE's veth interface to the bridge ====
[HN]# brctl addif br0 veth101.0

{{Note|There will be a delay of about 15 seconds(default for 2.6.18 kernel) while the bridge software runs STP to detect loops and transitions the veth interface to the forwarding state.
}}

==== Set up the default route for the VE ====
[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0

==== (Optional) Add VE↔HN routes ====
The above configuration provides the following connections:
* VE X ↔ VE Y (where VE X and VE Y can locate on any OVZ HN)
* VE ↔ Internet

Note that

* The accessability of the VE from the HN depends on the local gateway providing NAT(probably - yes)

* The accessability of the HN from the VE depends on the ISP gateway being aware of the local network(probably not)

So to provide VE ↔ HN accessibility despite the gateways' configuration you can add the following routes:

[HN]# ip route add 85.86.87.195 dev br0
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0

=== Resulting OpenVZ Node configuration ===
[[Image:PrivateIPs_fig2.gif|Resulting OpenVZ Node configuration]]

=== Making the configuration persistent ===

==== Set up a bridge on a HN ====
This can be done by configuring the <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.

Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
<pre>
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

To automatically create bridge <code>br0</code> you can create <code>ifcfg-br0</code>:
<pre>
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

and edit <code>ifcfg-eth0</code> to add the <code>eth0</code> interface into the bridge <code>br0</code>:
<pre>
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
</pre>

==== Edit the VE's configuration ====
Add these parameters to the <code>/etc/vz/conf/$VEID.conf</code> file which will be used during the network configuration:
* Add/change <code>CONFIG_CUSTOMIZED="yes"</code> (indicates that a custom script should be run on a VE start)
* Add <code>VETH_IP_ADDRESS="VE IP/MASK"</code> (a VE can have multiple IPs separated by spaces)
* Add <code>VE_DEFAULT_GATEWAY="VE DEFAULT GATEWAY"</code>
* Add <code>BRIDGEDEV="BRIDGE NAME"</code> (a bridge name to which the VE veth interface should be added)

An example:
<pre>
# Network customization section
CONFIG_CUSTOMIZED="yes"
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"
</pre>

==== Create a custom network configuration script ====
which should be called each time a VE is started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
<pre>
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a VE

GLOBALCONFIGFILE=/etc/vz/vz.conf
VECONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
brctl=/usr/sbin/brctl
ip=/sbin/ip
ifconfig=/sbin/ifconfig
. $GLOBALCONFIGFILE
. $VECONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
# getting 'ifname' parameter value
if [[ "$str" =~ "^ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VEIFNAME=${str#*=};
fi
# getting 'host_ifname' parameter value
if [[ "$str" =~ "^host_ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VZHOSTIF=${str#*=};
fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VEIFNAME" ]; then
echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
exit 1
fi

echo "Initializing interface $VZHOSTIF for VE$VEID."
$ifconfig $VZHOSTIF 0

VEROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
VEROUTEDEV=$BRIDGEDEV
$brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $VEIFNAME link in VE$VEID
$vzctl exec $VEID $ip link set $VEIFNAME up

for IP in $VETH_IP_ADDRESS; do
echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
$vzctl exec $VEID $ip address add $IP dev $VEIFNAME

# removing the netmask
IP_STRIP=${IP%%/*};

echo "Adding a route from VE0 to VE$VEID."
$ip route add $IP_STRIP dev $VEROUTEDEV
done

if [ -n "$VE0_IP" ]; then
echo "Adding a route from VE$VEID to VE0."
$vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
$vzctl exec $VEID \
$ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
fi

exit 0
</pre>

==== Make the script to be run on a VE start ====
In order to run above script on a VE start create the file <code>/etc/vz/vznet.conf</code> with the following contents:

EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"

{{Note|<code>/usr/sbin/vznetcfg.custom</code> should be executable.(chmod +x /usr/sbin/vznetcfg.custom)}}

==== Setting the route VE → HN ====
To set up a route from the VE to the HN, the custom script has to get a HN IP (the $VE0_IP variable in the script). There are several ways to specify it:

# Add an entry VE0_IP="VE0 IP" to the <code>$VEID.conf</code>
# Add an entry VE0_IP="VE0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
# Implement some smart algorithm to determine the VE0 IP right in the custom network configuration script

Each variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).

== An OpenVZ Hardware Node has two Ethernet interfaces ==
Assuming you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from external traffic.
Let's assign eth0 for the external traffic and eth1 for the local one.

If there is no need to make the VE accessible from the HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of the above configuration:
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration → [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]

It is nesessary to set a local IP for 'br0' to ensure VE ↔ HN connection availability.

== Putting VEs to different subnetworks ==
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_VE.27s_configuration|above configuration]].

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

[[Category: HOWTO]]
[[Category: Networking]]

User:SergeyIvanov

2008-02-11T04:52:36Z

Finist: New page: ==Using private IPs for Hardware Nodes changes?== Sergey, could you please explain the suggested changes for this page? Why adding another bridge b...

==[[Using_private_IPs_for_Hardware_Nodes|Using private IPs for Hardware Nodes]] changes?==
Sergey, could you please explain the suggested changes for this page? Why adding another bridge br1 is necessary? And if even "yes" - it should be created somewhere but no command for creation br1 in the change? Please correct me if I'm wrong here, but so far I have rolled back your addition to [[Using_private_IPs_for_Hardware_Nodes|Using private IPs for Hardware Nodes]]. --[[User:Finist|Finist]] 07:54, 11 February 2008 (MSK)

User:Finist

2008-02-11T04:34:15Z

Finist: New page: Konstantin Khorenko khorenko "at" openvz "." org

Konstantin Khorenko
 
khorenko "at" openvz "." org

Asterisk from source

2008-01-14T12:18:33Z

Finist: Asterisk tuning moved to Asterisk from source: obvious

==General==
Asterisk is free and open source code to create software PBX server. See [http://www.asterisk.org] for details. This package perfectly runs inside OpenVZ container. Some users run up to 60 containers with Asterisk deployed for production per single hardware node. Although the easiest way to install Asterisk into container is to use pre-build package from Linux distribution, occasionally one may need to have a possibility to build it from source tarball available on developer's site.

In order to do it the following remarks are worth reading:

==Building Asterisk in CT==
Asterisk PBX server itself is compiled out of the shelf in CT provided that develop application template is installed on this CT. The functionality of the resulting executable is enough to support simple VoIP telephony.

<pre>
wget http://downloads.digium.com/pub/asterisk/releases/asterisk-x.x.xx.tar.gz
tar xzf asterisk-x.x.xx.tar.gz
cd asterisk-x.x.xx.tar.gz
./configure
make
make install
make samples
</pre>

The last command is not needed in case you have your own configuration as it installs some sample configuration files.
To configure Asterisk itself see for example [http://www.digium.com/elqNow/elqRedir.htm?ref=http://downloads.oreilly.com/books/9780596510480.pdf].

==MeetMe problem==
Unfortunately, one particular module called MeetMe (conferencing tool) will be switched off from compilation. This happens due to external dependency on 'zaptel' package. Zaptel provides support for some hardware cards for FXO/FXS analog telephony marketed by Digium (the company behind Asterisk), and on top of that supplies so called ztdummy kernel module. Ztdummy works like a simple metronome which is required to synchronize multiple sound streams in case of conference call.

If you do not plan to use analog telephone lines, hence don't like in install the hardware, nothing is lost provided you run your HN with 2.6.XX kernel. You just need to play a little trick with Asterisk make system: download zaptel tarball from the same location as Asterisk itself, and copy it's header zaptel.h to location /usr/include/zaptel/zaptel.h in CT where you plan to build Asterisk. This tweaks MeetMe for installation.

==HN configuration==
Finally you need to make sure that on HN ztdummy kernel module is loaded and the access to /dev/zap/pseudo device file is granted to
CT:
<pre>
modprobe ztdummy
vzctl set 240 --devnodes zap/pseudo:rw --save
</pre>

[[Category:HOWTO]]

Using private IPs for Hardware Nodes

2007-11-30T17:03:49Z

Finist: venet -> veth misprint

Quick installation (legacy)

2007-11-21T12:33:10Z

Finist: Make link on the first usage of "HN".

This document briefly describes the steps needed to install OpenVZ on your machine.

This document is also available in the following languages: [http://forum.openvz.org/index.php?t=tree&goto=35&#msg_35 French], [http://forum.openvz.org/index.php?t=tree&goto=1805&#msg_1805 German],
[http://wiki.openvz.jp Japanese],
[[Quick_installation_(Spanish)|Spanish]].

OpenVZ consists of a kernel, user-level tools, and VE templates. This guide tells how to install the kernel and the tools.

== Requirements ==
This guide assumes you are running recent release of Fedora Core (like FC5) or RHEL/CentOS 4. Currently, OpenVZ kernel tries to support the same hardware that Red Hat kernels support. For full hardware compatibility list, see [http://www.swsoft.com/en/products/virtuozzo/hcl/ Virtuozzo HCL].

=== Filesystems ===
It is recommended to use a separate partition for VEs private directories (by default /vz/private/<veid>). The reason why you should do so is that if you wish to use OpenVZ per-VE disk quota, you won't be able to use usual Linux disk quotas on the same partition. Bear in mind, that per-VE quota in this context includes not only pure per-VE quota, but also usual Linux disk quota used in VE, not on [[HN]].

At least try to avoid using root partition for VEs, because the root user of VE will be able to overcome 5% disk space barrier in some situations. This way HN root partition can be completely filled and it will break the system.

OpenVZ per-VE disk quota is supported only for ext2/ext3 filesystems. So use one of these filesystems (ext3 is recommended) if you need per-VE disk quota.

=== rpm or yum? ===

In case you have yum utility available on your system, you may want to use it effectively to install and update OpenVZ packages. In case you don't have yum, or don't want to use it, you can use plain old rpm. Instructions for both rpm and yum are provided below.

=== yum pre-setup ===
If you want to use yum, you should set up OpenVZ yum repository first.

Download [http://download.openvz.org/openvz.repo openvz.repo] file and put it to your <code>/etc/yum.repos.d/</code> repository. This can be achieved by the following commands, as root:
<pre>
# cd /etc/yum.repos.d
# wget http://download.openvz.org/openvz.repo
# rpm --import http://download.openvz.org/RPM-GPG-Key-OpenVZ
</pre>

In case you can not cd to /etc/yum.repos.d, it means either yum is not installed on your system, or yum version is too old. In that case, just stick to rpm installation method.

== Kernel installation ==

{{Note|In case you want to recompile the kernel yourself rather than use the one provided by OpenVZ, see [[kernel build]].}}

First, you need to choose what “flavor” of the kernel you want to install. Please refer to [[Kernel flavors]] for more information.

=== Using yum ===
Run the following command
<pre>
# yum install ovzkernel[-flavor]
</pre>

Here <code>[-flavor]</code> is optional, and can be <code>-smp</code> or <code>-enterprise</code>. Refer to [[kernel flavors]] for more info.

=== Using rpm ===
Get the kernel binary RPM from the [http://openvz.org/download/kernel/ Download » Kernel] page, or directly from [http://download.openvz.org/kernel/ download.openvz.org/kernel], or from one of its [[Download mirrors|mirrors]]. You need only one kernel RPM so please [[Kernel flavors|choose the appropriate one]] depending on your hardware.

Next, install the kernel RPM you chose:

<pre>
# rpm -ihv ovzkernel[-flavor]*.rpm
</pre>

Here <code>[-flavor]</code> is optional, and can be <code>-smp</code> or <code>-enterprise</code>. Refer to [[kernel flavors]] for more info.

{{Note|<tt>rpm -U</tt> (where <tt>-U</tt> stands for ''upgrade'') should '''not''' be used, otherwise all currently installed kernels will be uninstalled.}}

== Configuring the bootloader ==

In case GRUB is used as the boot loader, it will be configured automatically: lines similar to these will be added to the <tt>/boot/grub/grub.conf</tt> file:

<pre>
title Fedora Core (2.6.8-022stab029.1)
root (hd0,0)
kernel /vmlinuz-2.6.8-022stab029.1 ro root=/dev/sda5 quiet rhgb vga=0x31B
initrd /initrd-2.6.8-022stab029.1.img
</pre>
Change <tt>Fedora Core</tt> to <tt>OpenVZ</tt> (just for clarity reasons, so the OpenVZ kernels will not be mixed up with non-OpenVZ ones). Remove extra arguments from the kernel line, leaving only the <tt>root=...</tt> parameter. The modifed portion of <tt>/etc/grub.conf</tt> should look like this:

<pre>
title OpenVZ (2.6.8-022stab029.1)
root (hd0,0)
kernel /vmlinuz-2.6.8-022stab029.1 ro root=/dev/sda5
initrd /initrd-2.6.8-022stab029.1.img
</pre>

== Configuring ==

Please make sure the following steps are performed before rebooting into OpenVZ kernel.

=== sysctl ===

There are a number of kernel parameters that should be set for OpenVZ to work correctly. These parameters are stored in <tt>/etc/sysctl.conf</tt> file. Here is the relevant part of the file; please edit it accordingly.

<pre>
# On Hardware Node we generally need
# packet forwarding enabled and proxy arp disabled
net.ipv4.ip_forward = 1
net.ipv4.conf.default.proxy_arp = 0
# Enables source route verification
net.ipv4.conf.all.rp_filter = 1
# Enables the magic-sysrq key
kernel.sysrq = 1
# TCP Explict Congestion Notification
#net.ipv4.tcp_ecn = 0
# we do not want all our interfaces to send redirects
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
</pre>

=== SELinux ===

SELinux should be disabled. To that effect, put the following line to <code>/etc/sysconfig/selinux</code>:
<pre>
SELINUX=disabled
</pre>

=== Conntracks ===

In the stable OpenVZ kernels (those that are 2.6.8-based) netfilter connection tracking for [[VE0]] is disabled by default. If you have a stateful firewall enabled on the host node (it is there by default) you should either disable it, or enable connection tracking for [[VE0]].

To enable conntracks for VE0, add the following line to <code>/etc/modprobe.conf</code> file:
<pre>
options ip_conntrack ip_conntrack_enable_ve0=1
</pre>

{{Note|In kernels later than 2.6.8, connection tracking is enabled by default.}}

== Rebooting into OpenVZ kernel ==

Now reboot the machine and choose "OpenVZ" on the boot loader menu. If the OpenVZ kernel has been booted successfully, proceed to installing the user-level tools for OpenVZ.

== Installing the utilities ==

OpenVZ needs some user-level tools installed. Those are:

; vzctl
: A utility to control OpenVZ VPSs (create, destroy, start, stop, set parameters etc.)
; vzquota
: A utility to manage quotas for VPSs. Mostly used indirectly (by vzctl).

=== Using yum ===

<pre>
# yum install vzctl vzquota
</pre>

=== Using rpm ===

Download the binary RPMs of these utilities from [http://openvz.org/download/utils/ Download » Utils], or directly from [http://download.openvz.org/utils/ download.openvz.org/utils], or from one of its [[Download mirrors|mirrors]]. Install them:

<pre>
# rpm -Uhv vzctl*.rpm vzquota*.rpm
</pre>

If rpm complains about unresolved dependencies, you'll have to satisfy them first, then repeat the installation.

When all the tools are installed, start the OpenVZ subsystem.

== Starting OpenVZ ==

As root, execute the following command:

<pre>
# /sbin/service vz start
</pre>

This will load all the needed OpenVZ kernel modules. This script should also start all the VPSs marked to be auto-started on machine boot (there aren't any yet).

During the next reboot, this script should be executed automatically.

== Next steps ==

OpenVZ is now set up on your machine. To load OpenVZ kernel by default, edit the default line in the /boot/grub/grub.conf file to point to the OpenVZ kernel. For example, if the OpenVZ kernel is the first kernel mentioned in the file, put it as default 0. See man grub.conf for more details.

The next step is to prepare the [[OS template]]: please continue to [[OS template cache preparation]] document.

[[Category: Installation]]
[[Category: HOWTO]]

Using private IPs for Hardware Nodes

2007-09-13T13:57:39Z

Finist: /etc/vz/vznet.conf doesn't hasve to be executable

This article describes how to assign public IPs to VEs running on OVZ Hardware Nodes in case you have a following network topology:
[[Image:PrivateIPs_fig1.gif|An initial network topology]]

== Prerequisites ==
This configuration was tested on a RHEL5 OVZ Hardware Node and a VE based on a Fedora Core 5 template.
Other host OSes and templates might require some configuration changes, please, add corresponding OS specific changes if you've faced any. 
This article assumes the presence of 'brctl', 'ip', 'ifconfig' utils thus might require installation of missed packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.

This article assumes you have already [[Quick installation|installed OpenVZ]], prepared the [[OS template cache]](s) and have [[Basic_operations_in_OpenVZ_environment|VE(s) created]]. If not, follow the links to perform the steps needed.
{{Note|don't assign an IP after VE creation.}}
 

== (1) An OVZ Hardware Node has the only one ethernet interface ==
(assume eth0)

=== Hardware Node configuration ===

==== Create a bridge device ====
<pre>[HN]# brctl addbr br0</pre>

==== Remove an IP from eth0 interface ====
<pre>[HN]# ifconfig eth0 0</pre>

==== Add eth0 interface into the bridge ====
<pre>[HN]# brctl addif br0 eth0</pre>

==== Assign the IP to the bridge ====
(the same that was assigned on eth0 earlier)
<pre>[HN]# ifconfig br0 10.0.0.2/24</pre>

==== Resurrect the default routing ====
<pre>[HN]# ip route add default via 10.0.0.1 dev br0</pre>

{{Note|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}

==== A script example ====
<pre>
[HN]# cat /tmp/br_add
#!/bin/bash

brctl addbr br0
ifconfig eth0 0
brctl addif br0 eth0
ifconfig br0 10.0.0.2/24
ip route add default via 10.0.0.1 dev br0
</pre>

<pre>[HN]# /tmp/br_add >/dev/null 2>&1 &</pre>
 
=== VE configuration ===

==== Start a VE ====
<pre>[HN]# vzctl start 101</pre>

==== Add a [[Virtual_Ethernet_device|veth interface]] to the VE ====
<pre>[HN]# vzctl set 101 --netif_add eth0 --save</pre>

==== Set up an IP to the newly created VE's veth interface ====
<pre>[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26</pre>

==== Add the VE's veth interface to the bridge ====
<pre>[HN]# brctl addif br0 veth101.0</pre>
{{Note|veth interface will work as expected only after it is turned by the bridge into the forwarding state after some delay.
In 2.6.18 kernel it is 15 sec by default.
}}

==== Set up the default route for the VE ====
<pre>[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0</pre>

==== (Optional) Add routes VE <-> HN ====
The configuration above provides following connections available:
<pre>
VE X <-> VE Y (where VE X and VE Y can locate on any OVZ HN)
VE <-> Internet
</pre>
* A VE accessibility from the HN depends on if the local gateway provides NAT or not (probably - yes).
* A HN accessibility from a VE depends on if the ISP gateway is aware about the local network addresses (most probably - no).

So to provide VE <-> HN accessibility despite the gateways' configuration you can add following route rules:
<pre>
[HN]# ip route add 85.86.87.195 dev br0
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0
</pre>

=== The resulted OVZ Node configuration ===
[[Image:PrivateIPs_fig2.gif|The resulted OVZ Node configuration]]

=== Making the configuration persistent ===

==== Set up a bridge on a HN ====
This can be done by configuring <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.

Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
<pre>
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>
 
To make bridge <code>br0</code> automatically created you can create <code>ifcfg-br0</code>:
<pre>
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

and edit <code>ifcfg-eth0</code> file to add <code>eth0</code> interface into the bridge <code>br0</code>:
<pre>
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
</pre>

==== Edit the VE's configuration ====
Add some parameters to the <code>/etc/vz/conf/$VEID.conf</code> which will be used during the network configuration:
* Add/change CONFIG_CUSTOMIZED="yes" (indicates that a custom script should be run on a VE start)
* Add VETH_IP_ADDRESS="<VE IP>/<MASK>" (a VE can have multiple IPs separated by spaces)
* Add VE_DEFAULT_GATEWAY="<VE DEFAULT GATEWAY>"
* Add BRIDGEDEV="<BRIDGE NAME>" (a bridge name to which the VE veth interface should be added)

An example:
<pre>
# Network customization section
CONFIG_CUSTOMIZED="yes"
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"
</pre>

==== Create a custom network configuration script ====
which should be called each time a VE started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
<pre>
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a VE

GLOBALCONFIGFILE=/etc/vz/vz.conf
VECONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
brctl=/usr/sbin/brctl
ip=/sbin/ip
ifconfig=/sbin/ifconfig
. $GLOBALCONFIGFILE
. $VECONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
# getting 'ifname' parameter value
if [[ "$str" =~ "^ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VEIFNAME=${str#*=};
fi
# getting 'host_ifname' parameter value
if [[ "$str" =~ "^host_ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VZHOSTIF=${str#*=};
fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VEIFNAME" ]; then
echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
exit 1
fi

echo "Initializing interface $VZHOSTIF for VE$VEID."
$ifconfig $VZHOSTIF 0

VEROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
VEROUTEDEV=$BRIDGEDEV
$brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $VEIFNAME link in VE$VEID
$vzctl exec $VEID $ip link set $VEIFNAME up

for IP in $VETH_IP_ADDRESS; do
echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
$vzctl exec $VEID $ip address add $IP dev $VEIFNAME

# removing the netmask
IP_STRIP=${IP%%/*};

echo "Adding a route from VE0 to VE$VEID."
$ip route add $IP_STRIP dev $VEROUTEDEV
done

if [ -n "$VE0_IP" ]; then
echo "Adding a route from VE$VEID to VE0."
$vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
$vzctl exec $VEID \
$ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
fi

exit 0
</pre>

==== Make the script to be run on a VE start ====
In order to run above script on a VE start create the following <code>/etc/vz/vznet.conf</code> file:
<pre>
EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"
</pre>
{{Note|<code>/usr/sbin/vznetcfg.custom</code> should be executable.}}

==== Setting the route VE -> HN ====
To set up a route from VE to HN the custom script has to get a HN IP (the $VE0_IP variable in the script). There can be different approaches to specify it:
# Add an entry VE0_IP="VE0 IP" to the <code>$VEID.conf</code>
# Add an entry VE0_IP="VE0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
# Implement some smart algorithm to determine the VE0 IP right in the custom network configuration script
Every variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).

== (2) An OVZ Hardware Node has two ethernet interfaces ==
Assume you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from the external traffic.
Let's assign eth0 for the external traffic and eth1 for the local one.

If there is no aim to make VE accessible from HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of above configuration:
* Hardware Node configuration -> [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration -> [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]

For the VE <-> HN connections availability it's nesessary to set an IP (local) to the 'br0'.

== (3) Putting VEs to different subnetworks ==
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_VE.27s_configuration|above configuration]].

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

[[Category: HOWTO]]
[[Category: Networking]]

Using private IPs for Hardware Nodes

2007-09-13T13:49:36Z

Finist: Initialize a veth interface on HN only once.

This article describes how to assign public IPs to VEs running on OVZ Hardware Nodes in case you have a following network topology:
[[Image:PrivateIPs_fig1.gif|An initial network topology]]

== Prerequisites ==
This configuration was tested on a RHEL5 OVZ Hardware Node and a VE based on a Fedora Core 5 template.
Other host OSes and templates might require some configuration changes, please, add corresponding OS specific changes if you've faced any. 
This article assumes the presence of 'brctl', 'ip', 'ifconfig' utils thus might require installation of missed packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.

This article assumes you have already [[Quick installation|installed OpenVZ]], prepared the [[OS template cache]](s) and have [[Basic_operations_in_OpenVZ_environment|VE(s) created]]. If not, follow the links to perform the steps needed.
{{Note|don't assign an IP after VE creation.}}
 

== (1) An OVZ Hardware Node has the only one ethernet interface ==
(assume eth0)

=== Hardware Node configuration ===

==== Create a bridge device ====
<pre>[HN]# brctl addbr br0</pre>

==== Remove an IP from eth0 interface ====
<pre>[HN]# ifconfig eth0 0</pre>

==== Add eth0 interface into the bridge ====
<pre>[HN]# brctl addif br0 eth0</pre>

==== Assign the IP to the bridge ====
(the same that was assigned on eth0 earlier)
<pre>[HN]# ifconfig br0 10.0.0.2/24</pre>

==== Resurrect the default routing ====
<pre>[HN]# ip route add default via 10.0.0.1 dev br0</pre>

{{Note|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}

==== A script example ====
<pre>
[HN]# cat /tmp/br_add
#!/bin/bash

brctl addbr br0
ifconfig eth0 0
brctl addif br0 eth0
ifconfig br0 10.0.0.2/24
ip route add default via 10.0.0.1 dev br0
</pre>

<pre>[HN]# /tmp/br_add >/dev/null 2>&1 &</pre>
 
=== VE configuration ===

==== Start a VE ====
<pre>[HN]# vzctl start 101</pre>

==== Add a [[Virtual_Ethernet_device|veth interface]] to the VE ====
<pre>[HN]# vzctl set 101 --netif_add eth0 --save</pre>

==== Set up an IP to the newly created VE's veth interface ====
<pre>[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26</pre>

==== Add the VE's veth interface to the bridge ====
<pre>[HN]# brctl addif br0 veth101.0</pre>
{{Note|veth interface will work as expected only after it is turned by the bridge into the forwarding state after some delay.
In 2.6.18 kernel it is 15 sec by default.
}}

==== Set up the default route for the VE ====
<pre>[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0</pre>

==== (Optional) Add routes VE <-> HN ====
The configuration above provides following connections available:
<pre>
VE X <-> VE Y (where VE X and VE Y can locate on any OVZ HN)
VE <-> Internet
</pre>
* A VE accessibility from the HN depends on if the local gateway provides NAT or not (probably - yes).
* A HN accessibility from a VE depends on if the ISP gateway is aware about the local network addresses (most probably - no).

So to provide VE <-> HN accessibility despite the gateways' configuration you can add following route rules:
<pre>
[HN]# ip route add 85.86.87.195 dev br0
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0
</pre>

=== The resulted OVZ Node configuration ===
[[Image:PrivateIPs_fig2.gif|The resulted OVZ Node configuration]]

=== Making the configuration persistent ===

==== Set up a bridge on a HN ====
This can be done by configuring <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.

Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
<pre>
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>
 
To make bridge <code>br0</code> automatically created you can create <code>ifcfg-br0</code>:
<pre>
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

and edit <code>ifcfg-eth0</code> file to add <code>eth0</code> interface into the bridge <code>br0</code>:
<pre>
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
</pre>

==== Edit the VE's configuration ====
Add some parameters to the <code>/etc/vz/conf/$VEID.conf</code> which will be used during the network configuration:
* Add/change CONFIG_CUSTOMIZED="yes" (indicates that a custom script should be run on a VE start)
* Add VETH_IP_ADDRESS="<VE IP>/<MASK>" (a VE can have multiple IPs separated by spaces)
* Add VE_DEFAULT_GATEWAY="<VE DEFAULT GATEWAY>"
* Add BRIDGEDEV="<BRIDGE NAME>" (a bridge name to which the VE veth interface should be added)

An example:
<pre>
# Network customization section
CONFIG_CUSTOMIZED="yes"
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"
</pre>

==== Create a custom network configuration script ====
which should be called each time a VE started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
<pre>
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a VE

GLOBALCONFIGFILE=/etc/vz/vz.conf
VECONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
brctl=/usr/sbin/brctl
ip=/sbin/ip
ifconfig=/sbin/ifconfig
. $GLOBALCONFIGFILE
. $VECONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
# getting 'ifname' parameter value
if [[ "$str" =~ "^ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VEIFNAME=${str#*=};
fi
# getting 'host_ifname' parameter value
if [[ "$str" =~ "^host_ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VZHOSTIF=${str#*=};
fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VEIFNAME" ]; then
echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
exit 1
fi

echo "Initializing interface $VZHOSTIF for VE$VEID."
$ifconfig $VZHOSTIF 0

VEROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
VEROUTEDEV=$BRIDGEDEV
$brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $VEIFNAME link in VE$VEID
$vzctl exec $VEID $ip link set $VEIFNAME up

for IP in $VETH_IP_ADDRESS; do
echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
$vzctl exec $VEID $ip address add $IP dev $VEIFNAME

# removing the netmask
IP_STRIP=${IP%%/*};

echo "Adding a route from VE0 to VE$VEID."
$ip route add $IP_STRIP dev $VEROUTEDEV
done

if [ -n "$VE0_IP" ]; then
echo "Adding a route from VE$VEID to VE0."
$vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
$vzctl exec $VEID \
$ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
fi

exit 0
</pre>

==== Make the script to be run on a VE start ====
In order to run above script on a VE start create the following <code>/etc/vz/vznet.conf</code> file:
<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"
</pre>
{{Note|both <code>/etc/vz/vznet.conf</code> and <code>/usr/sbin/vznetcfg.custom</code> should be executable files.}}

==== Setting the route VE -> HN ====
To set up a route from VE to HN the custom script has to get a HN IP (the $VE0_IP variable in the script). There can be different approaches to specify it:
# Add an entry VE0_IP="VE0 IP" to the <code>$VEID.conf</code>
# Add an entry VE0_IP="VE0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
# Implement some smart algorithm to determine the VE0 IP right in the custom network configuration script
Every variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).

== (2) An OVZ Hardware Node has two ethernet interfaces ==
Assume you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from the external traffic.
Let's assign eth0 for the external traffic and eth1 for the local one.

If there is no aim to make VE accessible from HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of above configuration:
* Hardware Node configuration -> [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration -> [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]

For the VE <-> HN connections availability it's nesessary to set an IP (local) to the 'br0'.

== (3) Putting VEs to different subnetworks ==
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_VE.27s_configuration|above configuration]].

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

[[Category: HOWTO]]
[[Category: Networking]]

Using private IPs for Hardware Nodes

2007-09-13T12:36:58Z

Finist: A device newly added to the bridge starts working only after some delay.

This article describes how to assign public IPs to VEs running on OVZ Hardware Nodes in case you have a following network topology:
[[Image:PrivateIPs_fig1.gif|An initial network topology]]

== Prerequisites ==
This configuration was tested on a RHEL5 OVZ Hardware Node and a VE based on a Fedora Core 5 template.
Other host OSes and templates might require some configuration changes, please, add corresponding OS specific changes if you've faced any. 
This article assumes the presence of 'brctl', 'ip', 'ifconfig' utils thus might require installation of missed packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.

This article assumes you have already [[Quick installation|installed OpenVZ]], prepared the [[OS template cache]](s) and have [[Basic_operations_in_OpenVZ_environment|VE(s) created]]. If not, follow the links to perform the steps needed.
{{Note|don't assign an IP after VE creation.}}
 

== (1) An OVZ Hardware Node has the only one ethernet interface ==
(assume eth0)

=== Hardware Node configuration ===

==== Create a bridge device ====
<pre>[HN]# brctl addbr br0</pre>

==== Remove an IP from eth0 interface ====
<pre>[HN]# ifconfig eth0 0</pre>

==== Add eth0 interface into the bridge ====
<pre>[HN]# brctl addif br0 eth0</pre>

==== Assign the IP to the bridge ====
(the same that was assigned on eth0 earlier)
<pre>[HN]# ifconfig br0 10.0.0.2/24</pre>

==== Resurrect the default routing ====
<pre>[HN]# ip route add default via 10.0.0.1 dev br0</pre>

{{Note|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}

==== A script example ====
<pre>
[HN]# cat /tmp/br_add
#!/bin/bash

brctl addbr br0
ifconfig eth0 0
brctl addif br0 eth0
ifconfig br0 10.0.0.2/24
ip route add default via 10.0.0.1 dev br0
</pre>

<pre>[HN]# /tmp/br_add >/dev/null 2>&1 &</pre>
 
=== VE configuration ===

==== Start a VE ====
<pre>[HN]# vzctl start 101</pre>

==== Add a [[Virtual_Ethernet_device|veth interface]] to the VE ====
<pre>[HN]# vzctl set 101 --netif_add eth0 --save</pre>

==== Set up an IP to the newly created VE's veth interface ====
<pre>[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26</pre>

==== Add the VE's veth interface to the bridge ====
<pre>[HN]# brctl addif br0 veth101.0</pre>
{{Note|veth interface will work as expected only after it is turned by the bridge into the forwarding state after some delay.
In 2.6.18 kernel it is 15 sec by default.
}}

==== Set up the default route for the VE ====
<pre>[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0</pre>

==== (Optional) Add routes VE <-> HN ====
The configuration above provides following connections available:
<pre>
VE X <-> VE Y (where VE X and VE Y can locate on any OVZ HN)
VE <-> Internet
</pre>
* A VE accessibility from the HN depends on if the local gateway provides NAT or not (probably - yes).
* A HN accessibility from a VE depends on if the ISP gateway is aware about the local network addresses (most probably - no).

So to provide VE <-> HN accessibility despite the gateways' configuration you can add following route rules:
<pre>
[HN]# ip route add 85.86.87.195 dev br0
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0
</pre>

=== The resulted OVZ Node configuration ===
[[Image:PrivateIPs_fig2.gif|The resulted OVZ Node configuration]]

=== Making the configuration persistent ===

==== Set up a bridge on a HN ====
This can be done by configuring <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.

Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
<pre>
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>
 
To make bridge <code>br0</code> automatically created you can create <code>ifcfg-br0</code>:
<pre>
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

and edit <code>ifcfg-eth0</code> file to add <code>eth0</code> interface into the bridge <code>br0</code>:
<pre>
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
</pre>

==== Edit the VE's configuration ====
Add some parameters to the <code>/etc/vz/conf/$VEID.conf</code> which will be used during the network configuration:
* Add/change CONFIG_CUSTOMIZED="yes" (indicates that a custom script should be run on a VE start)
* Add VETH_IP_ADDRESS="<VE IP>/<MASK>" (a VE can have multiple IPs separated by spaces)
* Add VE_DEFAULT_GATEWAY="<VE DEFAULT GATEWAY>"
* Add BRIDGEDEV="<BRIDGE NAME>" (a bridge name to which the VE veth interface should be added)

An example:
<pre>
# Network customization section
CONFIG_CUSTOMIZED="yes"
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"
</pre>

==== Create a custom network configuration script ====
which should be called each time a VE started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
<pre>
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a VE

GLOBALCONFIGFILE=/etc/vz/vz.conf
VECONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
ip=/sbin/ip
. $GLOBALCONFIGFILE
. $VECONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
# getting 'ifname' parameter value
if [[ "$str" =~ "^ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VEIFNAME=${str#*=};
fi
# getting 'host_ifname' parameter value
if [[ "$str" =~ "^host_ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VZHOSTIF=${str#*=};
fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VEIFNAME" ]; then
echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
exit 1
fi

for IP in $VETH_IP_ADDRESS; do
echo "Initializing interface $VZHOSTIF for VE$VEID."
/sbin/ifconfig $VZHOSTIF 0
done

VEROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
VEROUTEDEV=$BRIDGEDEV
/usr/sbin/brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $VEIFNAME link in VE$VEID
$vzctl exec $VEID $ip link set $VEIFNAME up

for IP in $VETH_IP_ADDRESS; do
echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
$vzctl exec $VEID $ip address add $IP dev $VEIFNAME

# removing the netmask
IP_STRIP=${IP%%/*};

echo "Adding a route from VE0 to VE$VEID."
$ip route add $IP_STRIP dev $VEROUTEDEV
done

if [ -n "$VE0_IP" ]; then
echo "Adding a route from VE$VEID to VE0."
$vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
$vzctl exec $VEID \
$ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
fi

exit 0
</pre>

==== Make the script to be run on a VE start ====
In order to run above script on a VE start create the following <code>/etc/vz/vznet.conf</code> file:
<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"
</pre>
{{Note|both <code>/etc/vz/vznet.conf</code> and <code>/usr/sbin/vznetcfg.custom</code> should be executable files.}}

==== Setting the route VE -> HN ====
To set up a route from VE to HN the custom script has to get a HN IP (the $VE0_IP variable in the script). There can be different approaches to specify it:
# Add an entry VE0_IP="VE0 IP" to the <code>$VEID.conf</code>
# Add an entry VE0_IP="VE0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
# Implement some smart algorithm to determine the VE0 IP right in the custom network configuration script
Every variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).

== (2) An OVZ Hardware Node has two ethernet interfaces ==
Assume you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from the external traffic.
Let's assign eth0 for the external traffic and eth1 for the local one.

If there is no aim to make VE accessible from HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of above configuration:
* Hardware Node configuration -> [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration -> [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]

For the VE <-> HN connections availability it's nesessary to set an IP (local) to the 'br0'.

== (3) Putting VEs to different subnetworks ==
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_VE.27s_configuration|above configuration]].

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

[[Category: HOWTO]]
[[Category: Networking]]

Virtual Ethernet device

2007-09-06T14:33:50Z

Finist: 'bridged'->'virtual': confusing definition of veth.

'''Virtual ethernet device''' is an ethernet-like device which can be used inside a [[VE]]. Unlike
[[venet]] network device, veth device has a MAC address. Due to this, it can be used in configurations, when veth is bridged to ethX or other device and VE user fully sets up his networking himself,
including IPs, gateways etc.

Virtual ethernet device consist of two ethernet devices - one in [[VE0]] and another one
in VE. These devices are connected to each other, so if a packet goes to one
device it will come out from the other device.

== Virtual ethernet device usage ==

=== Kernel module ===
First of all, make sure the <code>vzethdev</code> module is loaded:
<pre>
# lsmod | grep vzeth
vzethdev 8224 0
vzmon 35164 5 vzethdev,vznetdev,vzrst,vzcpt
vzdev 3080 4 vzethdev,vznetdev,vzmon,vzdquota
</pre>

In case it is not loaded, load it:
<pre>
# modprobe vzethdev
</pre>

You might want to add the module to <code>/etc/init.d/vz script</code>, so it will be loaded during startup.

{{Note|since vzctl version 3.0.11, vzethdev is loaded by /etc/init.d/vz}}

=== MAC addresses ===
In the below commands, you should use random MAC addresses. Do not use MAC addresses of real eth devices, because this can lead to collisions.

MAC addresses must be entered in XX:XX:XX:XX:XX:XX format.

There is a utility script available for generating MAC addresses: http://www.easyvmx.com/software/easymac.sh. It is to be used like this:

chmod +x easymac.sh
./easymac.sh -R

=== Adding veth to a VE ===

==== syntax vzctl version < 3.0.14 ====

<pre>
vzctl set <VEID> --veth_add <dev_name>,<dev_addr>,<ve_dev_name>,<ve_dev_addr>
</pre>

Here
* <tt>dev_name</tt> is the ethernet device name that you are creating on the [[VE0|host system]]
* <tt>dev_addr</tt> is its MAC address
* <tt>ve_dev_name</tt> is the corresponding ethernet device name you are creating on the VE
* <tt>ve_dev_addr</tt> is its MAC address

{{Note| that this option is incremental, so devices are added to already existing ones.}}

NB there are no spaces after the commas

Example:

<pre>
vzctl set 101 --veth_add veth101.0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save
</pre>
After executing this command <tt>veth</tt> device will be created for VE 101 and veth configuration will be saved to a VE configuration file.
Host-side ethernet device will have <tt>veth101.0</tt> name and <tt>00:12:34:56:78:9A</tt> MAC address.
VE-side ethernet device will have <tt>eth0</tt> name and <tt>00:12:34:56:78:9B</tt> MAC address.

==== syntax vzctl version >= 3.0.14 ====

Read Update infos about [http://openvz.org/news/updates/vzctl-3.0.14-1 vzctl 3.0.14]

<pre>
vzctl set <VEID> --netif_add <ifname>[,<mac>,<host_ifname>,<host_mac]
</pre>

Here
* <tt>ifname</tt> is the ethernet device name in the VE
* <tt>mac</tt> is its MAC address in the VE
* <tt>host_ifname</tt> is the ethernet device name on the host ([[VE0]])
* <tt>host_mac</tt> is its MAC address on the host ([[VE0]])

{{Note|All parameters except ifname are optional and are automatically generated if not specified.}}

Example:

<pre>
vzctl set 101 --netif_add eth0,00:12:34:56:78:9A,veth101.0,00:12:34:56:78:9B --save
</pre>

=== Removing veth from a VE ===

==== syntax vzctl version < 3.0.14 ====

<pre>
vzctl set <VEID> --veth_del <dev_name>
</pre>
Here <tt>dev_name</tt> is the ethernet device name in the [[VE0|host system]].

Example:
<pre>
vzctl set 101 --veth_del veth101.0 --save
</pre>
After executing this command veth device with host-side ethernet name veth101.0 will be removed from VE 101 and veth configuration will be updated in VE config file.

==== syntax vzctl version >= 3.0.14 ====
<pre>
vzctl set <VEID> --netif_del <dev_name>|all
</pre>

Here
* <code>dev_name</code> is the ethernet device name in the [[VE]].

{{Note|If you want to remove all ethernet devices in VE, use <code>all</code>.}}

Example:

<pre>
vzctl set 101 --netif_del eth0 --save
</pre>

== Common configurations with virtual ethernet devices ==
Module <tt>vzethdev</tt> must be loaded to operate with veth devices.

=== Simple configuration with virtual ethernet device ===

==== Start a VE ====
<pre>
[host-node]# vzctl start 101
</pre>

==== Add veth device to VE ====
<pre>
[host-node]# vzctl set 101 --veth_add veth101.0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save
</pre>

==== Configure devices in VE0 ====
<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/proxy_arp
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
</pre>

==== Configure device in VE ====
<pre>
[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0
[ve-101]# /sbin/ip addr add 192.168.0.101 dev eth0
[ve-101]# /sbin/ip route add default dev eth0
</pre>

==== Add route in [[VE0]] ====
<pre>
[host-node]# ip route add 192.168.0.101 dev veth101.0
</pre>

=== Virtual ethernet device with IPv6 ===

==== Start [[VE]] ====
<pre>
[host-node]# vzctl start 101
</pre>

==== Add veth device to [[VE]] ====
<pre>
[host-node]# vzctl set 101 --veth_add veth101.0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save
</pre>

==== Configure devices in [[VE0]] ====
<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
</pre>

==== Configure device in [[VE]] ====
<pre>
[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0
</pre>

==== Start router advertisement daemon (radvd) for IPv6 in VE0 ====
First you need to edit radvd configuration file. Here is a simple example of <tt>/etc/radv.conf</tt>:
<pre>
interface veth101.0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvHomeAgentFlag off;

prefix 3ffe:2400:0:0::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
};

interface eth0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvHomeAgentFlag off;

prefix 3ffe:0302:0011:0002::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
};
</pre>

Then, start radvd:
<pre>
[host-node]# /etc/init.d/radvd start
</pre>

==== Add IPv6 addresses to devices in [[VE0]] ====
<pre>
[host-node]# ip addr add dev veth101.0 3ffe:2400::212:34ff:fe56:789a/64
[host-node]# ip addr add dev eth0 3ffe:0302:0011:0002:211:22ff:fe33:4455/64
</pre>

=== Virtual ethernet devices can be joined in one bridge ===
Perform steps 1 - 4 from Simple configuration chapter for several VEs and/or veth devices

==== Create bridge device ====
<pre>
[host-node]# brctl addbr vzbr0
</pre>

==== Add veth devices to bridge ====
<pre>
[host-node]# brctl addif vzbr0 veth101.0
...
[host-node]# brctl addif vzbr0 veth101.n
[host-node]# brctl addif vzbr0 veth102.0
...
...
[host-node]# brctl addif vzbr0 vethXXX.N
</pre>

==== Configure bridge device ====
<pre>
[host-node]# ifconfig vzbr0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/vzbr0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/vzbr0/proxy_arp
</pre>

==== Add routes in [[VE0]] ====
<pre>
[host-node]# ip route add 192.168.101.1 dev vzbr0
...
[host-node]# ip route add 192.168.101.n dev vzbr0
[host-node]# ip route add 192.168.102.1 dev vzbr0
...
...
[host-node]# ip route add 192.168.XXX.N dev vzbr0
</pre>

Thus you'll have more convinient configuration, i.e. all routes to VEs will be through this bridge and VEs can communicate with each other even without these routes.

=== Making a veth-device persistent ===

At the moment, it is not possible to have the commands needed for a persistent veth being made automatically be vzctl. A bugreport ( http://bugzilla.openvz.org/show_bug.cgi?id=301 ) has already been made. Until then, here's a way to make the above steps persistent.

1. First, edit the VE's configuration to specify what the veth's IP address(es) should be, and to indicate that a custom script should be run when starting up a VE.
* Open up /etc/vz/conf/VEID.conf
* Comment out any IP_ADDRESS entries to prevent a VENET-device from being created in the VE
* Add or change the entry CONFIG_CUSTOMIZED="yes"
* Add an entry VETH_IP_ADDRESS="<VE IP>" The VE IP can have multiple IPs, separated by spaces

2. Now to create that "custom script". The following helper script will check the configuration file for IP addresses and for the veth interface, and configure the IP routing accordingly. Create the script /usr/sbin/vznetaddroute to have the following, and then <code>chmod 0500 /usr/sbin/vznetaddroute</code> to make it executable.

<pre>
#!/bin/bash
# /usr/sbin/vznetaddroute
# a script to bring up virtual network interfaces (veth's) in a VE

CONFIGFILE=/etc/vz/conf/$VEID.conf
. $CONFIGFILE
VZHOSTIF=`echo $NETIF |sed 's/^.*host_ifname=$.*$,.*$/\1/g'`

if [ ! -n "$VETH_IP_ADDRESS" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
exit 1
fi

for IP in $VETH_IP_ADDRESS; do
echo "Adding interface $VZHOSTIF and route $IP for VE$VEID to VE0"
/sbin/ifconfig $VZHOSTIF 0
echo 1 > /proc/sys/net/ipv4/conf/$VZHOSTIF/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/$VZHOSTIF/forwarding
/sbin/ip route add $IP dev $VZHOSTIF
done

exit 0
</pre>

3. Now create /etc/vz/vznet.conf containing the following. This is what defines the "custom script" as being the vznetaddroute which you just created.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetaddroute"
</pre>

4. Of course, the VE's operating system will need to be configured with those IP address(es) as well. Consult the manual for your VE's OS for details.

That's it! At this point, when you restart the VE you should see a new line in the output, indicating that the interface is being configured and a new route being added. And you should be able to ping the host, and to enter the VE and use the network.

=== Virtual ethernet devices + VLAN ===
This configuration can be done by adding vlan device to the previous configuration.

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

== External links ==
* [http://www.tldp.org/HOWTO/Linux+IPv6-HOWTO/hints-daemons-radvd.html Linux IPv6 HOWTO, a chapter about radvd]

[[Category: Networking]]
[[Category: HOWTO]]

Using private IPs for Hardware Nodes

2007-08-31T14:36:45Z

Finist: misprint eth0 -> eth1

This article describes how to assign public IPs to VEs running on OVZ Hardware Nodes in case you have a following network topology:
[[Image:PrivateIPs_fig1.gif|An initial network topology]]

== Prerequisites ==
This configuration was tested on a RHEL5 OVZ Hardware Node and a VE based on a Fedora Core 5 template.
Other host OSes and templates might require some configuration changes, please, add corresponding OS specific changes if you've faced any. 
This article assumes the presence of 'brctl', 'ip', 'ifconfig' utils thus might require installation of missed packages like 'bridge-utils'/'iproute'/'net-tools' or others which contain those utilities.

This article assumes you have already [[Quick installation|installed OpenVZ]], prepared the [[OS template cache]](s) and have [[Basic_operations_in_OpenVZ_environment|VE(s) created]]. If not, follow the links to perform the steps needed.
{{Note|don't assign an IP after VE creation.}}
 

== (1) An OVZ Hardware Node has the only one ethernet interface ==
(assume eth0)

=== Hardware Node configuration ===

==== Create a bridge device ====
<pre>[HN]# brctl addbr br0</pre>

==== Remove an IP from eth0 interface ====
<pre>[HN]# ifconfig eth0 0</pre>

==== Add eth0 interface into the bridge ====
<pre>[HN]# brctl addif br0 eth0</pre>

==== Assign the IP to the bridge ====
(the same that was assigned on eth0 earlier)
<pre>[HN]# ifconfig br0 10.0.0.2/24</pre>

==== Resurrect the default routing ====
<pre>[HN]# ip route add default via 10.0.0.1 dev br0</pre>

{{Note|if you are '''configuring''' the node '''remotely''' you '''must''' prepare a '''script''' with the above commands and run it in background with the redirected output or you'll '''lose the access''' to the Node.}}

==== A script example ====
<pre>
[HN]# cat /tmp/br_add
#!/bin/bash

brctl addbr br0
ifconfig eth0 0
brctl addif br0 eth0
ifconfig br0 10.0.0.2/24
ip route add default via 10.0.0.1 dev br0
</pre>

<pre>[HN]# /tmp/br_add >/dev/null 2>&1 &</pre>
 
=== VE configuration ===

==== Start a VE ====
<pre>[HN]# vzctl start 101</pre>

==== Add a [[Virtual_Ethernet_device|veth interface]] to the VE ====
<pre>[HN]# vzctl set 101 --netif_add eth0 --save</pre>

==== Set up an IP to the newly created VE's veth interface ====
<pre>[HN]# vzctl exec 101 ifconfig eth0 85.86.87.195/26</pre>

==== Add the VE's veth interface to the bridge ====
<pre>[HN]# brctl addif br0 veth101.0</pre>

==== Set up the default route for the VE ====
<pre>[HN]# vzctl exec 101 ip route add default via 85.86.87.193 dev eth0</pre>

==== (Optional) Add routes VE <-> HN ====
The configuration above provides following connections available:
<pre>
VE X <-> VE Y (where VE X and VE Y can locate on any OVZ HN)
VE <-> Internet
</pre>
* A VE accessibility from the HN depends on if the local gateway provides NAT or not (probably - yes).
* A HN accessibility from a VE depends on if the ISP gateway is aware about the local network addresses (most probably - no).

So to provide VE <-> HN accessibility despite the gateways' configuration you can add following route rules:
<pre>
[HN]# ip route add 85.86.87.195 dev br0
[HN]# vzctl exec 101 ip route add 10.0.0.2 dev eth0
</pre>

=== The resulted OVZ Node configuration ===
[[Image:PrivateIPs_fig2.gif|The resulted OVZ Node configuration]]

=== Making the configuration persistent ===

==== Set up a bridge on a HN ====
This can be done by configuring <code>ifcfg-*</code> files located in <code>/etc/sysconfig/network-scripts/</code>.

Assuming you had a configuration file (e.g. <code>ifcfg-eth0</code>) like:
<pre>
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>
 
To make bridge <code>br0</code> automatically created you can create <code>ifcfg-br0</code>:
<pre>
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
IPADDR=10.0.0.2
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
</pre>

and edit <code>ifcfg-eth0</code> file to add <code>eth0</code> interface into the bridge <code>br0</code>:
<pre>
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
</pre>

==== Edit the VE's configuration ====
Add some parameters to the <code>/etc/vz/conf/$VEID.conf</code> which will be used during the network configuration:
* Add/change CONFIG_CUSTOMIZED="yes" (indicates that a custom script should be run on a VE start)
* Add VETH_IP_ADDRESS="<VE IP>/<MASK>" (a VE can have multiple IPs separated by spaces)
* Add VE_DEFAULT_GATEWAY="<VE DEFAULT GATEWAY>"
* Add BRIDGEDEV="<BRIDGE NAME>" (a bridge name to which the VE veth interface should be added)

An example:
<pre>
# Network customization section
CONFIG_CUSTOMIZED="yes"
VETH_IP_ADDRESS="85.86.87.195/26"
VE_DEFAULT_GATEWAY="85.86.87.193"
BRIDGEDEV="br0"
</pre>

==== Create a custom network configuration script ====
which should be called each time a VE started (e.g. <code>/usr/sbin/vznetcfg.custom</code>):
<pre>
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a VE

GLOBALCONFIGFILE=/etc/vz/vz.conf
VECONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
ip=/sbin/ip
. $GLOBALCONFIGFILE
. $VECONFIGFILE

NETIF_OPTIONS=`echo $NETIF | sed 's/,/\n/g'`
for str in $NETIF_OPTIONS; do \
# getting 'ifname' parameter value
if [[ "$str" =~ "^ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VEIFNAME=${str#*=};
fi
# getting 'host_ifname' parameter value
if [[ "$str" =~ "^host_ifname=" ]]; then
# remove the parameter name from the string (along with '=')
VZHOSTIF=${str#*=};
fi
done

if [ ! -n "$VETH_IP_ADDRESS" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
exit 1
fi

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VEIFNAME" ]; then
echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
exit 1
fi

for IP in $VETH_IP_ADDRESS; do
echo "Initializing interface $VZHOSTIF for VE$VEID."
/sbin/ifconfig $VZHOSTIF 0
done

VEROUTEDEV=$VZHOSTIF

if [ -n "$BRIDGEDEV" ]; then
echo "Adding interface $VZHOSTIF to the bridge $BRIDGEDEV."
VEROUTEDEV=$BRIDGEDEV
/usr/sbin/brctl addif $BRIDGEDEV $VZHOSTIF
fi

# Up the interface $VEIFNAME link in VE$VEID
$vzctl exec $VEID $ip link set $VEIFNAME up

for IP in $VETH_IP_ADDRESS; do
echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
$vzctl exec $VEID $ip address add $IP dev $VEIFNAME

# removing the netmask
IP_STRIP=${IP%%/*};

echo "Adding a route from VE0 to VE$VEID."
$ip route add $IP_STRIP dev $VEROUTEDEV
done

if [ -n "$VE0_IP" ]; then
echo "Adding a route from VE$VEID to VE0."
$vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
fi

if [ -n "$VE_DEFAULT_GATEWAY" ]; then
echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
$vzctl exec $VEID \
$ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
fi

exit 0
</pre>

==== Make the script to be run on a VE start ====
In order to run above script on a VE start create the following <code>/etc/vz/vznet.conf</code> file:
<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"
</pre>
{{Note|both <code>/etc/vz/vznet.conf</code> and <code>/usr/sbin/vznetcfg.custom</code> should be executable files.}}

==== Setting the route VE -> HN ====
To set up a route from VE to HN the custom script has to get a HN IP (the $VE0_IP variable in the script). There can be different approaches to specify it:
# Add an entry VE0_IP="VE0 IP" to the <code>$VEID.conf</code>
# Add an entry VE0_IP="VE0 IP" to the <code>/etc/vz/vz.conf</code> (the global configuration config file)
# Implement some smart algorithm to determine the VE0 IP right in the custom network configuration script
Every variant has its pros and cons, nevertheless for HN static IP configuration variant 2 seems to be acceptable (and the most simple).

== (2) An OVZ Hardware Node has two ethernet interfaces ==
Assume you have 2 interfaces eth0 and eth1 and want to separate local traffic (10.0.0.0/24) from the external traffic.
Let's assign eth0 for the external traffic and eth1 for the local one.

If there is no aim to make VE accessible from HN and vice versa, it's enough to replace 'br0' with 'eth1' in the following steps of above configuration:
* Hardware Node configuration -> [[Using_private_IPs_for_Hardware_Nodes#Assign_the_IP_to_the_bridge|Assign the IP to the bridge]]
* Hardware Node configuration -> [[Using_private_IPs_for_Hardware_Nodes#Resurrect_the_default_routing|Resurrect the default routing]]

For the VE <-> HN connections availability it's nesessary to set an IP (local) to the 'br0'.

== (3) Putting VEs to different subnetworks ==
It's enough to set up the correct $VETH_IP_ADDRESS and $VE_DEFAULT_GATEWAY values in the
[[Using_private_IPs_for_Hardware_Nodes#Edit_the_VE.27s_configuration|above configuration]].

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]

[[Category: HOWTO]]
[[Category: Networking]]