OpenVZ Virtuozzo Containers Wiki - User contributions [en]

Backup of a running container with vzdump

2010-05-20T03:02:23Z

StalkR: /* Synopsis */ updated man page

Vzdump is a utility to make consistent snapshots of running OpenVZ containers (and KVM virtual machines if you are using [http://pve.proxmox.com Proxmox VE]). It basically creates a tar archive of the container's private area, which also includes the CT configuration files.

There are several ways to provide consistency:

* Stop the CT during backup (very long downtime)
* Use rsync and suspend/resume (minimal downtime)
* Use LVM2 (no downtime)

Vzdump stores the backup on the disk in a single file. This file should go to a tape backup for archiving.

== Download ==
Download vzdump rpm or deb packages from http://download.openvz.org/contrib/utils/vzdump/ or for newest version, check http://www.proxmox.com/cms_proxmox/en/technology/oss-software/openvz/

For rpm based systems:
<pre>wget http://www.proxmox.com/cms_proxmox/cms/upload/vzdump/vzdump-1.2-5.noarch.rpm</pre>

For Debian based systems:
<pre>wget http://www.proxmox.com/cms_proxmox/cms/upload/vzdump/vzdump_1.2-5_all.deb</pre>

== Installation ==
For rpm based systems:
<pre>rpm -i vzdump-1.2-5.noarch.rpm</pre>

For Debian based systems:
<pre>dpkg -i vzdump_1.2-5_all.deb</pre>

== Synopsis ==

see also:
<pre>man vzdump</pre>

vzdump OPTIONS [--all | <VMID>]
--exclude VMID exclude VMID (assumes --all)
--exclude-path REGEX exclude certain files/directories. You
can use this option more than once to specify
multiple exclude paths
--stdexcludes exclude temporary files and logs
--compress compress dump file (gzip)
--storage STORAGE_ID store resulting files to STORAGE_ID (PVE only)
--script execute hook script
--dumpdir DIR store resulting files in DIR
--maxfiles N maximal number of backup files per VM.
--tmpdir DIR store temporary files in DIR. --suspend and --stop
are using this directory to store a copy of the VM.
--mailto EMAIL send notification mail to EMAIL. You can use
this option more than once to specify multiple
receivers
--stop stop/start VM if running
--suspend suspend/resume VM when running
--snapshot use LVM snapshot when running
--size MB LVM snapshot size (default 1024)
--bwlimit KBPS limit I/O bandwidth; KBytes per second
--lockwait MINUTES maximal time to wait for the global
lock. vzdump uses a global lock file to make
sure that only one instance is running
(running several instance puts too much load
on a server). Default is 180 (3 hours).
--stopwait MINUTES maximal time to wait until a VM is stopped.

== Examples ==
Use a running container, for example install this: [[Proxmox Mail Gateway in container]].

=== Backup ===

Simply dump CT 777 — no snapshot, just archive the container private area and configuration files to the default dump directory (usually <code>/vz/dump/</code>).

vzdump 777

Use rsync and suspend/resume to create a snapshot (minimal downtime).

vzdump --suspend 777

Backup all containers and send notification mails to root.

vzdump --suspend --all --mailto root

Use LVM2 to create snapshots (no downtime).

vzdump --dumpdir /space/backup --snapshot 777

Note that using LVM2 and vzdump to create snapshots requires 512Mb of free space in your VG as described [http://weblogs.amtex.nl/index.php?blog=2&title=using_vzdump_snapshot_to_backup_without_downtime&more=1&c=1&tb=1&pb=1 here].

=== Restore ===

Restore the above backup to CT 600:

vzrestore /space/backup/vzdump-777.tar 600
OR
vzdump --restore /space/backup/vzdump-777.tar 600

== Bugs ==
'''(not fixed in 1.2-4)'''
The rsync command used by vzdump to create the backup in suspend mode partially ignores the "--exclude-path" option.
In fact, even if the excluded paths won't appear in the final output, the whole VPS will be moved to the temporary directory, meaning that you need as much free disk space as your VPS size to use vzdump. It can be an issue in the case of a file server handling many files...

''Workaround:'' A workaround has been proposed on OpenVZ forum, see below for the excerpt. (http://forum.openvz.org/index.php?t=msg&goto=36924&)
<pre>
User: tatawaki
Messages: 3
Registered: December 2008 Junior Member
From: *sbm.shawcable.net

Line 694:
my $rsyncopts = "--stats --numeric-ids --bwlimit=${opt_bwlimit}";

# changes to

my $rsyncopts = "--stats --numeric-ids --bwlimit=${opt_bwlimit}";
$rsyncopts = $rsyncopts." --exclude-from=/home/backups/exclude_vzdump.txt";

the txt file contains <VEID>/home/:

1094/home/
510/opt/
...
</pre>

'''(fixed in 1.1-1)'''
vzdump will fail under Debian Etch in version 1.0-2 if it is invoked with parameter "--snapshot" and if the logical volume name contains a hyphen.

''Workaround:'' One possible workaround is to rename the logical volume in question thus it doesn't contain any hyphen.
A bug report was sent to proxmox on 02 June 2008.
Other distributions or versions may be affected, too.

== Hooks ==

http://nachtmann.it/blog/vzdump-hook-ftp-backup-script - Backup to FTP with limited capacity

[[Category: HOWTO]]

Backup of a running container with vzdump

2010-05-20T03:00:26Z

StalkR: /* Installation */ updated names for version 1.2-5

Vzdump is a utility to make consistent snapshots of running OpenVZ containers (and KVM virtual machines if you are using [http://pve.proxmox.com Proxmox VE]). It basically creates a tar archive of the container's private area, which also includes the CT configuration files.

There are several ways to provide consistency:

* Stop the CT during backup (very long downtime)
* Use rsync and suspend/resume (minimal downtime)
* Use LVM2 (no downtime)

Vzdump stores the backup on the disk in a single file. This file should go to a tape backup for archiving.

== Download ==
Download vzdump rpm or deb packages from http://download.openvz.org/contrib/utils/vzdump/ or for newest version, check http://www.proxmox.com/cms_proxmox/en/technology/oss-software/openvz/

For rpm based systems:
<pre>wget http://www.proxmox.com/cms_proxmox/cms/upload/vzdump/vzdump-1.2-5.noarch.rpm</pre>

For Debian based systems:
<pre>wget http://www.proxmox.com/cms_proxmox/cms/upload/vzdump/vzdump_1.2-5_all.deb</pre>

== Installation ==
For rpm based systems:
<pre>rpm -i vzdump-1.2-5.noarch.rpm</pre>

For Debian based systems:
<pre>dpkg -i vzdump_1.2-5_all.deb</pre>

== Synopsis ==

see also:
<pre>man vzdump</pre>

vzdump OPTIONS [--all | <VMID>]

--exclude VPSID exclude VPSID (assumes --all)
--exclude-path REGEX exclude certain files/directories. You
can use this option more than once to specify
multiple exclude paths
--stdexcludes exclude temorary files and logs
--compress compress dump file (gzip)
--dumpdir DIR store resulting files in DIR
--tmpdir DIR store temporary files in DIR. --suspend and --stop are using this directory to store a copy of the VM.
--mailto EMAIL send notification mail to EMAIL. You can use
this option more than once to specify multiple
receivers
--stop stop/start VPS if running
--suspend suspend/resume VPS when running
--snapshot use LVM snapshot when running
--size MB LVM snapshot size (default 1024)
--bwlimit KBPS limit I/O bandwidth; KBytes per second
--lockwait MINUTES maximal time to wait for the global
lock. vzdump uses a global lock file to make
sure that only one instance is running
(running sereral instance puts too much load
on a server). Default is 180 (3 hours).
--stopwait MINUTES maximal time to wait until a VM is stopped.
--restore FILENAME restore FILENAME

== Examples ==
Use a running container, for example install this: [[Proxmox Mail Gateway in container]].

=== Backup ===

Simply dump CT 777 — no snapshot, just archive the container private area and configuration files to the default dump directory (usually <code>/vz/dump/</code>).

vzdump 777

Use rsync and suspend/resume to create a snapshot (minimal downtime).

vzdump --suspend 777

Backup all containers and send notification mails to root.

vzdump --suspend --all --mailto root

Use LVM2 to create snapshots (no downtime).

vzdump --dumpdir /space/backup --snapshot 777

Note that using LVM2 and vzdump to create snapshots requires 512Mb of free space in your VG as described [http://weblogs.amtex.nl/index.php?blog=2&title=using_vzdump_snapshot_to_backup_without_downtime&more=1&c=1&tb=1&pb=1 here].

=== Restore ===

Restore the above backup to CT 600:

vzrestore /space/backup/vzdump-777.tar 600
OR
vzdump --restore /space/backup/vzdump-777.tar 600

== Bugs ==
'''(not fixed in 1.2-4)'''
The rsync command used by vzdump to create the backup in suspend mode partially ignores the "--exclude-path" option.
In fact, even if the excluded paths won't appear in the final output, the whole VPS will be moved to the temporary directory, meaning that you need as much free disk space as your VPS size to use vzdump. It can be an issue in the case of a file server handling many files...

''Workaround:'' A workaround has been proposed on OpenVZ forum, see below for the excerpt. (http://forum.openvz.org/index.php?t=msg&goto=36924&)
<pre>
User: tatawaki
Messages: 3
Registered: December 2008 Junior Member
From: *sbm.shawcable.net

Line 694:
my $rsyncopts = "--stats --numeric-ids --bwlimit=${opt_bwlimit}";

# changes to

my $rsyncopts = "--stats --numeric-ids --bwlimit=${opt_bwlimit}";
$rsyncopts = $rsyncopts." --exclude-from=/home/backups/exclude_vzdump.txt";

the txt file contains <VEID>/home/:

1094/home/
510/opt/
...
</pre>

'''(fixed in 1.1-1)'''
vzdump will fail under Debian Etch in version 1.0-2 if it is invoked with parameter "--snapshot" and if the logical volume name contains a hyphen.

''Workaround:'' One possible workaround is to rename the logical volume in question thus it doesn't contain any hyphen.
A bug report was sent to proxmox on 02 June 2008.
Other distributions or versions may be affected, too.

== Hooks ==

http://nachtmann.it/blog/vzdump-hook-ftp-backup-script - Backup to FTP with limited capacity

[[Category: HOWTO]]

Backup of a running container with vzdump

2010-05-20T02:59:54Z

StalkR: /* Download */ updated links to vzdump version 1.2-5

Vzdump is a utility to make consistent snapshots of running OpenVZ containers (and KVM virtual machines if you are using [http://pve.proxmox.com Proxmox VE]). It basically creates a tar archive of the container's private area, which also includes the CT configuration files.

There are several ways to provide consistency:

* Stop the CT during backup (very long downtime)
* Use rsync and suspend/resume (minimal downtime)
* Use LVM2 (no downtime)

Vzdump stores the backup on the disk in a single file. This file should go to a tape backup for archiving.

== Download ==
Download vzdump rpm or deb packages from http://download.openvz.org/contrib/utils/vzdump/ or for newest version, check http://www.proxmox.com/cms_proxmox/en/technology/oss-software/openvz/

For rpm based systems:
<pre>wget http://www.proxmox.com/cms_proxmox/cms/upload/vzdump/vzdump-1.2-5.noarch.rpm</pre>

For Debian based systems:
<pre>wget http://www.proxmox.com/cms_proxmox/cms/upload/vzdump/vzdump_1.2-5_all.deb</pre>

== Installation ==
For rpm based systems:
<pre>rpm -i vzdump-1.1-2.noarch.rpm</pre>

For Debian based systems:
<pre>dpkg -i vzdump_1.1-2_all.deb</pre>

== Synopsis ==

see also:
<pre>man vzdump</pre>

vzdump OPTIONS [--all | <VMID>]

--exclude VPSID exclude VPSID (assumes --all)
--exclude-path REGEX exclude certain files/directories. You
can use this option more than once to specify
multiple exclude paths
--stdexcludes exclude temorary files and logs
--compress compress dump file (gzip)
--dumpdir DIR store resulting files in DIR
--tmpdir DIR store temporary files in DIR. --suspend and --stop are using this directory to store a copy of the VM.
--mailto EMAIL send notification mail to EMAIL. You can use
this option more than once to specify multiple
receivers
--stop stop/start VPS if running
--suspend suspend/resume VPS when running
--snapshot use LVM snapshot when running
--size MB LVM snapshot size (default 1024)
--bwlimit KBPS limit I/O bandwidth; KBytes per second
--lockwait MINUTES maximal time to wait for the global
lock. vzdump uses a global lock file to make
sure that only one instance is running
(running sereral instance puts too much load
on a server). Default is 180 (3 hours).
--stopwait MINUTES maximal time to wait until a VM is stopped.
--restore FILENAME restore FILENAME

== Examples ==
Use a running container, for example install this: [[Proxmox Mail Gateway in container]].

=== Backup ===

Simply dump CT 777 — no snapshot, just archive the container private area and configuration files to the default dump directory (usually <code>/vz/dump/</code>).

vzdump 777

Use rsync and suspend/resume to create a snapshot (minimal downtime).

vzdump --suspend 777

Backup all containers and send notification mails to root.

vzdump --suspend --all --mailto root

Use LVM2 to create snapshots (no downtime).

vzdump --dumpdir /space/backup --snapshot 777

Note that using LVM2 and vzdump to create snapshots requires 512Mb of free space in your VG as described [http://weblogs.amtex.nl/index.php?blog=2&title=using_vzdump_snapshot_to_backup_without_downtime&more=1&c=1&tb=1&pb=1 here].

=== Restore ===

Restore the above backup to CT 600:

vzrestore /space/backup/vzdump-777.tar 600
OR
vzdump --restore /space/backup/vzdump-777.tar 600

== Bugs ==
'''(not fixed in 1.2-4)'''
The rsync command used by vzdump to create the backup in suspend mode partially ignores the "--exclude-path" option.
In fact, even if the excluded paths won't appear in the final output, the whole VPS will be moved to the temporary directory, meaning that you need as much free disk space as your VPS size to use vzdump. It can be an issue in the case of a file server handling many files...

''Workaround:'' A workaround has been proposed on OpenVZ forum, see below for the excerpt. (http://forum.openvz.org/index.php?t=msg&goto=36924&)
<pre>
User: tatawaki
Messages: 3
Registered: December 2008 Junior Member
From: *sbm.shawcable.net

Line 694:
my $rsyncopts = "--stats --numeric-ids --bwlimit=${opt_bwlimit}";

# changes to

my $rsyncopts = "--stats --numeric-ids --bwlimit=${opt_bwlimit}";
$rsyncopts = $rsyncopts." --exclude-from=/home/backups/exclude_vzdump.txt";

the txt file contains <VEID>/home/:

1094/home/
510/opt/
...
</pre>

'''(fixed in 1.1-1)'''
vzdump will fail under Debian Etch in version 1.0-2 if it is invoked with parameter "--snapshot" and if the logical volume name contains a hyphen.

''Workaround:'' One possible workaround is to rename the logical volume in question thus it doesn't contain any hyphen.
A bug report was sent to proxmox on 02 June 2008.
Other distributions or versions may be affected, too.

== Hooks ==

http://nachtmann.it/blog/vzdump-hook-ftp-backup-script - Backup to FTP with limited capacity

[[Category: HOWTO]]

Debian template creation

2009-09-17T06:16:39Z

StalkR: /* Preparing for and packing template cache */ remove /etc/hostname file before packing template

These are rough instructions of how to manually create basic Debian Etch (4.0) template cache, which can be used to create OpenVZ [[VE]]s based on Debian Etch (4.0). (see also <tt>/usr/share/doc/vzctl/README.Debian</tt> in the ''vzctl'' Debian package)

'''Notes:'''
* You shouldn't be running as root, but as a user that is permitted to use sudo instead. It's a dangerous idea, run as root at your peril.
* Anywhere you see <tt>/vz</tt>, you might really need to use <tt>/var/lib/vz</tt> instead, especially on a Debian Etch host.
* Anywhere you see <tt>http://http.us.debian.org/debian/</tt>, you can substitute your favorite Debian mirror. ([http://www.debian.org/mirror/list List of official Debian Mirrors])

== Prerequisites ==
You need to have a working copy of <tt>debootstrap</tt> running on your hardware node.

For Debian:
sudo apt-get install debootstrap

For Gentoo:
sudo emerge debootstrap

For Fedora (at least Fedora 8 have it, not sure about earlier versions):
sudo yum install debootstrap

For other distros you might need to install it from sources, or search for an appropriate package for your distribution. An RPM is available on the [http://forum.openvz.org/index.php?t=tree&th=142&mid=584 OpenVZ Forum].

== Bootstrapping Debian ==

You can install different releases of Debian into a VE's private directory using the debootstrap command.

The command parameters are:

debootstrap --arch ARCH NAME DIRECTORY URL

Specify your architecture instead of <tt>i386</tt> if you're using something other than i386/x86. For example, for AMD64/x86_64, use <tt>amd64</tt> or for ia64, use <tt>ia64</tt>.

We use VE ID of 777 for this example; surely it can be any other unused ID.

=== Etch (current Debian stable) ===

debootstrap --arch i386 etch /vz/private/777 http://http.us.debian.org/debian/

=== Sarge (old release) ===

debootstrap sarge /vz/private/777 http://http.us.debian.org/debian

=== Woody (very old release) ===

debootstrap woody /vz/private/777 http://archive.debian.org/

== Preparing and starting the VE ==

=== Setting VE config ===
First, we need a config for the [[VE]]:
sudo vzctl set 777 --applyconfig vps.basic --save

=== Setting VE OSTEMPLATE ===
Also, we need <tt>OSTEMPLATE</tt> to be set in VE configuration file, for the [[vzctl]] to work properly.

sudo sh -c 'echo "OSTEMPLATE=debian-4.0" >> /etc/vz/conf/777.conf'

=== Setting VE IP address ===
For the [[VE]] to be able to download updates from the Internet, we need a valid IP address for it:
sudo vzctl set 777 --ipadd x.x.x.x --save

{{Note|if you use private IP for the VE, you have to set up NAT as described in [[Using NAT for VE with private IPs]].}}

=== Setting DNS server for VE ===
For the [[VE]] to be able to download updates from the Internet, we also need to specify a DNS for it:
sudo vzctl set 777 --nameserver x.x.x.x --save

=== Starting VE ===
Now start the VE:
sudo vzctl start 777

== Customizing the installation ==
A few things need to be done inside a newly created VE for it to become suitable for OpenVZ. All those things are done inside the VE, so first command is:
sudo vzctl enter 777
export PATH=/sbin:/usr/sbin:/bin:/usr/bin

{{Warning|Do not run the commands below on the hardware node, they are only to be run within the VE!}}

=== Set Debian repositories ===
cat <<EOF > /etc/apt/sources.list
deb http://http.us.debian.org/debian etch main contrib
deb http://security.debian.org etch/updates main contrib
EOF

=== Get new security updates ===
apt-get update
apt-get upgrade

=== Install some more packages ===
Installing packages could be an interactive process so the system might ask some questions. You can install more packages if you'd like. For example:
apt-get install ssh quota

=== Set sane permissions for <tt>/root</tt> directory ===
chmod 700 /root

=== Disable root login ===
This will disable root login by default.
usermod -L root

=== Disable getty ===
Disable running <tt>getty</tt>s on terminals as a VE does not have any:
sed -i -e '/getty/d' /etc/inittab

=== Disable <tt>sync()</tt> for syslog ===
Turn off doing <tt>sync()</tt> on every write for <tt>syslog</tt>'s log files, to improve I/O performance:
<pre>sed -i -e 's@$[[:space:]]$$/var/log/$@\1-\2@' /etc/*syslog.conf</pre>

=== Fix <tt>/etc/mtab</tt> ===
Link <tt>/etc/mtab</tt> to <tt>/proc/mounts</tt>, so <tt>df</tt> and friends will work:
rm -f /etc/mtab
ln -s /proc/mounts /etc/mtab

=== Remove some unneeded packages ===
If you have any packages you'd like to remove, now's the time for it. Here's an example:
dpkg --purge modutils ppp pppoeconf pppoe pppconfig

=== Disable services ===
Do not start some services, stick to bare minimum:
update-rc.d -f klogd remove
update-rc.d -f quotarpc remove
update-rc.d -f exim4 remove
update-rc.d -f inetd remove

=== Fix SSH host keys ===
This is only useful if you installed SSH. Each individual [[VE]] should have its own pair of SSH host keys. The code below will wipe out the existing SSH keys and instruct the newly-created [[VE]] to create new SSH keys on first boot.


<source lang="bash">
rm -f /etc/ssh/ssh_host_*
cat << EOF > /etc/rc2.d/S15ssh_gen_host_keys
#!/bin/bash
ssh-keygen -f /etc/ssh/ssh_host_rsa_key -t rsa -N ''
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -t dsa -N ''
rm -f \$0
EOF
chmod a+x /etc/rc2.d/S15ssh_gen_host_keys
</source>

=== Clean packages ===
After installing packages, you'll have some junk packages laying around in your cache. Since you don't want your template to have those, this command will wipe them out.
apt-get clean

Now everything is done. Exit from the template and go back to the hardware node.
exit

== Preparing for and packing template cache ==

We don't need an IP for the VE anymore, and we definitely do not need it in template cache, so remove it:
sudo vzctl set 777 --ipdel all --save

Also, remove DNS server and search domain information from ''/etc/resolv.conf'' file '''in VE''':
sudo nano /vz/private/777/etc/resolv.conf

Also, remove ''/etc/hostname'' file '''in VE''':
sudo rm -f /vz/private/777/etc/hostname

Stop the VE:
sudo vzctl stop 777

Go to the VE directory:
cd /vz/private/777

Now create a cached OS tarball. In the command below, you'll want to replace <tt>i386</tt> with your architecture (i386, amd64, ia64, etc).
sudo tar -zcf /vz/template/cache/debian-4.0-i386-minimal.tar.gz .

Look at the resulting tarball to see its size is sane:
# ls -lh /vz/template/cache
-rw-r--r-- 1 root root 51M Apr 10 03:16 debian-4.0-i386-minimal.tar.gz

== Checking if template cache works ==
We can now create a VE based on the just-created template cache. Be sure to change <tt>i386</tt> to your architecture just like you did when you named the tarball above.
sudo vzctl create 123456 --ostemplate debian-4.0-i386-minimal

Now make sure that it works:
sudo vzctl start 123456
sudo vzctl exec 123456 ps ax

You should see that a few processes are running.

== Final cleanup ==
Stop and remove the test VE you just created:
sudo vzctl stop 123456
sudo vzctl destroy 123456
sudo rm /etc/vz/conf/123456.conf.destroyed

Finally, let's remove the VE we used for OS template cache creation:
sudo vzctl destroy 777
sudo rm /etc/vz/conf/777.conf.destroyed

[[Category: HOWTO]]
[[Category: Templates]]
[[Category: Debian]]

Virtual Ethernet device

2009-09-12T00:36:17Z

StalkR: /* Configure device and add route in CT0 */ add -> adds

'''Virtual Ethernet device''' is an Ethernet-like device which can be used
inside a [[container]]. Unlike [[venet]] network device, [[veth]] device
has a MAC address, therefore it can be used in configurations, when veth
is bridged to ethX or other device and container's user fully sets up
his networking himself, including IPs, gateways etc.

Virtual Ethernet device consist of two Ethernet devices --
the one in [[CT0]] and another one in CT. These devices are connected
to each other, so if a packet goes to one
device it will come out from the other device.

== Virtual Ethernet device usage ==

=== Kernel module ===
First of all, make sure the <code>vzethdev</code> module is loaded:
<pre>
# lsmod | grep vzeth
vzethdev 8224 0
vzmon 35164 5 vzethdev,vznetdev,vzrst,vzcpt
vzdev 3080 4 vzethdev,vznetdev,vzmon,vzdquota
</pre>

In case it is not loaded, load it:
<pre>
# modprobe vzethdev
</pre>

{{Note|in vzctl < 3.0.11, vzethdev is not autoloaded by <code>/etc/init.d/vz</code> script, so you have to edit it to load this module.}}

=== MAC addresses ===
In the below commands, you should use random MAC addresses. Do not use MAC addresses of real eth devices, because this can lead to collisions.

MAC addresses must be entered in XX:XX:XX:XX:XX:XX format.

YOU MAY NOT NEED TO GENERATE MAC ADDRESSES BY HAND BECAUSE vzctl --veth_add
MAY GENERATE THEM AUTOMATICALLY AS NECESSARY.

Nevertheless, there is a utility script available for generating MAC addresses: http://www.easyvmx.com/software/easymac.sh. It is to be used like this:

chmod +x easymac.sh
./easymac.sh -R

=== Adding veth to a CT ===

==== syntax vzctl version > 3.0.22 ====

vzctl set <CTID> --netif_add <ifname>[,<mac>,<host_ifname>,<host_mac>,<bridge>]

Here
* <tt>ifname</tt> is the Ethernet device name in the CT
* <tt>mac</tt> is its MAC address in the CT
* <tt>host_ifname</tt> is the Ethernet device name on the host ([[CT0]])
* <tt>host_mac</tt> is its MAC address on the host ([[CT0]])
* <tt>bridge</tt> is an optional parameter which can be used in custom network start scripts to automatically add the interface to a bridge.

{{Note|All parameters except <code>ifname</code> are optional and are automatically generated if not specified.}}

Example:

vzctl set 101 --netif_add eth0 --save

Or, if you want to specify everything:

vzctl set 101 --netif_add eth0,00:12:34:56:78:9A,veth101.0,00:12:34:56:78:9B --save

Or, if you want to specify the bridge and leave the other values autogenerated:

vzctl set 101 --netif_add eth0,,,,vmbr1 --save

==== syntax vzctl version >= 3.0.14 ====

Syntax is the same as above, but without a <bridge> parameter.

==== syntax vzctl version < 3.0.14 ====

vzctl set <CTID> --veth_add <dev_name>,<dev_addr>,<ve_dev_name>,<ve_dev_addr>

Here
* <tt>dev_name</tt> is the Ethernet device name that you are creating on the [[CT0|host system]]
* <tt>dev_addr</tt> is its MAC address
* <tt>ve_dev_name</tt> is the corresponding Ethernet device name you are creating on the CT
* <tt>ve_dev_addr</tt> is its MAC address

{{Note|this option is incremental, so devices are added to already existing ones.}}

NB there should no spaces after the commas.

Example:
<pre>
[host-node] ifconfig eth0
...
HWaddress 00:12:34:56:78:9B
...
</pre>

[host-node] easymac.sh -R
00:12:34:56:78:9A

vzctl set 101 --veth_add veth101.0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save

After executing this command <tt>veth</tt> device will be created for CT 101 and veth configuration will be saved to a CT configuration file.
Host-side Ethernet device will have <tt>veth101.0</tt> name and <tt>00:12:34:56:78:9A</tt> MAC address.
CT-side Ethernet device will have <tt>eth0</tt> name and <tt>00:12:34:56:78:9B</tt> MAC address.

=== Removing veth from a CT ===

==== syntax vzctl version >= 3.0.14 ====

vzctl set <CTID> --netif_del <dev_name>|all

Here
* <code>dev_name</code> is the Ethernet device name in the [[CT]].

{{Note|If you want to remove all Ethernet devices in CT, use <code>all</code>.}}

Example:

vzctl set 101 --netif_del eth0 --save

==== syntax vzctl version < 3.0.14 ====

vzctl set <CTID> --veth_del <dev_name>

Here <tt>dev_name</tt> is the Ethernet device name in the [[CT0|host system]].

Example:

vzctl set 101 --veth_del veth101.0 --save

After executing this command veth device with host-side Ethernet name
<code>veth101.0</code> will be removed from CT101 and veth configuration
will be updated in CT config file.

== Common configurations with virtual Ethernet devices ==
Module <tt>vzethdev</tt> must be loaded to operate with veth devices.

=== Simple configuration with virtual Ethernet device ===

Assuming you have 192.168.0.0/24 on your LAN, you will learn how to integrate a container in this LAN using veth.

==== Start a CT ====

[host-node]# vzctl start 101

==== Add veth device to CT ====

[host-node]# vzctl set 101 --netif_add eth0 --save

This allocates a MAC address and associates it with the host eth0 port.

==== Configure devices in CT0 ====
<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/proxy_arp
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
</pre>

Notes:
* These files did not exist for me when trying ([[User:Mrjcleaver|Mrjcleaver]] 14:04, 31 May 2008 (EDT))

==== Configure device in CT ====
<pre>
[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0
[ve-101]# /sbin/ip addr add 192.168.0.101 dev eth0
[ve-101]# /sbin/ip route add default dev eth0
</pre>

Notes:
* Until you ifconfig eth0 it won't appear. When you do it will use the mac address netif_add added earlier
* 192.168.0.101 is chosen to be an [[unrouteable private ip address]]. Where 101 reminds you that it is node 101.
* The "ip route" tells all traffic to head to "device eth0"
* In theory you could [[use dhcpd with OpenVZ]] and dhclient to pick up an DHCP address from your router instead of hardwiring it
** http://openvz.org/pipermail/users/2005-November/000020.html

==== Add route in [[CT0]] ====

[host-node]# ip route add 192.168.0.101 dev veth101.0

=== Using a directly routed IPv4 with virtual Ethernet device ===

==== Situation ====
Hardware Node (HN/CT0) has 192.168.0.1/24 with router 192.168.0.254.

We also know that IPv4 10.0.0.1/32 is directly routed to 192.168.0.1 (this is called a ''fail-over IP'').

We want to give this directly routed IPv4 address to a container (CT).

==== Start container ====

[host-node]# vzctl start 101

==== Add veth device to CT ====

[host-node]# vzctl set 101 --netif_add eth0 --save

This allocates a MAC address and associates it with the host eth0 port.

==== Configure device and add route in CT0 ====

<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# ip route add 10.0.0.1 dev veth101.0
</pre>

You can automatize this at VPS creation by using a mount script <tt>$VEID.mount</tt>.

The problem here is that the ''veth'' interface appears in CT0 '''after''' VPS has started, therefore we cannot directly use the commands in the mount script. We launch a shell script (enclosed by { }) in background (operator '''&''') that waits for the interface to be ready and then adds the IP route.

Contents of the mount script <tt>/etc/vz/conf/101.mount</tt>:
<pre>
#!/bin/bash
# This script source VPS configuration files in the same order as vzctl does

# if one of these files does not exist then something is really broken
[ -f /etc/vz/vz.conf ] || exit 1
[ -f $VE_CONFFILE ] || exit 1

# source both files. Note the order, it is important
. /etc/vz/vz.conf
. $VE_CONFFILE

# Configure veth with IP after VPS has started
{
IP=X.Y.Z.T
DEV=veth101.0
while sleep 1; do
/sbin/ifconfig $DEV 0 >/dev/null 2>&1
if [ $? -eq 0 ]; then
/sbin/ip route add $IP dev $DEV
break
fi
done
} &
</pre>

==== Make sure IPv4 forwarding is enabled in CT0 ====

<pre>
[host-node]# echo 1 > /proc/sys/net/ipv4/ip_forward
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
</pre>
You can permanently set this by using <tt>/etc/sysctl.conf</tt>.

==== Configure device in CT ====

1. Configure IP address

2. Add gateway

3. Add default route

<pre>
[ve-101]# /sbin/ifconfig eth0 10.0.0.1 netmask 255.255.255.255
[ve-101]# /sbin/ip route add 192.168.0.1 dev eth0
[ve-101]# /sbin/ip route default via 192.168.0.1
</pre>

In a Debian container, you can configure this permanently by using <tt>/etc/network/interfaces</tt>:
<pre>
auto eth0
iface eth0 inet static
address 10.0.0.1
netmask 255.255.255.255
up /sbin/ip route add 192.168.0.1 dev eth0
up /sbin/ip route add default via 192.168.0.1
</pre>

=== Virtual Ethernet device with IPv6 ===

==== Start a [[CT]] ====

[host-node]# vzctl start 101

==== Add veth device to the [[CT]] ====

[host-node]# vzctl set 101 --veth_add eth0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save

==== Configure devices in the [[CT0]] ====

[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/all/forwarding

==== Configure device in [[CT]] ====

[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0

==== Start router advertisement daemon (radvd) for IPv6 in CT0 ====
First you need to edit radvd configuration file. Here is a simple example of <tt>/etc/radv.conf</tt>:
<pre>
interface veth101.0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvHomeAgentFlag off;

prefix 3ffe:2400:0:0::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
};

interface eth0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvHomeAgentFlag off;

prefix 3ffe:0302:0011:0002::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
};
</pre>

Then, start radvd:

[host-node]# /etc/init.d/radvd start

==== Add IPv6 addresses to devices in [[CT0]] ====
<pre>
[host-node]# ip addr add dev veth101.0 3ffe:2400::212:34ff:fe56:789a/64
[host-node]# ip addr add dev eth0 3ffe:0302:0011:0002:211:22ff:fe33:4455/64
</pre>

=== Virtual Ethernet devices can be joined in one bridge ===
Perform steps 1 - 4 from Simple configuration chapter for several containers and/or veth devices

==== Create bridge device ====
<pre>
[host-node]# brctl addbr vzbr0
</pre>

==== Add veth devices to bridge ====
<pre>
[host-node]# brctl addif vzbr0 veth101.0
...
[host-node]# brctl addif vzbr0 veth101.n
[host-node]# brctl addif vzbr0 veth102.0
...
...
[host-node]# brctl addif vzbr0 vethXXX.N
</pre>

==== Configure bridge device ====
<pre>
[host-node]# ifconfig vzbr0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/vzbr0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/vzbr0/proxy_arp
</pre>

==== Add routes in [[CT0]] ====
<pre>
[host-node]# ip route add 192.168.101.1 dev vzbr0
...
[host-node]# ip route add 192.168.101.n dev vzbr0
[host-node]# ip route add 192.168.102.1 dev vzbr0
...
...
[host-node]# ip route add 192.168.XXX.N dev vzbr0
</pre>

Thus you'll have more convinient configuration, i.e. all routes to containers will be through this bridge and containers can communicate with each other even without these routes.

=== Making a veth-device persistent ===
According to http://bugzilla.openvz.org/show_bug.cgi?id=301 , a bug that stopped the veth device persistent was "Obsoleted now when --veth_add/--veth_del are introduced"

See http://wiki.openvz.org/w/index.php?title=Virtual_Ethernet_device&diff=5990&oldid=5989#Making_a_veth-device_persistent for a workaround that used to be described in this section.

That's it! At this point, when you restart the CT you should see a new line in the output, indicating that the interface is being configured and a new route being added. And you should be able to ping the host, and to enter the CT and use the network.

=== Making a bridged veth-device persistent ===

Like the above example, here it is how to add the veth device to a bridge in a persistent way.

==== method for vzctl version > 3.0.22 ====

Newer versions of vzctl includes a 'vznetaddbr' script, which makes use of the new <bridge> parameter of the --netif_add switch.

Just create /etc/vz/vznet.conf containing the following.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetaddbr"
</pre>

The script uses 'vmbr0' as default bridge name when no bridge is specified.

==== method for vzctl version <= 3.0.22 ====

Older vzctl doesn't offer an automatic function to do this.

1. First, edit the CT's configuration to specify what is the host bridge , and to indicate that a custom script should be run when starting up a CT.
* Open up /etc/vz/conf/CTID.conf
* Comment out any IP_ADDRESS entries to prevent a CTNET-device from being created in the CT
* Add or change the entry CONFIG_CUSTOMIZED="yes"
* Add an entry VZHOSTBR="<bridge if>" which is the bridge interface (already configured and up), you want to extend.

2. Now to create that "custom script". The following helper script will check the configuration file for the bridge interface name and for the veth interface, and add the interface to the bridge. Create the script /usr/sbin/vznetaddbr to have the following, and then <code>chmod 0500 /usr/sbin/vznetaddbr</code> to make it executable.

<pre>
#!/bin/bash
# /usr/sbin/vznetaddbr
# a script to add virtual network interfaces (veth's) in a CT to a bridge on CT0

CONFIGFILE=/etc/vz/conf/$VEID.conf
. $CONFIGFILE
VZHOSTIF=`echo $NETIF |sed 's/^.*host_ifname=$.*$,.*$/\1/g'`

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE CT$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VZHOSTBR" ]; then
echo "According to $CONFIGFILE CT$VEID has no bridge interface configured."
exit 1
fi

echo "Adding interface $VZHOSTIF to bridge $VZHOSTBR on CT0 for CT$VEID"
/sbin/ifconfig $VZHOSTIF 0
echo 1 > /proc/sys/net/ipv4/conf/$VZHOSTIF/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/$VZHOSTIF/forwarding
/usr/sbin/brctl addif $VZHOSTBR $VZHOSTIF

exit 0
</pre>

3. Now create /etc/vz/vznet.conf containing the following. This is what defines the "custom script" as being the vznetaddbr which you just created.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetaddbr"
</pre>

This may not work for particularily old versions of vzctl, e.g., the version 3.0.11 that ships with Debian Etch. For those versions, you can try a hack: Use the custom script <code>/etc/vz/conf/$VID.mount</code> which is available, even in these old versions. But it gets called too early, before the networking has been set up. But it can start some background process, which waits and occasionally polls until $VZHOSTIF has become available. Here is one way to go about it:

<pre>
#!/bin/bash

CONFIGFILE="/etc/vz/conf/$VEID.conf"

if [ -f "$CONFIGFILE" ]
then
. "$CONFIGFILE"
VZHOSTIF=`echo $NETIF |sed 's/^.*host_ifname=$.*$,.*$/\1/g'`
export VZHOSTIF
export VZHOSTBR

# Fork into the background and try a few times,
# until the host side of the interface appears:
/bin/bash -c 'for i in 5 10 20 40 80 160
do
if ifconfig -a | grep -q "$VZHOSTIF"
then
exec /usr/sbin/vznetaddbr
else
sleep "$i"
fi
done
' &

# In the meantime, let the CT's start process continue,
# or else the interface will never appear:
exit 0
else
$0: Config file "$CONFIGFILE" does not exist.
exit 1
fi
</pre>

4. Of course, the CT's operating system will need to have . Consult the manual for your CT's OS for details.

When the CT is started, the veth specified in the NETIF value is added to the bridge specified. You can check this by doing <code>brctl show</code>

Inside the CT you can configure the interface statically or using dhcp, as a real interface attached to a switch on the lan.

=== Virtual Ethernet devices + VLAN ===
This configuration can be done by adding vlan device to the previous configuration.

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]
* [[Using private IPs for Hardware Nodes]]
* Troubleshooting: [[Bridge doesn't forward packets]]

== External links ==
* [http://www.tldp.org/HOWTO/Linux+IPv6-HOWTO/hints-daemons-radvd.html Linux IPv6 HOWTO, a chapter about radvd]
* [http://vireso.blogspot.com/2008/02/2-veth-with-2-brindges-on-openvz-at.html 2 veth with 2 bridges setup]

[[Category: Networking]]
[[Category: HOWTO]]

Virtual Ethernet device

2009-09-12T00:21:22Z

StalkR: /* Configure device and add route in CT0 */ adding script to automatize route creation + new subsection to make sure ipv4 forwarding is enabled in CT0

'''Virtual Ethernet device''' is an Ethernet-like device which can be used
inside a [[container]]. Unlike [[venet]] network device, [[veth]] device
has a MAC address, therefore it can be used in configurations, when veth
is bridged to ethX or other device and container's user fully sets up
his networking himself, including IPs, gateways etc.

Virtual Ethernet device consist of two Ethernet devices --
the one in [[CT0]] and another one in CT. These devices are connected
to each other, so if a packet goes to one
device it will come out from the other device.

== Virtual Ethernet device usage ==

=== Kernel module ===
First of all, make sure the <code>vzethdev</code> module is loaded:
<pre>
# lsmod | grep vzeth
vzethdev 8224 0
vzmon 35164 5 vzethdev,vznetdev,vzrst,vzcpt
vzdev 3080 4 vzethdev,vznetdev,vzmon,vzdquota
</pre>

In case it is not loaded, load it:
<pre>
# modprobe vzethdev
</pre>

{{Note|in vzctl < 3.0.11, vzethdev is not autoloaded by <code>/etc/init.d/vz</code> script, so you have to edit it to load this module.}}

=== MAC addresses ===
In the below commands, you should use random MAC addresses. Do not use MAC addresses of real eth devices, because this can lead to collisions.

MAC addresses must be entered in XX:XX:XX:XX:XX:XX format.

YOU MAY NOT NEED TO GENERATE MAC ADDRESSES BY HAND BECAUSE vzctl --veth_add
MAY GENERATE THEM AUTOMATICALLY AS NECESSARY.

Nevertheless, there is a utility script available for generating MAC addresses: http://www.easyvmx.com/software/easymac.sh. It is to be used like this:

chmod +x easymac.sh
./easymac.sh -R

=== Adding veth to a CT ===

==== syntax vzctl version > 3.0.22 ====

vzctl set <CTID> --netif_add <ifname>[,<mac>,<host_ifname>,<host_mac>,<bridge>]

Here
* <tt>ifname</tt> is the Ethernet device name in the CT
* <tt>mac</tt> is its MAC address in the CT
* <tt>host_ifname</tt> is the Ethernet device name on the host ([[CT0]])
* <tt>host_mac</tt> is its MAC address on the host ([[CT0]])
* <tt>bridge</tt> is an optional parameter which can be used in custom network start scripts to automatically add the interface to a bridge.

{{Note|All parameters except <code>ifname</code> are optional and are automatically generated if not specified.}}

Example:

vzctl set 101 --netif_add eth0 --save

Or, if you want to specify everything:

vzctl set 101 --netif_add eth0,00:12:34:56:78:9A,veth101.0,00:12:34:56:78:9B --save

Or, if you want to specify the bridge and leave the other values autogenerated:

vzctl set 101 --netif_add eth0,,,,vmbr1 --save

==== syntax vzctl version >= 3.0.14 ====

Syntax is the same as above, but without a <bridge> parameter.

==== syntax vzctl version < 3.0.14 ====

vzctl set <CTID> --veth_add <dev_name>,<dev_addr>,<ve_dev_name>,<ve_dev_addr>

Here
* <tt>dev_name</tt> is the Ethernet device name that you are creating on the [[CT0|host system]]
* <tt>dev_addr</tt> is its MAC address
* <tt>ve_dev_name</tt> is the corresponding Ethernet device name you are creating on the CT
* <tt>ve_dev_addr</tt> is its MAC address

{{Note|this option is incremental, so devices are added to already existing ones.}}

NB there should no spaces after the commas.

Example:
<pre>
[host-node] ifconfig eth0
...
HWaddress 00:12:34:56:78:9B
...
</pre>

[host-node] easymac.sh -R
00:12:34:56:78:9A

vzctl set 101 --veth_add veth101.0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save

After executing this command <tt>veth</tt> device will be created for CT 101 and veth configuration will be saved to a CT configuration file.
Host-side Ethernet device will have <tt>veth101.0</tt> name and <tt>00:12:34:56:78:9A</tt> MAC address.
CT-side Ethernet device will have <tt>eth0</tt> name and <tt>00:12:34:56:78:9B</tt> MAC address.

=== Removing veth from a CT ===

==== syntax vzctl version >= 3.0.14 ====

vzctl set <CTID> --netif_del <dev_name>|all

Here
* <code>dev_name</code> is the Ethernet device name in the [[CT]].

{{Note|If you want to remove all Ethernet devices in CT, use <code>all</code>.}}

Example:

vzctl set 101 --netif_del eth0 --save

==== syntax vzctl version < 3.0.14 ====

vzctl set <CTID> --veth_del <dev_name>

Here <tt>dev_name</tt> is the Ethernet device name in the [[CT0|host system]].

Example:

vzctl set 101 --veth_del veth101.0 --save

After executing this command veth device with host-side Ethernet name
<code>veth101.0</code> will be removed from CT101 and veth configuration
will be updated in CT config file.

== Common configurations with virtual Ethernet devices ==
Module <tt>vzethdev</tt> must be loaded to operate with veth devices.

=== Simple configuration with virtual Ethernet device ===

Assuming you have 192.168.0.0/24 on your LAN, you will learn how to integrate a container in this LAN using veth.

==== Start a CT ====

[host-node]# vzctl start 101

==== Add veth device to CT ====

[host-node]# vzctl set 101 --netif_add eth0 --save

This allocates a MAC address and associates it with the host eth0 port.

==== Configure devices in CT0 ====
<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/proxy_arp
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
</pre>

Notes:
* These files did not exist for me when trying ([[User:Mrjcleaver|Mrjcleaver]] 14:04, 31 May 2008 (EDT))

==== Configure device in CT ====
<pre>
[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0
[ve-101]# /sbin/ip addr add 192.168.0.101 dev eth0
[ve-101]# /sbin/ip route add default dev eth0
</pre>

Notes:
* Until you ifconfig eth0 it won't appear. When you do it will use the mac address netif_add added earlier
* 192.168.0.101 is chosen to be an [[unrouteable private ip address]]. Where 101 reminds you that it is node 101.
* The "ip route" tells all traffic to head to "device eth0"
* In theory you could [[use dhcpd with OpenVZ]] and dhclient to pick up an DHCP address from your router instead of hardwiring it
** http://openvz.org/pipermail/users/2005-November/000020.html

==== Add route in [[CT0]] ====

[host-node]# ip route add 192.168.0.101 dev veth101.0

=== Using a directly routed IPv4 with virtual Ethernet device ===

==== Situation ====
Hardware Node (HN/CT0) has 192.168.0.1/24 with router 192.168.0.254.

We also know that IPv4 10.0.0.1/32 is directly routed to 192.168.0.1 (this is called a ''fail-over IP'').

We want to give this directly routed IPv4 address to a container (CT).

==== Start container ====

[host-node]# vzctl start 101

==== Add veth device to CT ====

[host-node]# vzctl set 101 --netif_add eth0 --save

This allocates a MAC address and associates it with the host eth0 port.

==== Configure device and add route in CT0 ====

<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# ip route add 10.0.0.1 dev veth101.0
</pre>

You can automatize this at VPS creation by using a mount script <tt>$VEID.mount</tt>.

The problem here is that the ''veth'' interface appears in CT0 '''after''' VPS has started, therefore we cannot directly use the commands in the mount script. We launch a shell script (enclosed by { }) in background (operator '''&''') that waits for the interface to be ready and then add the IP route.

Contents of the mount script <tt>/etc/vz/conf/101.mount</tt>:
<pre>
#!/bin/bash
# This script source VPS configuration files in the same order as vzctl does

# if one of these files does not exist then something is really broken
[ -f /etc/vz/vz.conf ] || exit 1
[ -f $VE_CONFFILE ] || exit 1

# source both files. Note the order, it is important
. /etc/vz/vz.conf
. $VE_CONFFILE

# Configure veth with IP after VPS has started
{
IP=X.Y.Z.T
DEV=veth101.0
while sleep 1; do
/sbin/ifconfig $DEV 0 >/dev/null 2>&1
if [ $? -eq 0 ]; then
/sbin/ip route add $IP dev $DEV
break
fi
done
} &
</pre>

==== Make sure IPv4 forwarding is enabled in CT0 ====

<pre>
[host-node]# echo 1 > /proc/sys/net/ipv4/ip_forward
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
</pre>
You can permanently set this by using <tt>/etc/sysctl.conf</tt>.

==== Configure device in CT ====

1. Configure IP address

2. Add gateway

3. Add default route

<pre>
[ve-101]# /sbin/ifconfig eth0 10.0.0.1 netmask 255.255.255.255
[ve-101]# /sbin/ip route add 192.168.0.1 dev eth0
[ve-101]# /sbin/ip route default via 192.168.0.1
</pre>

In a Debian container, you can configure this permanently by using <tt>/etc/network/interfaces</tt>:
<pre>
auto eth0
iface eth0 inet static
address 10.0.0.1
netmask 255.255.255.255
up /sbin/ip route add 192.168.0.1 dev eth0
up /sbin/ip route add default via 192.168.0.1
</pre>

=== Virtual Ethernet device with IPv6 ===

==== Start a [[CT]] ====

[host-node]# vzctl start 101

==== Add veth device to the [[CT]] ====

[host-node]# vzctl set 101 --veth_add eth0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save

==== Configure devices in the [[CT0]] ====

[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/all/forwarding

==== Configure device in [[CT]] ====

[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0

==== Start router advertisement daemon (radvd) for IPv6 in CT0 ====
First you need to edit radvd configuration file. Here is a simple example of <tt>/etc/radv.conf</tt>:
<pre>
interface veth101.0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvHomeAgentFlag off;

prefix 3ffe:2400:0:0::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
};

interface eth0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvHomeAgentFlag off;

prefix 3ffe:0302:0011:0002::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
};
</pre>

Then, start radvd:

[host-node]# /etc/init.d/radvd start

==== Add IPv6 addresses to devices in [[CT0]] ====
<pre>
[host-node]# ip addr add dev veth101.0 3ffe:2400::212:34ff:fe56:789a/64
[host-node]# ip addr add dev eth0 3ffe:0302:0011:0002:211:22ff:fe33:4455/64
</pre>

=== Virtual Ethernet devices can be joined in one bridge ===
Perform steps 1 - 4 from Simple configuration chapter for several containers and/or veth devices

==== Create bridge device ====
<pre>
[host-node]# brctl addbr vzbr0
</pre>

==== Add veth devices to bridge ====
<pre>
[host-node]# brctl addif vzbr0 veth101.0
...
[host-node]# brctl addif vzbr0 veth101.n
[host-node]# brctl addif vzbr0 veth102.0
...
...
[host-node]# brctl addif vzbr0 vethXXX.N
</pre>

==== Configure bridge device ====
<pre>
[host-node]# ifconfig vzbr0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/vzbr0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/vzbr0/proxy_arp
</pre>

==== Add routes in [[CT0]] ====
<pre>
[host-node]# ip route add 192.168.101.1 dev vzbr0
...
[host-node]# ip route add 192.168.101.n dev vzbr0
[host-node]# ip route add 192.168.102.1 dev vzbr0
...
...
[host-node]# ip route add 192.168.XXX.N dev vzbr0
</pre>

Thus you'll have more convinient configuration, i.e. all routes to containers will be through this bridge and containers can communicate with each other even without these routes.

=== Making a veth-device persistent ===
According to http://bugzilla.openvz.org/show_bug.cgi?id=301 , a bug that stopped the veth device persistent was "Obsoleted now when --veth_add/--veth_del are introduced"

See http://wiki.openvz.org/w/index.php?title=Virtual_Ethernet_device&diff=5990&oldid=5989#Making_a_veth-device_persistent for a workaround that used to be described in this section.

That's it! At this point, when you restart the CT you should see a new line in the output, indicating that the interface is being configured and a new route being added. And you should be able to ping the host, and to enter the CT and use the network.

=== Making a bridged veth-device persistent ===

Like the above example, here it is how to add the veth device to a bridge in a persistent way.

==== method for vzctl version > 3.0.22 ====

Newer versions of vzctl includes a 'vznetaddbr' script, which makes use of the new <bridge> parameter of the --netif_add switch.

Just create /etc/vz/vznet.conf containing the following.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetaddbr"
</pre>

The script uses 'vmbr0' as default bridge name when no bridge is specified.

==== method for vzctl version <= 3.0.22 ====

Older vzctl doesn't offer an automatic function to do this.

1. First, edit the CT's configuration to specify what is the host bridge , and to indicate that a custom script should be run when starting up a CT.
* Open up /etc/vz/conf/CTID.conf
* Comment out any IP_ADDRESS entries to prevent a CTNET-device from being created in the CT
* Add or change the entry CONFIG_CUSTOMIZED="yes"
* Add an entry VZHOSTBR="<bridge if>" which is the bridge interface (already configured and up), you want to extend.

2. Now to create that "custom script". The following helper script will check the configuration file for the bridge interface name and for the veth interface, and add the interface to the bridge. Create the script /usr/sbin/vznetaddbr to have the following, and then <code>chmod 0500 /usr/sbin/vznetaddbr</code> to make it executable.

<pre>
#!/bin/bash
# /usr/sbin/vznetaddbr
# a script to add virtual network interfaces (veth's) in a CT to a bridge on CT0

CONFIGFILE=/etc/vz/conf/$VEID.conf
. $CONFIGFILE
VZHOSTIF=`echo $NETIF |sed 's/^.*host_ifname=$.*$,.*$/\1/g'`

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE CT$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VZHOSTBR" ]; then
echo "According to $CONFIGFILE CT$VEID has no bridge interface configured."
exit 1
fi

echo "Adding interface $VZHOSTIF to bridge $VZHOSTBR on CT0 for CT$VEID"
/sbin/ifconfig $VZHOSTIF 0
echo 1 > /proc/sys/net/ipv4/conf/$VZHOSTIF/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/$VZHOSTIF/forwarding
/usr/sbin/brctl addif $VZHOSTBR $VZHOSTIF

exit 0
</pre>

3. Now create /etc/vz/vznet.conf containing the following. This is what defines the "custom script" as being the vznetaddbr which you just created.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetaddbr"
</pre>

This may not work for particularily old versions of vzctl, e.g., the version 3.0.11 that ships with Debian Etch. For those versions, you can try a hack: Use the custom script <code>/etc/vz/conf/$VID.mount</code> which is available, even in these old versions. But it gets called too early, before the networking has been set up. But it can start some background process, which waits and occasionally polls until $VZHOSTIF has become available. Here is one way to go about it:

<pre>
#!/bin/bash

CONFIGFILE="/etc/vz/conf/$VEID.conf"

if [ -f "$CONFIGFILE" ]
then
. "$CONFIGFILE"
VZHOSTIF=`echo $NETIF |sed 's/^.*host_ifname=$.*$,.*$/\1/g'`
export VZHOSTIF
export VZHOSTBR

# Fork into the background and try a few times,
# until the host side of the interface appears:
/bin/bash -c 'for i in 5 10 20 40 80 160
do
if ifconfig -a | grep -q "$VZHOSTIF"
then
exec /usr/sbin/vznetaddbr
else
sleep "$i"
fi
done
' &

# In the meantime, let the CT's start process continue,
# or else the interface will never appear:
exit 0
else
$0: Config file "$CONFIGFILE" does not exist.
exit 1
fi
</pre>

4. Of course, the CT's operating system will need to have . Consult the manual for your CT's OS for details.

When the CT is started, the veth specified in the NETIF value is added to the bridge specified. You can check this by doing <code>brctl show</code>

Inside the CT you can configure the interface statically or using dhcp, as a real interface attached to a switch on the lan.

=== Virtual Ethernet devices + VLAN ===
This configuration can be done by adding vlan device to the previous configuration.

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]
* [[Using private IPs for Hardware Nodes]]
* Troubleshooting: [[Bridge doesn't forward packets]]

== External links ==
* [http://www.tldp.org/HOWTO/Linux+IPv6-HOWTO/hints-daemons-radvd.html Linux IPv6 HOWTO, a chapter about radvd]
* [http://vireso.blogspot.com/2008/02/2-veth-with-2-brindges-on-openvz-at.html 2 veth with 2 bridges setup]

[[Category: Networking]]
[[Category: HOWTO]]

Virtual Ethernet device

2009-09-07T01:20:35Z

StalkR: /* Configure device in CT */ netmask missing

'''Virtual Ethernet device''' is an Ethernet-like device which can be used
inside a [[container]]. Unlike [[venet]] network device, [[veth]] device
has a MAC address, therefore it can be used in configurations, when veth
is bridged to ethX or other device and container's user fully sets up
his networking himself, including IPs, gateways etc.

Virtual Ethernet device consist of two Ethernet devices --
the one in [[CT0]] and another one in CT. These devices are connected
to each other, so if a packet goes to one
device it will come out from the other device.

== Virtual Ethernet device usage ==

=== Kernel module ===
First of all, make sure the <code>vzethdev</code> module is loaded:
<pre>
# lsmod | grep vzeth
vzethdev 8224 0
vzmon 35164 5 vzethdev,vznetdev,vzrst,vzcpt
vzdev 3080 4 vzethdev,vznetdev,vzmon,vzdquota
</pre>

In case it is not loaded, load it:
<pre>
# modprobe vzethdev
</pre>

{{Note|in vzctl < 3.0.11, vzethdev is not autoloaded by <code>/etc/init.d/vz</code> script, so you have to edit it to load this module.}}

=== MAC addresses ===
In the below commands, you should use random MAC addresses. Do not use MAC addresses of real eth devices, because this can lead to collisions.

MAC addresses must be entered in XX:XX:XX:XX:XX:XX format.

YOU MAY NOT NEED TO GENERATE MAC ADDRESSES BY HAND BECAUSE vzctl --veth_add
MAY GENERATE THEM AUTOMATICALLY AS NECESSARY.

Nevertheless, there is a utility script available for generating MAC addresses: http://www.easyvmx.com/software/easymac.sh. It is to be used like this:

chmod +x easymac.sh
./easymac.sh -R

=== Adding veth to a CT ===

==== syntax vzctl version > 3.0.22 ====

vzctl set <CTID> --netif_add <ifname>[,<mac>,<host_ifname>,<host_mac>,<bridge>]

Here
* <tt>ifname</tt> is the Ethernet device name in the CT
* <tt>mac</tt> is its MAC address in the CT
* <tt>host_ifname</tt> is the Ethernet device name on the host ([[CT0]])
* <tt>host_mac</tt> is its MAC address on the host ([[CT0]])
* <tt>bridge</tt> is an optional parameter which can be used in custom network start scripts to automatically add the interface to a bridge.

{{Note|All parameters except <code>ifname</code> are optional and are automatically generated if not specified.}}

Example:

vzctl set 101 --netif_add eth0 --save

Or, if you want to specify everything:

vzctl set 101 --netif_add eth0,00:12:34:56:78:9A,veth101.0,00:12:34:56:78:9B --save

Or, if you want to specify the bridge and leave the other values autogenerated:

vzctl set 101 --netif_add eth0,,,,vmbr1 --save

==== syntax vzctl version >= 3.0.14 ====

Syntax is the same as above, but without a <bridge> parameter.

==== syntax vzctl version < 3.0.14 ====

vzctl set <CTID> --veth_add <dev_name>,<dev_addr>,<ve_dev_name>,<ve_dev_addr>

Here
* <tt>dev_name</tt> is the Ethernet device name that you are creating on the [[CT0|host system]]
* <tt>dev_addr</tt> is its MAC address
* <tt>ve_dev_name</tt> is the corresponding Ethernet device name you are creating on the CT
* <tt>ve_dev_addr</tt> is its MAC address

{{Note|this option is incremental, so devices are added to already existing ones.}}

NB there should no spaces after the commas.

Example:
<pre>
[host-node] ifconfig eth0
...
HWaddress 00:12:34:56:78:9B
...
</pre>

[host-node] easymac.sh -R
00:12:34:56:78:9A

vzctl set 101 --veth_add veth101.0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save

After executing this command <tt>veth</tt> device will be created for CT 101 and veth configuration will be saved to a CT configuration file.
Host-side Ethernet device will have <tt>veth101.0</tt> name and <tt>00:12:34:56:78:9A</tt> MAC address.
CT-side Ethernet device will have <tt>eth0</tt> name and <tt>00:12:34:56:78:9B</tt> MAC address.

=== Removing veth from a CT ===

==== syntax vzctl version >= 3.0.14 ====

vzctl set <CTID> --netif_del <dev_name>|all

Here
* <code>dev_name</code> is the Ethernet device name in the [[CT]].

{{Note|If you want to remove all Ethernet devices in CT, use <code>all</code>.}}

Example:

vzctl set 101 --netif_del eth0 --save

==== syntax vzctl version < 3.0.14 ====

vzctl set <CTID> --veth_del <dev_name>

Here <tt>dev_name</tt> is the Ethernet device name in the [[CT0|host system]].

Example:

vzctl set 101 --veth_del veth101.0 --save

After executing this command veth device with host-side Ethernet name
<code>veth101.0</code> will be removed from CT101 and veth configuration
will be updated in CT config file.

== Common configurations with virtual Ethernet devices ==
Module <tt>vzethdev</tt> must be loaded to operate with veth devices.

=== Simple configuration with virtual Ethernet device ===

Assuming you have 192.168.0.0/24 on your LAN, you will learn how to integrate a container in this LAN using veth.

==== Start a CT ====

[host-node]# vzctl start 101

==== Add veth device to CT ====

[host-node]# vzctl set 101 --netif_add eth0 --save

This allocates a MAC address and associates it with the host eth0 port.

==== Configure devices in CT0 ====
<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/proxy_arp
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
</pre>

Notes:
* These files did not exist for me when trying ([[User:Mrjcleaver|Mrjcleaver]] 14:04, 31 May 2008 (EDT))

==== Configure device in CT ====
<pre>
[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0
[ve-101]# /sbin/ip addr add 192.168.0.101 dev eth0
[ve-101]# /sbin/ip route add default dev eth0
</pre>

Notes:
* Until you ifconfig eth0 it won't appear. When you do it will use the mac address netif_add added earlier
* 192.168.0.101 is chosen to be an [[unrouteable private ip address]]. Where 101 reminds you that it is node 101.
* The "ip route" tells all traffic to head to "device eth0"
* In theory you could [[use dhcpd with OpenVZ]] and dhclient to pick up an DHCP address from your router instead of hardwiring it
** http://openvz.org/pipermail/users/2005-November/000020.html

==== Add route in [[CT0]] ====

[host-node]# ip route add 192.168.0.101 dev veth101.0

=== Using a directly routed IPv4 with virtual Ethernet device ===

==== Situation ====
Hardware Node (HN/CT0) has 192.168.0.1/24 with router 192.168.0.254.

We also know that IPv4 10.0.0.1/32 is directly routed to 192.168.0.1 (this is called a ''fail-over IP'').

We want to give this directly routed IPv4 address to a container (CT).

==== Start container ====

[host-node]# vzctl start 101

==== Add veth device to CT ====

[host-node]# vzctl set 101 --netif_add eth0 --save

This allocates a MAC address and associates it with the host eth0 port.

==== Configure device and add route in CT0 ====

<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# ip route add 10.0.0.1 dev veth101.0
</pre>

Make sure that you have IPv4 forwarding enabled.
<pre>
[host-node]# echo 1 > /proc/sys/net/ipv4/ip_forward
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
</pre>
You can permanently set this by using <tt>/etc/sysctl.conf</tt>.

==== Configure device in CT ====

1. Configure IP address

2. Add gateway

3. Add default route

<pre>
[ve-101]# /sbin/ifconfig eth0 10.0.0.1 netmask 255.255.255.255
[ve-101]# /sbin/ip route add 192.168.0.1 dev eth0
[ve-101]# /sbin/ip route default via 192.168.0.1
</pre>

In a Debian container, you can configure this permanently by using <tt>/etc/network/interfaces</tt>:
<pre>
auto eth0
iface eth0 inet static
address 10.0.0.1
netmask 255.255.255.255
up /sbin/ip route add 192.168.0.1 dev eth0
up /sbin/ip route add default via 192.168.0.1
</pre>

=== Virtual Ethernet device with IPv6 ===

==== Start a [[CT]] ====

[host-node]# vzctl start 101

==== Add veth device to the [[CT]] ====

[host-node]# vzctl set 101 --veth_add eth0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save

==== Configure devices in the [[CT0]] ====

[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/all/forwarding

==== Configure device in [[CT]] ====

[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0

==== Start router advertisement daemon (radvd) for IPv6 in CT0 ====
First you need to edit radvd configuration file. Here is a simple example of <tt>/etc/radv.conf</tt>:
<pre>
interface veth101.0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvHomeAgentFlag off;

prefix 3ffe:2400:0:0::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
};

interface eth0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvHomeAgentFlag off;

prefix 3ffe:0302:0011:0002::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
};
</pre>

Then, start radvd:

[host-node]# /etc/init.d/radvd start

==== Add IPv6 addresses to devices in [[CT0]] ====
<pre>
[host-node]# ip addr add dev veth101.0 3ffe:2400::212:34ff:fe56:789a/64
[host-node]# ip addr add dev eth0 3ffe:0302:0011:0002:211:22ff:fe33:4455/64
</pre>

=== Virtual Ethernet devices can be joined in one bridge ===
Perform steps 1 - 4 from Simple configuration chapter for several containers and/or veth devices

==== Create bridge device ====
<pre>
[host-node]# brctl addbr vzbr0
</pre>

==== Add veth devices to bridge ====
<pre>
[host-node]# brctl addif vzbr0 veth101.0
...
[host-node]# brctl addif vzbr0 veth101.n
[host-node]# brctl addif vzbr0 veth102.0
...
...
[host-node]# brctl addif vzbr0 vethXXX.N
</pre>

==== Configure bridge device ====
<pre>
[host-node]# ifconfig vzbr0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/vzbr0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/vzbr0/proxy_arp
</pre>

==== Add routes in [[CT0]] ====
<pre>
[host-node]# ip route add 192.168.101.1 dev vzbr0
...
[host-node]# ip route add 192.168.101.n dev vzbr0
[host-node]# ip route add 192.168.102.1 dev vzbr0
...
...
[host-node]# ip route add 192.168.XXX.N dev vzbr0
</pre>

Thus you'll have more convinient configuration, i.e. all routes to containers will be through this bridge and containers can communicate with each other even without these routes.

=== Making a veth-device persistent ===
According to http://bugzilla.openvz.org/show_bug.cgi?id=301 , a bug that stopped the veth device persistent was "Obsoleted now when --veth_add/--veth_del are introduced"

See http://wiki.openvz.org/w/index.php?title=Virtual_Ethernet_device&diff=5990&oldid=5989#Making_a_veth-device_persistent for a workaround that used to be described in this section.

That's it! At this point, when you restart the CT you should see a new line in the output, indicating that the interface is being configured and a new route being added. And you should be able to ping the host, and to enter the CT and use the network.

=== Making a bridged veth-device persistent ===

Like the above example, here it is how to add the veth device to a bridge in a persistent way.

==== method for vzctl version > 3.0.22 ====

Newer versions of vzctl includes a 'vznetaddbr' script, which makes use of the new <bridge> parameter of the --netif_add switch.

Just create /etc/vz/vznet.conf containing the following.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetaddbr"
</pre>

The script uses 'vmbr0' as default bridge name when no bridge is specified.

==== method for vzctl version <= 3.0.22 ====

Older vzctl doesn't offer an automatic function to do this.

1. First, edit the CT's configuration to specify what is the host bridge , and to indicate that a custom script should be run when starting up a CT.
* Open up /etc/vz/conf/CTID.conf
* Comment out any IP_ADDRESS entries to prevent a CTNET-device from being created in the CT
* Add or change the entry CONFIG_CUSTOMIZED="yes"
* Add an entry VZHOSTBR="<bridge if>" which is the bridge interface (already configured and up), you want to extend.

2. Now to create that "custom script". The following helper script will check the configuration file for the bridge interface name and for the veth interface, and add the interface to the bridge. Create the script /usr/sbin/vznetaddbr to have the following, and then <code>chmod 0500 /usr/sbin/vznetaddbr</code> to make it executable.

<pre>
#!/bin/bash
# /usr/sbin/vznetaddbr
# a script to add virtual network interfaces (veth's) in a CT to a bridge on CT0

CONFIGFILE=/etc/vz/conf/$VEID.conf
. $CONFIGFILE
VZHOSTIF=`echo $NETIF |sed 's/^.*host_ifname=$.*$,.*$/\1/g'`

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE CT$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VZHOSTBR" ]; then
echo "According to $CONFIGFILE CT$VEID has no bridge interface configured."
exit 1
fi

echo "Adding interface $VZHOSTIF to bridge $VZHOSTBR on CT0 for CT$VEID"
/sbin/ifconfig $VZHOSTIF 0
echo 1 > /proc/sys/net/ipv4/conf/$VZHOSTIF/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/$VZHOSTIF/forwarding
/usr/sbin/brctl addif $VZHOSTBR $VZHOSTIF

exit 0
</pre>

3. Now create /etc/vz/vznet.conf containing the following. This is what defines the "custom script" as being the vznetaddbr which you just created.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetaddbr"
</pre>

This may not work for particularily old versions of vzctl, e.g., the version 3.0.11 that ships with Debian Etch. For those versions, you can try a hack: Use the custom script <code>/etc/vz/conf/$VID.mount</code> which is available, even in these old versions. But it gets called too early, before the networking has been set up. But it can start some background process, which waits and occasionally polls until $VZHOSTIF has become available. Here is one way to go about it:

<pre>
#!/bin/bash

CONFIGFILE="/etc/vz/conf/$VEID.conf"

if [ -f "$CONFIGFILE" ]
then
. "$CONFIGFILE"
VZHOSTIF=`echo $NETIF |sed 's/^.*host_ifname=$.*$,.*$/\1/g'`
export VZHOSTIF
export VZHOSTBR

# Fork into the background and try a few times,
# until the host side of the interface appears:
/bin/bash -c 'for i in 5 10 20 40 80 160
do
if ifconfig -a | grep -q "$VZHOSTIF"
then
exec /usr/sbin/vznetaddbr
else
sleep "$i"
fi
done
' &

# In the meantime, let the CT's start process continue,
# or else the interface will never appear:
exit 0
else
$0: Config file "$CONFIGFILE" does not exist.
exit 1
fi
</pre>

4. Of course, the CT's operating system will need to have . Consult the manual for your CT's OS for details.

When the CT is started, the veth specified in the NETIF value is added to the bridge specified. You can check this by doing <code>brctl show</code>

Inside the CT you can configure the interface statically or using dhcp, as a real interface attached to a switch on the lan.

=== Virtual Ethernet devices + VLAN ===
This configuration can be done by adding vlan device to the previous configuration.

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]
* [[Using private IPs for Hardware Nodes]]
* Troubleshooting: [[Bridge doesn't forward packets]]

== External links ==
* [http://www.tldp.org/HOWTO/Linux+IPv6-HOWTO/hints-daemons-radvd.html Linux IPv6 HOWTO, a chapter about radvd]
* [http://vireso.blogspot.com/2008/02/2-veth-with-2-brindges-on-openvz-at.html 2 veth with 2 bridges setup]

[[Category: Networking]]
[[Category: HOWTO]]

Virtual Ethernet device

2009-09-07T01:19:05Z

StalkR: /* Common configurations with virtual Ethernet devices */ new section : Using a directly routed IPv4 with virtual Ethernet device

'''Virtual Ethernet device''' is an Ethernet-like device which can be used
inside a [[container]]. Unlike [[venet]] network device, [[veth]] device
has a MAC address, therefore it can be used in configurations, when veth
is bridged to ethX or other device and container's user fully sets up
his networking himself, including IPs, gateways etc.

Virtual Ethernet device consist of two Ethernet devices --
the one in [[CT0]] and another one in CT. These devices are connected
to each other, so if a packet goes to one
device it will come out from the other device.

== Virtual Ethernet device usage ==

=== Kernel module ===
First of all, make sure the <code>vzethdev</code> module is loaded:
<pre>
# lsmod | grep vzeth
vzethdev 8224 0
vzmon 35164 5 vzethdev,vznetdev,vzrst,vzcpt
vzdev 3080 4 vzethdev,vznetdev,vzmon,vzdquota
</pre>

In case it is not loaded, load it:
<pre>
# modprobe vzethdev
</pre>

{{Note|in vzctl < 3.0.11, vzethdev is not autoloaded by <code>/etc/init.d/vz</code> script, so you have to edit it to load this module.}}

=== MAC addresses ===
In the below commands, you should use random MAC addresses. Do not use MAC addresses of real eth devices, because this can lead to collisions.

MAC addresses must be entered in XX:XX:XX:XX:XX:XX format.

YOU MAY NOT NEED TO GENERATE MAC ADDRESSES BY HAND BECAUSE vzctl --veth_add
MAY GENERATE THEM AUTOMATICALLY AS NECESSARY.

Nevertheless, there is a utility script available for generating MAC addresses: http://www.easyvmx.com/software/easymac.sh. It is to be used like this:

chmod +x easymac.sh
./easymac.sh -R

=== Adding veth to a CT ===

==== syntax vzctl version > 3.0.22 ====

vzctl set <CTID> --netif_add <ifname>[,<mac>,<host_ifname>,<host_mac>,<bridge>]

Here
* <tt>ifname</tt> is the Ethernet device name in the CT
* <tt>mac</tt> is its MAC address in the CT
* <tt>host_ifname</tt> is the Ethernet device name on the host ([[CT0]])
* <tt>host_mac</tt> is its MAC address on the host ([[CT0]])
* <tt>bridge</tt> is an optional parameter which can be used in custom network start scripts to automatically add the interface to a bridge.

{{Note|All parameters except <code>ifname</code> are optional and are automatically generated if not specified.}}

Example:

vzctl set 101 --netif_add eth0 --save

Or, if you want to specify everything:

vzctl set 101 --netif_add eth0,00:12:34:56:78:9A,veth101.0,00:12:34:56:78:9B --save

Or, if you want to specify the bridge and leave the other values autogenerated:

vzctl set 101 --netif_add eth0,,,,vmbr1 --save

==== syntax vzctl version >= 3.0.14 ====

Syntax is the same as above, but without a <bridge> parameter.

==== syntax vzctl version < 3.0.14 ====

vzctl set <CTID> --veth_add <dev_name>,<dev_addr>,<ve_dev_name>,<ve_dev_addr>

Here
* <tt>dev_name</tt> is the Ethernet device name that you are creating on the [[CT0|host system]]
* <tt>dev_addr</tt> is its MAC address
* <tt>ve_dev_name</tt> is the corresponding Ethernet device name you are creating on the CT
* <tt>ve_dev_addr</tt> is its MAC address

{{Note|this option is incremental, so devices are added to already existing ones.}}

NB there should no spaces after the commas.

Example:
<pre>
[host-node] ifconfig eth0
...
HWaddress 00:12:34:56:78:9B
...
</pre>

[host-node] easymac.sh -R
00:12:34:56:78:9A

vzctl set 101 --veth_add veth101.0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save

After executing this command <tt>veth</tt> device will be created for CT 101 and veth configuration will be saved to a CT configuration file.
Host-side Ethernet device will have <tt>veth101.0</tt> name and <tt>00:12:34:56:78:9A</tt> MAC address.
CT-side Ethernet device will have <tt>eth0</tt> name and <tt>00:12:34:56:78:9B</tt> MAC address.

=== Removing veth from a CT ===

==== syntax vzctl version >= 3.0.14 ====

vzctl set <CTID> --netif_del <dev_name>|all

Here
* <code>dev_name</code> is the Ethernet device name in the [[CT]].

{{Note|If you want to remove all Ethernet devices in CT, use <code>all</code>.}}

Example:

vzctl set 101 --netif_del eth0 --save

==== syntax vzctl version < 3.0.14 ====

vzctl set <CTID> --veth_del <dev_name>

Here <tt>dev_name</tt> is the Ethernet device name in the [[CT0|host system]].

Example:

vzctl set 101 --veth_del veth101.0 --save

After executing this command veth device with host-side Ethernet name
<code>veth101.0</code> will be removed from CT101 and veth configuration
will be updated in CT config file.

== Common configurations with virtual Ethernet devices ==
Module <tt>vzethdev</tt> must be loaded to operate with veth devices.

=== Simple configuration with virtual Ethernet device ===

Assuming you have 192.168.0.0/24 on your LAN, you will learn how to integrate a container in this LAN using veth.

==== Start a CT ====

[host-node]# vzctl start 101

==== Add veth device to CT ====

[host-node]# vzctl set 101 --netif_add eth0 --save

This allocates a MAC address and associates it with the host eth0 port.

==== Configure devices in CT0 ====
<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/proxy_arp
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
</pre>

Notes:
* These files did not exist for me when trying ([[User:Mrjcleaver|Mrjcleaver]] 14:04, 31 May 2008 (EDT))

==== Configure device in CT ====
<pre>
[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0
[ve-101]# /sbin/ip addr add 192.168.0.101 dev eth0
[ve-101]# /sbin/ip route add default dev eth0
</pre>

Notes:
* Until you ifconfig eth0 it won't appear. When you do it will use the mac address netif_add added earlier
* 192.168.0.101 is chosen to be an [[unrouteable private ip address]]. Where 101 reminds you that it is node 101.
* The "ip route" tells all traffic to head to "device eth0"
* In theory you could [[use dhcpd with OpenVZ]] and dhclient to pick up an DHCP address from your router instead of hardwiring it
** http://openvz.org/pipermail/users/2005-November/000020.html

==== Add route in [[CT0]] ====

[host-node]# ip route add 192.168.0.101 dev veth101.0

=== Using a directly routed IPv4 with virtual Ethernet device ===

==== Situation ====
Hardware Node (HN/CT0) has 192.168.0.1/24 with router 192.168.0.254.

We also know that IPv4 10.0.0.1/32 is directly routed to 192.168.0.1 (this is called a ''fail-over IP'').

We want to give this directly routed IPv4 address to a container (CT).

==== Start container ====

[host-node]# vzctl start 101

==== Add veth device to CT ====

[host-node]# vzctl set 101 --netif_add eth0 --save

This allocates a MAC address and associates it with the host eth0 port.

==== Configure device and add route in CT0 ====

<pre>
[host-node]# ifconfig veth101.0 0
[host-node]# ip route add 10.0.0.1 dev veth101.0
</pre>

Make sure that you have IPv4 forwarding enabled.
<pre>
[host-node]# echo 1 > /proc/sys/net/ipv4/ip_forward
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/veth101.0/forwarding
</pre>
You can permanently set this by using <tt>/etc/sysctl.conf</tt>.

==== Configure device in CT ====

1. Configure IP address

2. Add gateway

3. Add default route

<pre>
[ve-101]# /sbin/ifconfig eth0 10.0.0.1
[ve-101]# /sbin/ip route add 192.168.0.1 dev eth0
[ve-101]# /sbin/ip route default via 192.168.0.1
</pre>

In a Debian container, you can configure this permanently by using <tt>/etc/network/interfaces</tt>:
<pre>
auto eth0
iface eth0 inet static
address 10.0.0.1
netmask 255.255.255.255
up /sbin/ip route add 192.168.0.1 dev eth0
up /sbin/ip route add default via 192.168.0.1
</pre>

=== Virtual Ethernet device with IPv6 ===

==== Start a [[CT]] ====

[host-node]# vzctl start 101

==== Add veth device to the [[CT]] ====

[host-node]# vzctl set 101 --veth_add eth0,00:12:34:56:78:9A,eth0,00:12:34:56:78:9B --save

==== Configure devices in the [[CT0]] ====

[host-node]# ifconfig veth101.0 0
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/veth101.0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/eth0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv6/conf/all/forwarding

==== Configure device in [[CT]] ====

[host-node]# vzctl enter 101
[ve-101]# /sbin/ifconfig eth0 0

==== Start router advertisement daemon (radvd) for IPv6 in CT0 ====
First you need to edit radvd configuration file. Here is a simple example of <tt>/etc/radv.conf</tt>:
<pre>
interface veth101.0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvHomeAgentFlag off;

prefix 3ffe:2400:0:0::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
};

interface eth0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvHomeAgentFlag off;

prefix 3ffe:0302:0011:0002::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
};
</pre>

Then, start radvd:

[host-node]# /etc/init.d/radvd start

==== Add IPv6 addresses to devices in [[CT0]] ====
<pre>
[host-node]# ip addr add dev veth101.0 3ffe:2400::212:34ff:fe56:789a/64
[host-node]# ip addr add dev eth0 3ffe:0302:0011:0002:211:22ff:fe33:4455/64
</pre>

=== Virtual Ethernet devices can be joined in one bridge ===
Perform steps 1 - 4 from Simple configuration chapter for several containers and/or veth devices

==== Create bridge device ====
<pre>
[host-node]# brctl addbr vzbr0
</pre>

==== Add veth devices to bridge ====
<pre>
[host-node]# brctl addif vzbr0 veth101.0
...
[host-node]# brctl addif vzbr0 veth101.n
[host-node]# brctl addif vzbr0 veth102.0
...
...
[host-node]# brctl addif vzbr0 vethXXX.N
</pre>

==== Configure bridge device ====
<pre>
[host-node]# ifconfig vzbr0 0
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/vzbr0/forwarding
[host-node]# echo 1 > /proc/sys/net/ipv4/conf/vzbr0/proxy_arp
</pre>

==== Add routes in [[CT0]] ====
<pre>
[host-node]# ip route add 192.168.101.1 dev vzbr0
...
[host-node]# ip route add 192.168.101.n dev vzbr0
[host-node]# ip route add 192.168.102.1 dev vzbr0
...
...
[host-node]# ip route add 192.168.XXX.N dev vzbr0
</pre>

Thus you'll have more convinient configuration, i.e. all routes to containers will be through this bridge and containers can communicate with each other even without these routes.

=== Making a veth-device persistent ===
According to http://bugzilla.openvz.org/show_bug.cgi?id=301 , a bug that stopped the veth device persistent was "Obsoleted now when --veth_add/--veth_del are introduced"

See http://wiki.openvz.org/w/index.php?title=Virtual_Ethernet_device&diff=5990&oldid=5989#Making_a_veth-device_persistent for a workaround that used to be described in this section.

That's it! At this point, when you restart the CT you should see a new line in the output, indicating that the interface is being configured and a new route being added. And you should be able to ping the host, and to enter the CT and use the network.

=== Making a bridged veth-device persistent ===

Like the above example, here it is how to add the veth device to a bridge in a persistent way.

==== method for vzctl version > 3.0.22 ====

Newer versions of vzctl includes a 'vznetaddbr' script, which makes use of the new <bridge> parameter of the --netif_add switch.

Just create /etc/vz/vznet.conf containing the following.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetaddbr"
</pre>

The script uses 'vmbr0' as default bridge name when no bridge is specified.

==== method for vzctl version <= 3.0.22 ====

Older vzctl doesn't offer an automatic function to do this.

1. First, edit the CT's configuration to specify what is the host bridge , and to indicate that a custom script should be run when starting up a CT.
* Open up /etc/vz/conf/CTID.conf
* Comment out any IP_ADDRESS entries to prevent a CTNET-device from being created in the CT
* Add or change the entry CONFIG_CUSTOMIZED="yes"
* Add an entry VZHOSTBR="<bridge if>" which is the bridge interface (already configured and up), you want to extend.

2. Now to create that "custom script". The following helper script will check the configuration file for the bridge interface name and for the veth interface, and add the interface to the bridge. Create the script /usr/sbin/vznetaddbr to have the following, and then <code>chmod 0500 /usr/sbin/vznetaddbr</code> to make it executable.

<pre>
#!/bin/bash
# /usr/sbin/vznetaddbr
# a script to add virtual network interfaces (veth's) in a CT to a bridge on CT0

CONFIGFILE=/etc/vz/conf/$VEID.conf
. $CONFIGFILE
VZHOSTIF=`echo $NETIF |sed 's/^.*host_ifname=$.*$,.*$/\1/g'`

if [ ! -n "$VZHOSTIF" ]; then
echo "According to $CONFIGFILE CT$VEID has no veth interface configured."
exit 1
fi

if [ ! -n "$VZHOSTBR" ]; then
echo "According to $CONFIGFILE CT$VEID has no bridge interface configured."
exit 1
fi

echo "Adding interface $VZHOSTIF to bridge $VZHOSTBR on CT0 for CT$VEID"
/sbin/ifconfig $VZHOSTIF 0
echo 1 > /proc/sys/net/ipv4/conf/$VZHOSTIF/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/$VZHOSTIF/forwarding
/usr/sbin/brctl addif $VZHOSTBR $VZHOSTIF

exit 0
</pre>

3. Now create /etc/vz/vznet.conf containing the following. This is what defines the "custom script" as being the vznetaddbr which you just created.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/sbin/vznetaddbr"
</pre>

This may not work for particularily old versions of vzctl, e.g., the version 3.0.11 that ships with Debian Etch. For those versions, you can try a hack: Use the custom script <code>/etc/vz/conf/$VID.mount</code> which is available, even in these old versions. But it gets called too early, before the networking has been set up. But it can start some background process, which waits and occasionally polls until $VZHOSTIF has become available. Here is one way to go about it:

<pre>
#!/bin/bash

CONFIGFILE="/etc/vz/conf/$VEID.conf"

if [ -f "$CONFIGFILE" ]
then
. "$CONFIGFILE"
VZHOSTIF=`echo $NETIF |sed 's/^.*host_ifname=$.*$,.*$/\1/g'`
export VZHOSTIF
export VZHOSTBR

# Fork into the background and try a few times,
# until the host side of the interface appears:
/bin/bash -c 'for i in 5 10 20 40 80 160
do
if ifconfig -a | grep -q "$VZHOSTIF"
then
exec /usr/sbin/vznetaddbr
else
sleep "$i"
fi
done
' &

# In the meantime, let the CT's start process continue,
# or else the interface will never appear:
exit 0
else
$0: Config file "$CONFIGFILE" does not exist.
exit 1
fi
</pre>

4. Of course, the CT's operating system will need to have . Consult the manual for your CT's OS for details.

When the CT is started, the veth specified in the NETIF value is added to the bridge specified. You can check this by doing <code>brctl show</code>

Inside the CT you can configure the interface statically or using dhcp, as a real interface attached to a switch on the lan.

=== Virtual Ethernet devices + VLAN ===
This configuration can be done by adding vlan device to the previous configuration.

== See also ==
* [[Virtual network device]]
* [[Differences between venet and veth]]
* [[Using private IPs for Hardware Nodes]]
* Troubleshooting: [[Bridge doesn't forward packets]]

== External links ==
* [http://www.tldp.org/HOWTO/Linux+IPv6-HOWTO/hints-daemons-radvd.html Linux IPv6 HOWTO, a chapter about radvd]
* [http://vireso.blogspot.com/2008/02/2-veth-with-2-brindges-on-openvz-at.html 2 veth with 2 bridges setup]

[[Category: Networking]]
[[Category: HOWTO]]