OpenVZ Virtuozzo Containers Wiki - User contributions [en]

Multiple network interfaces and ARP flux

2007-02-18T18:01:24Z

Estellt: Added what version of vzctl is needed.

==Overview==
This page discusses working with multiple network interfaces on the Hardware Node (HN), how this results in ARP Flux, and how to address this.

==The Simple Case==
In the simple case you have multiple network interfaces on the HN, all with IP addresses in the same subnet. Each of your Virtual Environments (VE's) also have IP addresses in the same subnet. You don't care which interfaces your VE's use.

So, no action is required. Everything just works. Setup OpenVZ normally.

The only downside is '''ARP flux'''. This describes the usually harmless condition where the network address (layer 3) drifts between multiple hardware addresses (layer 2). While this may cause some confusion to anyone trouble shooting, or generate alarms on network monitoring systems, it doesn't interrupt network traffic.

For an example of what this may look like, see the example and tcpdump captures below.

==A More Complex Case==
Let's say you have three network interfaces on the HN, all with IP addresses on the same subnet. Each of your VE's also have IP addresses on the same subnet. But now you ''do'' care which interface your VE's use.

For example, you want some of your VE's to always use eth3, and some to use eth4. But none of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.

===Example Network Setup===
To make this clear we'll use the following HN configuration. We'll also have another system to act as the client.

{| align="center" border="1" cellpadding=5
! System !! Interface !! MAC Address !! IP Address
|-
| HN || eth0 || 00:0c:29:b3:a2:54 || 192.168.18.10
|-
| HN || eth3 || 00:0c:29:b3:a2:68 || 192.168.18.11
|-
| HN || eth4 || 00:0c:29:b3:a2:5e || 192.168.18.12
|-
| client || eth0 || 00:0c:29:d2:c7:aa || 192.168.18.129
|}

===HN ARP Flux===
The first issue is fixing the '''ARP flux''' noted above. Any client on the network broadcasting an ARP "who has" message for any of the HN addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.

====Example One - HN ARP Flux====
For example, the following is a tcpdump capture from executing <code>ping -c2 192.168.18.10</code> from the client system.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:68
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The ARP "who has" message generated replies from all three MAC addresses on the HN. In this case the client took the MAC address for eth4. The three ICMP messages are then sent to eth4, but all the replies come from eth0. Normally this behavior isn't a problem, though it may generate some false alarms for a network monitor as it appears someone could be executing a man in the middle attack.

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

====A Simple Fix That May Work====
If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:

<pre>sysctl -w net.ipv4.conf.all.arp_filter=1</pre>

However, if they are all on the same IP network, which is the case here, then this won't achieve the desired results.

====A More Effective Solution====
The following can be added to your /etc/sysctl.conf file once you've tested it.

<pre>
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
</pre>

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

====Example Two - HN ARP Flux Corrected====
Now we repeat the ping command, after the arp cache on the client has been cleared.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The desired affect has been achieved. Only interface eth0 on the HN responds to the ARP message and the other interfaces are silent.

===Adding some VE's===
Now that the HN is behaving as expected, let's add some VE's and see what happens.

====VE Network Setup====
The case we're addressing is when the VE's are on the same subnet as the HN. So we create two new VE's and assign the addresses as follows.

{| align="center" border="1" cellpadding=5
! VEID || IP
|-
| 101 || 192.168.18.101
|-
| 102 || 192.168.18.102
|}

====Example Three - VE ARP Flux====
From the client system on you should be able to ping both VE's. However, looking at the ARP traffic with tcpdump you'll see that once again the network address associated with each VE will be subject to ARP flux, drifting between all three link layer addresses over time.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.101: ICMP echo request, id 43311, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.101 > 192.168.18.129: ICMP echo reply, id 43311, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.101: ICMP echo request, id 43311, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.101 > 192.168.18.129: ICMP echo reply, id 43311, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

====The ARP Cache====
The reasons for this can be found from executing the following command on the HN to display the ARP cache.

<pre>arp -an</pre>

<pre>
? (192.168.18.129) at 00:0C:29:D2:C7:AA [ether] on eth0
? (192.168.18.102) at <from_interface> PERM PUB on eth3
? (192.168.18.102) at <from_interface> PERM PUB on eth4
? (192.168.18.102) at <from_interface> PERM PUB on eth0
? (192.168.18.101) at <from_interface> PERM PUB on eth3
? (192.168.18.101) at <from_interface> PERM PUB on eth4
? (192.168.18.101) at <from_interface> PERM PUB on eth0
</pre>

Another view is obtained from the following command on the HN.

<pre>cat /proc/net/arp</pre>

<pre>
IP address HW type Flags HW address Mask Device
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth0
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth0
</pre>

What this shows is that each VE's IP address is associated with each HN's interface. Therefore each interface will respond to any ARP "who has" query.

====The Cause====
These entries are created by the vzarp function in the vps_functions script, which are called by vps-net_add, vps-net_del and vps-stop. The result of this function in our case is to execute the following commands:

<pre>
/sbin/ip neigh add proxy 192.168.18.101 dev eth0
/sbin/ip neigh add proxy 192.168.18.101 dev eth4
/sbin/ip neigh add proxy 192.168.18.101 dev eth3
/sbin/ip neigh add proxy 192.168.18.102 dev eth0
/sbin/ip neigh add proxy 192.168.18.102 dev eth4
/sbin/ip neigh add proxy 192.168.18.102 dev eth3
</pre>

In addition, the following ARP messages are sent when VEID 101 is started.

<pre>
00:0c:29:b3:a2:54 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.10
00:0c:29:b3:a2:5e > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.12
00:0c:29:b3:a2:68 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.11
00:0c:29:b3:a2:54 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:5e > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:68 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:5e > 00:0c:29:b3:a2:68, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:5e > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:68 > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:68 > 00:0c:29:b3:a2:5e, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:54 > 00:0c:29:b3:a2:5e, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:54 > 00:0c:29:b3:a2:68, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
</pre>

What we see here is the result of vzarpipdetect, another function in vps_functions called by vps-net_add. An ARP "who has" message is sent by each interface and answered by the other interfaces.

What we want is to only add the IP addresses of our VE's to specific devices, not to all devices. This will prevent the ARP flux problem for our VE's.

====The Quick Fix====
Unfortunately this involves editing the OpenVZ scripts. The only case we really care about is vps-net_add, as the others execute <code>ip neigh del proxy</code>.

Manually editing the vzarp script is a quick fix, but not advised. Creating your own ''fork'' of OpenVZ is difficult to maintain and may have unintended side affects.

Fortunately there is a feature which will allow custom scripts to run during VE startup.

This approach is also described in http://wiki.openvz.org/Virtual_Ethernet_device.

In the VE configuration file, /etc/vz/conf/$VEID.conf, add the line

<pre>
CONFIG_CUSTOMIZED="yes"
</pre>

Now create the file /etc/vz/vznet.conf or /etc/vz/vznetcfg. Note that this will only work with a recent version of OpenVZ (vzctl-3.0.14) as the change was introduced in December, 2006. The file name seems to have changed between the two listed here so some trial and error may be required.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/lib/vzctl/scripts/vznet-custom"
</pre>

Finally create the file /usr/lib/vzctl/scripts/vznet-custom and add your custom commands.

TODO: Discuss contents of script and provide examples. Still need to work on getting this to work.

==Testing Environment==
All of the examples have been generated and tested using Debian Etch for the HN and Debian Stable for the VE's. VMware Workstation was used to create the test networks. The client is the BackTrack live CD from Remote Exploit. If you have different results from other releases of Linux please edit this page.

[[Category:HOWTO]]
[[Category:Networking]]
[[Category:Debian]]

Multiple network interfaces and ARP flux

2007-02-18T17:44:44Z

Estellt:

==Overview==
This page discusses working with multiple network interfaces on the Hardware Node (HN), how this results in ARP Flux, and how to address this.

==The Simple Case==
In the simple case you have multiple network interfaces on the HN, all with IP addresses in the same subnet. Each of your Virtual Environments (VE's) also have IP addresses in the same subnet. You don't care which interfaces your VE's use.

So, no action is required. Everything just works. Setup OpenVZ normally.

The only downside is '''ARP flux'''. This describes the usually harmless condition where the network address (layer 3) drifts between multiple hardware addresses (layer 2). While this may cause some confusion to anyone trouble shooting, or generate alarms on network monitoring systems, it doesn't interrupt network traffic.

For an example of what this may look like, see the example and tcpdump captures below.

==A More Complex Case==
Let's say you have three network interfaces on the HN, all with IP addresses on the same subnet. Each of your VE's also have IP addresses on the same subnet. But now you ''do'' care which interface your VE's use.

For example, you want some of your VE's to always use eth3, and some to use eth4. But none of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.

===Example Network Setup===
To make this clear we'll use the following HN configuration. We'll also have another system to act as the client.

{| align="center" border="1" cellpadding=5
! System !! Interface !! MAC Address !! IP Address
|-
| HN || eth0 || 00:0c:29:b3:a2:54 || 192.168.18.10
|-
| HN || eth3 || 00:0c:29:b3:a2:68 || 192.168.18.11
|-
| HN || eth4 || 00:0c:29:b3:a2:5e || 192.168.18.12
|-
| client || eth0 || 00:0c:29:d2:c7:aa || 192.168.18.129
|}

===HN ARP Flux===
The first issue is fixing the '''ARP flux''' noted above. Any client on the network broadcasting an ARP "who has" message for any of the HN addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.

====Example One - HN ARP Flux====
For example, the following is a tcpdump capture from executing <code>ping -c2 192.168.18.10</code> from the client system.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:68
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The ARP "who has" message generated replies from all three MAC addresses on the HN. In this case the client took the MAC address for eth4. The three ICMP messages are then sent to eth4, but all the replies come from eth0. Normally this behavior isn't a problem, though it may generate some false alarms for a network monitor as it appears someone could be executing a man in the middle attack.

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

====A Simple Fix That May Work====
If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:

<pre>sysctl -w net.ipv4.conf.all.arp_filter=1</pre>

However, if they are all on the same IP network, which is the case here, then this won't achieve the desired results.

====A More Effective Solution====
The following can be added to your /etc/sysctl.conf file once you've tested it.

<pre>
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
</pre>

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

====Example Two - HN ARP Flux Corrected====
Now we repeat the ping command, after the arp cache on the client has been cleared.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The desired affect has been achieved. Only interface eth0 on the HN responds to the ARP message and the other interfaces are silent.

===Adding some VE's===
Now that the HN is behaving as expected, let's add some VE's and see what happens.

====VE Network Setup====
The case we're addressing is when the VE's are on the same subnet as the HN. So we create two new VE's and assign the addresses as follows.

{| align="center" border="1" cellpadding=5
! VEID || IP
|-
| 101 || 192.168.18.101
|-
| 102 || 192.168.18.102
|}

====Example Three - VE ARP Flux====
From the client system on you should be able to ping both VE's. However, looking at the ARP traffic with tcpdump you'll see that once again the network address associated with each VE will be subject to ARP flux, drifting between all three link layer addresses over time.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.101: ICMP echo request, id 43311, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.101 > 192.168.18.129: ICMP echo reply, id 43311, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.101: ICMP echo request, id 43311, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.101 > 192.168.18.129: ICMP echo reply, id 43311, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

====The ARP Cache====
The reasons for this can be found from executing the following command on the HN to display the ARP cache.

<pre>arp -an</pre>

<pre>
? (192.168.18.129) at 00:0C:29:D2:C7:AA [ether] on eth0
? (192.168.18.102) at <from_interface> PERM PUB on eth3
? (192.168.18.102) at <from_interface> PERM PUB on eth4
? (192.168.18.102) at <from_interface> PERM PUB on eth0
? (192.168.18.101) at <from_interface> PERM PUB on eth3
? (192.168.18.101) at <from_interface> PERM PUB on eth4
? (192.168.18.101) at <from_interface> PERM PUB on eth0
</pre>

Another view is obtained from the following command on the HN.

<pre>cat /proc/net/arp</pre>

<pre>
IP address HW type Flags HW address Mask Device
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth0
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth0
</pre>

What this shows is that each VE's IP address is associated with each HN's interface. Therefore each interface will respond to any ARP "who has" query.

====The Cause====
These entries are created by the vzarp function in the vps_functions script, which are called by vps-net_add, vps-net_del and vps-stop. The result of this function in our case is to execute the following commands:

<pre>
/sbin/ip neigh add proxy 192.168.18.101 dev eth0
/sbin/ip neigh add proxy 192.168.18.101 dev eth4
/sbin/ip neigh add proxy 192.168.18.101 dev eth3
/sbin/ip neigh add proxy 192.168.18.102 dev eth0
/sbin/ip neigh add proxy 192.168.18.102 dev eth4
/sbin/ip neigh add proxy 192.168.18.102 dev eth3
</pre>

In addition, the following ARP messages are sent when VEID 101 is started.

<pre>
00:0c:29:b3:a2:54 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.10
00:0c:29:b3:a2:5e > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.12
00:0c:29:b3:a2:68 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.11
00:0c:29:b3:a2:54 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:5e > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:68 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:5e > 00:0c:29:b3:a2:68, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:5e > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:68 > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:68 > 00:0c:29:b3:a2:5e, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:54 > 00:0c:29:b3:a2:5e, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:54 > 00:0c:29:b3:a2:68, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
</pre>

What we see here is the result of vzarpipdetect, another function in vps_functions called by vps-net_add. An ARP "who has" message is sent by each interface and answered by the other interfaces.

What we want is to only add the IP addresses of our VE's to specific devices, not to all devices. This will prevent the ARP flux problem for our VE's.

====The Quick Fix====
Unfortunately this involves editing the OpenVZ scripts. The only case we really care about is vps-net_add, as the others execute <code>ip neigh del proxy</code>.

Manually editing the vzarp script is a quick fix, but not advised. Creating your own ''fork'' of OpenVZ is difficult to maintain and may have unintended side affects.

Fortunately there is a feature which will allow custom scripts to run during VE startup.

This approach is also described in http://wiki.openvz.org/Virtual_Ethernet_device.

In the VE configuration file, /etc/vz/conf/$VEID.conf, add the line

<pre>
CONFIG_CUSTOMIZED="yes"
</pre>

Now create the file /etc/vz/vznet.conf or /etc/vz/vznetcfg. Note that this will only work with a recent version of OpenVZ as the change was introduced in December, 2006. The file name seems to have changed between the two listed here so some trial and error will be required.

<pre>
#!/bin/bash
EXTERNAL_SCRIPT="/usr/lib/vzctl/scripts/vznet-custom"
</pre>

Finally create the file /usr/lib/vzctl/scripts/vznet-custom and add your custom commands.

TODO: Discuss contents of script and provide examples. Still need to work on getting this to work.

==Testing Environment==
All of the examples have been generated and tested using Debian Etch for the HN and Debian Stable for the VE's. VMware Workstation was used to create the test networks. The client is the BackTrack live CD from Remote Exploit. If you have different results from other releases of Linux please edit this page.

[[Category:HOWTO]]
[[Category:Networking]]
[[Category:Debian]]

Multiple network interfaces and ARP flux

2007-02-18T17:28:18Z

Estellt: Multiple Network Interfaces moved to Multiple Network Interfaces And ARP Flux

==Overview==
This page discusses working with multiple network interfaces on the Hardware Node (HN).

==The Simple Case==
In the simple case you have multiple network interfaces on the HN, all with IP addresses in the same subnet. Each of your Virtual Environments (VE's) also have IP addresses in the same subnet. You don't care which interfaces your VE's use.

So, no action is required. Everything just works. Setup OpenVZ normally.

The only downside is '''ARP flux'''. This describes the usually harmless condition where the network address (layer 3) drifts between multiple hardware addresses (layer 2). While this may cause some confusion to anyone trouble shooting, or generate alarms on network monitoring systems, it doesn't interrupt network traffic.

For an example of what this may look like, see the example and tcpdump captures below.

==A More Complex Case==
Let's say you have three network interfaces on the HN, all with IP addresses on the same subnet. Each of your VE's also have IP addresses on the same subnet. But now you ''do'' care which interface your VE's use.

For example, you want some of your VE's to always use eth3, and some to use eth4. But none of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.

===Example Network Setup===
To make this clear we'll use the following HN configuration. We'll also have another system to act as the client.

{| align="center" border="1" cellpadding=5
! System !! Interface !! MAC Address !! IP Address
|-
| HN || eth0 || 00:0c:29:b3:a2:54 || 192.168.18.10
|-
| HN || eth3 || 00:0c:29:b3:a2:68 || 192.168.18.11
|-
| HN || eth4 || 00:0c:29:b3:a2:5e || 192.168.18.12
|-
| client || eth0 || 00:0c:29:d2:c7:aa || 192.168.18.129
|}

===HN ARP Flux===
The first issue is fixing the '''ARP flux''' noted above. Any client on the network broadcasting an ARP "who has" message for any of the HN addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.

====Example One - HN ARP Flux====
For example, the following is a tcpdump capture from executing <code>ping -c2 192.168.18.10</code> from the client system.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:68
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The ARP "who has" message generated replies from all three MAC addresses on the HN. In this case the client took the MAC address for eth4. The three ICMP messages are then sent to eth4, but all the replies come from eth0. Normally this behavior isn't a problem, though it may generate some false alarms for a network monitor as it appears someone could be executing a man in the middle attack.

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

====A Simple Fix That May Work====
If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:

<pre>sysctl -w net.ipv4.conf.all.arp_filter=1</pre>

However, if they are all on the same IP network, which is the case here, then this won't achieve the desired results.

====A More Effective Solution====
The following can be added to your /etc/sysctl.conf file once you've tested it.

<pre>
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
</pre>

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

====Example Two - HN ARP Flux Corrected====
Now we repeat the ping command, after the arp cache on the client has been cleared.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The desired affect has been achieved. Only interface eth0 on the HN responds to the ARP message and the other interfaces are silent.

===Adding some VE's===
Now that the HN is behaving as expected, let's add some VE's and see what happens.

====VE Network Setup====
The case we're addressing is when the VE's are on the same subnet as the HN. So we create two new VE's and assign the addresses as follows.

{| align="center" border="1" cellpadding=5
! VEID || IP
|-
| 101 || 192.168.18.101
|-
| 102 || 192.168.18.102
|}

====Example Three - VE ARP Flux====
From the client system on you should be able to ping both VE's. However, looking at the ARP traffic with tcpdump you'll see that once again the network address associated with each VE will be subject to ARP flux, drifting between all three link layer addresses over time.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.101: ICMP echo request, id 43311, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.101 > 192.168.18.129: ICMP echo reply, id 43311, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.101: ICMP echo request, id 43311, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.101 > 192.168.18.129: ICMP echo reply, id 43311, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

====The ARP Cache====
The reasons for this can be found from executing the following command on the HN to display the ARP cache.

<pre>arp -an</pre>

<pre>
? (192.168.18.129) at 00:0C:29:D2:C7:AA [ether] on eth0
? (192.168.18.102) at <from_interface> PERM PUB on eth3
? (192.168.18.102) at <from_interface> PERM PUB on eth4
? (192.168.18.102) at <from_interface> PERM PUB on eth0
? (192.168.18.101) at <from_interface> PERM PUB on eth3
? (192.168.18.101) at <from_interface> PERM PUB on eth4
? (192.168.18.101) at <from_interface> PERM PUB on eth0
</pre>

Another view is obtained from the following command on the HN.

<pre>cat /proc/net/arp</pre>

<pre>
IP address HW type Flags HW address Mask Device
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth0
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth0
</pre>

What this shows is that each VE's IP address is associated with each HN's interface. Therefore each interface will respond to any ARP "who has" query.

====The Cause====
These entries are created by the vzarp function in the vps_functions script, which are called by vps-net_add, vps-net_del and vps-stop. The result of this function in our case is to execute the following commands:

<pre>
/sbin/ip neigh add proxy 192.168.18.101 dev eth0
/sbin/ip neigh add proxy 192.168.18.101 dev eth4
/sbin/ip neigh add proxy 192.168.18.101 dev eth3
/sbin/ip neigh add proxy 192.168.18.102 dev eth0
/sbin/ip neigh add proxy 192.168.18.102 dev eth4
/sbin/ip neigh add proxy 192.168.18.102 dev eth3
</pre>

In addition, the following ARP messages are sent when VEID 101 is started.

<pre>
00:0c:29:b3:a2:54 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.10
00:0c:29:b3:a2:5e > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.12
00:0c:29:b3:a2:68 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.11
00:0c:29:b3:a2:54 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:5e > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:68 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:5e > 00:0c:29:b3:a2:68, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:5e > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:68 > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:68 > 00:0c:29:b3:a2:5e, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:54 > 00:0c:29:b3:a2:5e, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:54 > 00:0c:29:b3:a2:68, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
</pre>

What we see here is the result of vzarpipdetect, another function in vps_functions called by vps-net_add. An ARP "who has" message is sent by each interface and answered by the other interfaces.

What we want is to only add the IP addresses of our VE's to specific devices, not to all devices. This will prevent the ARP flux problem for our VE's.

====The Quick Fix====
Unfortunately this involves editing the OpenVZ scripts. The only case we really care about is vps-net_add, as the others execute <code>ip neigh del proxy</code>.

TODO: Discuss changes to scripts.

==Testing Environment==
All of the examples have been generated and tested using Debian Etch for the HN and Debian Stable for the VE's. VMware Workstation was used to create the test networks. The client is the BackTrack live CD from Remote Exploit. If you have different results from other releases of Linux please edit this page.

[[Category:HOWTO]]
[[Category:Networking]]
[[Category:Debian]]

Multiple Network Interfaces

2007-02-18T17:28:18Z

Estellt: Multiple Network Interfaces moved to Multiple Network Interfaces And ARP Flux

#REDIRECT [[Multiple Network Interfaces And ARP Flux]]

Multiple network interfaces and ARP flux

2007-02-18T03:43:45Z

Estellt:

Multiple network interfaces and ARP flux

2007-02-18T03:32:38Z

Estellt:

Multiple network interfaces and ARP flux

2007-02-18T02:41:05Z

Estellt:

== Overview ==
This page discusses how to setup a HN with multiple network interfaces on the same physical network and on the same IP network. Then how to setup multiple VE's to use only one of these interfaces.

For example, you want some of your VE's to always use eth3, and some to use eth4. But none of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.

To make this clear we'll use the following HN configuration. We'll also have another system to act as the client.

{| align="center" border="1" cellpadding=5
! System !! Interface !! MAC Address !! IP Address
|-
| HN || eth0 || 00:0c:29:b3:a2:54 || 192.168.18.10
|-
| HN || eth3 || 00:0c:29:b3:a2:68 || 192.168.18.11
|-
| HN || eth4 || 00:0c:29:b3:a2:5e || 192.168.18.12
|-
| client || eth0 || 00:0c:29:d2:c7:aa || 192.168.18.129
|}

=== HN ARP Flux ===
The first issue is ARP flux. Any client on the network broadcasting an ARP "who has" message for any of these addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.

For example, the following is a tcpdump capture from executing <pre>ping -c2 192.168.18.10</pre> from another system on the network.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:68
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The ARP "who has" message generated replies from all three MAC addresses on the HN. In this case the client took the MAC address for eth4. The three ICMP messages are then sent to eth4, but all the replies com from eth0. Normally this behavior isn't a problem, though it may generate some false alarms for a network monitor as it appears someone could be executing a man in the middle attack.

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:

<code>sysctl -w net.ipv4.conf.all.arp_filter=1</code>

However, if they are all on the same IP network, which is the case here, then the following solution will work. This can be added to your /etc/sysctl.conf file once you've tested it.

<pre>
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
</pre>

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

Now we repeat the ping command, after the arp cache has been cleared.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The desired affect has been achieved. Only interface eth0 on the HN responds to the ARP message and the other interfaces are silent.

=== Adding some VE's ===

Now let's add some VE's to the HN as follows:

{| align="center" border="1" cellpadding=5
! VEID || IP
|-
| 101 || 192.168.18.101
|-
| 102 || 192.168.18.102
|}

From another system on the network you should be able to ping both. However, looking at the ARP traffic with tcpdump you'll see that once again the physical address associated with each VE will be subject to ARP flux, drifting between all three IP addresses over time.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.101: ICMP echo request, id 43311, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.101 > 192.168.18.129: ICMP echo reply, id 43311, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.101: ICMP echo request, id 43311, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.101 > 192.168.18.129: ICMP echo reply, id 43311, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The reasons for this can be found from executing the following command on the HN.

<pre>arp -an</pre>

<pre>
? (192.168.18.129) at 00:0C:29:D2:C7:AA [ether] on eth0
? (192.168.18.102) at <from_interface> PERM PUB on eth3
? (192.168.18.102) at <from_interface> PERM PUB on eth4
? (192.168.18.102) at <from_interface> PERM PUB on eth0
? (192.168.18.101) at <from_interface> PERM PUB on eth3
? (192.168.18.101) at <from_interface> PERM PUB on eth4
? (192.168.18.101) at <from_interface> PERM PUB on eth0
</pre>

Another view is obtained from the following command on the HN.

<pre>cat /proc/net/arp</pre>

<pre>
IP address HW type Flags HW address Mask Device
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth0
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth0
</pre>

What this shows is that each VE's IP address is associated with each HN's interface. Therefore each interface will respond to any ARP "who has" query.

These entries are created by the vzarp function in the vps_functions script, which are called by vps-net_add, vps-net_del and vps-stop. The result of this function in our case is to execute the following commands:

<pre>
/sbin/ip neigh add proxy 192.168.18.101 dev eth0
/sbin/ip neigh add proxy 192.168.18.101 dev eth4
/sbin/ip neigh add proxy 192.168.18.101 dev eth3
/sbin/ip neigh add proxy 192.168.18.102 dev eth0
/sbin/ip neigh add proxy 192.168.18.102 dev eth4
/sbin/ip neigh add proxy 192.168.18.102 dev eth3
</pre>

In addition, the following ARP messages are sent when VEID 101 is started.

<pre>
00:0c:29:b3:a2:54 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.10
00:0c:29:b3:a2:5e > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.12
00:0c:29:b3:a2:68 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.11
00:0c:29:b3:a2:54 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:5e > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:68 > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 (ff:ff:ff:ff:ff:ff) tell 192.168.18.101
00:0c:29:b3:a2:5e > 00:0c:29:b3:a2:68, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:5e > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:68 > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:68 > 00:0c:29:b3:a2:5e, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:54 > 00:0c:29:b3:a2:5e, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:54 > 00:0c:29:b3:a2:68, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
</pre>

What we see here is the result of vzarpipdetect, another function in vps_functions called by vps-net_add. An ARP "who has" message is sent by each interface and answered by the other interfaces.

What we want is to only add the IP addresses of our VE's to specific devices, not to all devices. This will prevent the ARP flux problem for our VE's.

Unfortunately this involves editing the OpenVZ scripts. The only case we really care about is vps-net_add, as the others execute <code>ip neigh del proxy</code>.

TODO: Discuss changes to scripts.

[[Category:HOWTO]]
[[Category:Networking]]

Multiple network interfaces and ARP flux

2007-02-17T20:10:06Z

Estellt:

== Overview ==
This page discusses how to setup a HN with multiple network interfaces on the same physical network and on the same IP network. Then how to setup multiple VE's to use only one of these interfaces.

For example, you want some of your VE's to always use eth3, and some to use eth4. But none of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.

To make this clear we'll use the following HN configuration. We'll also have another system to act as the client.

{| align="center" border="1" cellpadding=5
! System !! Interface !! MAC Address !! IP Address
|-
| HN || eth0 || 00:0c:29:b3:a2:54 || 192.168.18.10
|-
| HN || eth3 || 00:0c:29:b3:a2:68 || 192.168.18.11
|-
| HN || eth4 || 00:0c:29:b3:a2:5e || 192.168.18.12
|-
| client || eth0 || 00:0c:29:d2:c7:aa || 192.168.18.129
|}

=== HN ARP Flux ===
The first issue is ARP flux. Any client on the network broadcasting an ARP "who has" message for any of these addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.

For example, the following is a tcpdump capture from executing <pre>ping -c2 192.168.18.10</pre> from another system on the network.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:68
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The ARP "who has" message generated replies from all three MAC addresses on the HN. In this case the client took the MAC address for eth4. The three ICMP messages are then sent to eth4, but all the replies com from eth0. Normally this behavior isn't a problem, though it may generate some false alarms for a network monitor as it appears someone could be executing a man in the middle attack.

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:

<code>sysctl -w net.ipv4.conf.all.arp_filter=1</code>

However, if they are all on the same IP network, which is the case here, then the following solution will work. This can be added to your /etc/sysctl.conf file once you've tested it.

<pre>
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
</pre>

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

Now we repeat the ping command, after the arp cache has been cleared.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The desired affect has been achieved. Only interface eth0 on the HN responds to the ARP message and the other interfaces are silent.

=== Adding some VE's ===

Now let's add some VE's to the HN as follows:

{| align="center" border="1" cellpadding=5
! VEID || IP
|-
| 101 || 192.168.18.101
|-
| 102 || 192.168.18.102
|}

From another system on the network you should be able to ping both. However, looking at the ARP traffic with tcpdump you'll see that once again the physical address associated with each VE will be subject to ARP flux, drifting between all three IP addresses over time.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.101 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:68
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.101 is-at 00:0c:29:b3:a2:5e
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.101: ICMP echo request, id 43311, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.101 > 192.168.18.129: ICMP echo reply, id 43311, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.101: ICMP echo request, id 43311, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.101 > 192.168.18.129: ICMP echo reply, id 43311, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The reasons for this can be found from executing the following command on the HN.

<pre>arp -an</pre>

<pre>
? (192.168.18.129) at 00:0C:29:D2:C7:AA [ether] on eth0
? (192.168.18.102) at <from_interface> PERM PUB on eth3
? (192.168.18.102) at <from_interface> PERM PUB on eth4
? (192.168.18.102) at <from_interface> PERM PUB on eth0
? (192.168.18.101) at <from_interface> PERM PUB on eth3
? (192.168.18.101) at <from_interface> PERM PUB on eth4
? (192.168.18.101) at <from_interface> PERM PUB on eth0
</pre>

Another view is obtained from the following command on the HN.

<pre>cat /proc/net/arp</pre>

<pre>
IP address HW type Flags HW address Mask Device
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth0
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth0
</pre>

What this shows is that each VE's IP address is associated with each HN's interface. Therefore each interface will respond to any ARP "who has" query.

TODO: Discuss approach of <code>ip rule ...</code> and <code>ip route ...</code>.

[[Category:HOWTO]]
[[Category:Networking]]

Multiple network interfaces and ARP flux

2007-02-17T19:44:50Z

Estellt: Added examples of arpflux.

== Overview ==
This page discusses how to setup a HN with multiple network interfaces on the same physical network and on the same IP network. Then how to setup multiple VE's to use only one of these interfaces.

For example, you want some of your VE's to always use eth3, and some to use eth4. But none of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.

To make this clear we'll use the following HN configuration. We'll also have another system to act as the client.

{| align="center" border="1" cellpadding=5
! System !! Interface !! MAC Address !! IP Address
|-
| HN || eth0 || 00:0c:29:b3:a2:54 || 192.168.18.10
|-
| HN || eth3 || 00:0c:29:b3:a2:68 || 192.168.18.11
|-
| HN || eth4 || 00:0c:29:b3:a2:5e || 192.168.18.12
|-
| client || eth0 || 00:0c:29:d2:c7:aa || 192.168.18.129
|}

=== HN ARP Flux ===
The first issue is ARP flux. Any client on the network broadcasting an ARP "who has" message for any of these addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.

For example, the following is a tcpdump capture from executing <pre>ping -c2 192.168.18.10</pre> from another system on the network.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:68
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The ARP "who has" message generated replies from all three MAC addresses on the HN. In this case the client took the MAC address for eth4. The three ICMP messages are then sent to eth4, but all the replies com from eth0. Normally this behavior isn't a problem, though it may generate some false alarms for a network monitor as it appears someone could be executing a man in the middle attack.

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:

<code>sysctl -w net.ipv4.conf.all.arp_filter=1</code>

However, if they are all on the same IP network, which is the case here, then the following solution will work. This can be added to your /etc/sysctl.conf file once you've tested it.

<pre>
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
</pre>

The following output is from executing this command on the HN.

<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>

<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>

Now we repeat the ping command, after the arp cache has been cleared.

<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>

The desired affect has been achieved. Only interface eth0 on the HN responds to the ARP message and the other interfaces are silent.

=== Adding some VE's ===

Now let's add some VE's to the HN as follows:

{| align="center" border="1" cellpadding=5
! VEID || IP
|-
| 101 || 192.168.18.101
|-
| 102 || 192.168.18.102
|}

From another system on the network you should be able to ping both. However, looking at the ARP traffic with tcpdump you'll see that once again the physical address associated with each VE will be subject to ARP flux, drifting between all three IP addresses over time.

The reasons for this can be found from executing the following command on the HN.

<pre>arp -an</pre>

<pre>
? (192.168.18.129) at 00:0C:29:D2:C7:AA [ether] on eth0
? (192.168.18.102) at <from_interface> PERM PUB on eth3
? (192.168.18.102) at <from_interface> PERM PUB on eth4
? (192.168.18.102) at <from_interface> PERM PUB on eth0
? (192.168.18.101) at <from_interface> PERM PUB on eth3
? (192.168.18.101) at <from_interface> PERM PUB on eth4
? (192.168.18.101) at <from_interface> PERM PUB on eth0
</pre>

Another view is obtained from the following command on the HN.

<pre>cat /proc/net/arp</pre>

<pre>
IP address HW type Flags HW address Mask Device
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth0
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth3
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth4
192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth0
</pre>

What this shows is that each VE's IP address is associated with each HN's interface. Therefore each interface will respond to any ARP "who has" query.

TODO: Discuss approach of <code>ip rule ...</code> and <code>ip route ...</code>.

[[Category:HOWTO]]
[[Category:Networking]]

Multiple network interfaces and ARP flux

2007-02-16T22:19:17Z

Estellt:

Multiple network interfaces and ARP flux

2007-02-16T21:55:11Z

Estellt: New page: This page discusses how to setup a HN with multiple network interfaces on the same physical network and on the same IP network. For example: <pre> eth0 = 00:0c:29:b3:a2:54 with ip address...

This page discusses how to setup a HN with multiple network interfaces on the same physical network and on the same IP network.

For example:
<pre>
eth0 = 00:0c:29:b3:a2:54 with ip address 192.168.18.10
eth3 = 00:0c:29:b3:a2:68 with ip address 192.168.18.11
eth4 = 00:0c:29:b3:a2:fe with ip address 192.168.18.12
</pre>
The first issue is ARP flux. Any client on the network who broadcasts an ARP "who has" message for any of these addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.

If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:

<code>sysctl -w net.ipv4.conf.all.arp_filter=1</code>

However, if they are all on the same IP network, which is the case here, then the following solution will work. This can be added to your /etc/sysctl.conf file once you've tested it.

<pre>
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
</pre>

Using another system on the network you should be able to execute the following:
<pre>
arping -c3 192.168.18.10
arping -c3 192.168.18.11
apring -c3 192.168.18.12
</pre>
Each of these should only generate responses from the network interface associated with that IP address on the HN.
----
Now, let's say you want some of your VE's to always use eth3, and some to use eth4. None of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.

Setup two VE's as follows:
<pre>
VEID = 101 IP = 192.168.18.101
VEID = 102 IP = 192.168.18.102
</pre>
From another system on the network you should be able to ping both. However, looking at the ARP traffic with tcpdump or using arping you'll see that once again the physical address associated with each VE will be subject to ARP flux, drifting between all three IP addresses over time.

TODO: Discuss approach of <code>ip rule ...</code> and <code>ip route ...</code>.