Multiple network interfaces and ARP flux

Overview

This page discusses how to setup a HN with multiple network interfaces on the same physical network and on the same IP network. Then how to setup multiple VE's to use only one of these interfaces.

For example, you want some of your VE's to always use eth3, and some to use eth4. But none of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.

To make this clear we'll use the following HN configuration.

Interface	MAC Address	IP Address
eth0	00:0c:29:b3:a2:54	192.168.18.10
eth3	00:0c:29:b3:a2:68	192.168.18.11
eth4	00:0c:29:b3:a2:fe	192.168.18.12

HN ARP Flux

The first issue is ARP flux. Any client on the network broadcasting an ARP "who has" message for any of these addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.

If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:

sysctl -w net.ipv4.conf.all.arp_filter=1

However, if they are all on the same IP network, which is the case here, then the following solution will work. This can be added to your /etc/sysctl.conf file once you've tested it.

sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2

Using another system on the network you should be able to execute the following:

arping -c3 192.168.18.10
arping -c3 192.168.18.11
apring -c3 192.168.18.12

Each of these should only generate responses from the network interface associated with that IP address on the HN.

Adding some VE's

Setup two VE's as follows:

VEID = 101  IP = 192.168.18.101
VEID = 102  IP = 192.168.18.102

From another system on the network you should be able to ping both. However, looking at the ARP traffic with tcpdump or using arping you'll see that once again the physical address associated with each VE will be subject to ARP flux, drifting between all three IP addresses over time.

TODO: Discuss approach of ip rule ... and ip route ....

TODO: Add examples of ARP flux before fix is applied, and after.