Multiple network interfaces and ARP flux

This page discusses how to setup a HN with multiple network interfaces on the same physical network and on the same IP network.

For example:

eth0 = 00:0c:29:b3:a2:54 with ip address 192.168.18.10
eth3 = 00:0c:29:b3:a2:68 with ip address 192.168.18.11
eth4 = 00:0c:29:b3:a2:fe with ip address 192.168.18.12

The first issue is ARP flux. Any client on the network who broadcasts an ARP "who has" message for any of these addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.

If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:

sysctl -w net.ipv4.conf.all.arp_filter=1

However, if they are all on the same IP network, which is the case here, then the following solution will work. This can be added to your /etc/sysctl.conf file once you've tested it.

sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2

Using another system on the network you should be able to execute the following:

arping -c3 192.168.18.10
arping -c3 192.168.18.11
apring -c3 192.168.18.12

Each of these should only generate responses from the network interface associated with that IP address on the HN.

---

Now, let's say you want some of your VE's to always use eth3, and some to use eth4. None of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.

Setup two VE's as follows:

VEID = 101  IP = 192.168.18.101
VEID = 102  IP = 192.168.18.102

From another system on the network you should be able to ping both. However, looking at the ARP traffic with tcpdump or using arping you'll see that once again the physical address associated with each VE will be subject to ARP flux, drifting between all three IP addresses over time.

TODO: Discuss approach of ip rule ... and ip route ....