Changes

Jump to: navigation, search

Multiple network interfaces and ARP flux

6,665 bytes added, 19:44, 17 February 2007
Added examples of arpflux.
For example, you want some of your VE's to always use eth3, and some to use eth4. But none of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.
To make this clear we'll use the following HN configuration. We'll also have another system to act as the client.
{| align="center" border="1" cellpadding=5
! System !! Interface !! MAC Address !! IP Address
|-
| HN || eth0 || 00:0c:29:b3:a2:54 || 192.168.18.10
|-
| HN || eth3 || 00:0c:29:b3:a2:68 || 192.168.18.11
|-
| HN || eth4 || 00:0c:29:b3:a2:fe 5e || 192.168.18.12|-| client || eth0 || 00:0c:29:d2:c7:aa || 192.168.18.129
|}
=== HN ARP Flux ===
The first issue is ARP flux. Any client on the network broadcasting an ARP "who has" message for any of these addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.
 
For example, the following is a tcpdump capture from executing <pre>ping -c2 192.168.18.10</pre> from another system on the network.
 
<pre>
00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-has 192.168.18.10 tell 192.168.18.129
00:0c:29:b3:a2:5e > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:5e
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:54
00:0c:29:b3:a2:68 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-at 00:0c:29:b3:a2:68
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 1, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 1, length 64
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:5e, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32313, seq 2, length 64
00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-has 192.168.18.129 tell 192.168.18.10
00:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>
 
The ARP "who has" message generated replies from all three MAC addresses on the HN. In this case the client took the MAC address for eth4. The three ICMP messages are then sent to eth4, but all the replies com from eth0. Normally this behavior isn't a problem, though it may generate some false alarms for a network monitor as it appears someone could be executing a man in the middle attack.
 
The following output is from executing this command on the HN.
 
<pre>sysctl -a | grep net.ipv4.conf.*.arp</pre>
 
<pre>
net.ipv4.conf.venet0.arp_accept = 0
net.ipv4.conf.venet0.arp_ignore = 0
net.ipv4.conf.venet0.arp_announce = 0
net.ipv4.conf.venet0.arp_filter = 0
net.ipv4.conf.venet0.proxy_arp = 0
net.ipv4.conf.eth4.arp_accept = 0
net.ipv4.conf.eth4.arp_ignore = 0
net.ipv4.conf.eth4.arp_announce = 0
net.ipv4.conf.eth4.arp_filter = 0
net.ipv4.conf.eth4.proxy_arp = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.proxy_arp = 0
</pre>
If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:
</pre>
Using another system The following output is from executing this command on the network you should be able to execute HN. <pre>sysctl -a | grep net.ipv4.conf.*.arp</pre> <pre>net.ipv4.conf.venet0.arp_accept = 0net.ipv4.conf.venet0.arp_ignore = 0net.ipv4.conf.venet0.arp_announce = 0net.ipv4.conf.venet0.arp_filter = 0net.ipv4.conf.venet0.proxy_arp = 0net.ipv4.conf.eth4.arp_accept = 0net.ipv4.conf.eth4.arp_ignore = 0net.ipv4.conf.eth4.arp_announce = 0net.ipv4.conf.eth4.arp_filter = 0net.ipv4.conf.eth4.proxy_arp = 0net.ipv4.conf.eth3.arp_accept = 0net.ipv4.conf.eth3.arp_ignore = 0net.ipv4.conf.eth3.arp_announce = 0net.ipv4.conf.eth3.arp_filter = 0net.ipv4.conf.eth3.proxy_arp = 0net.ipv4.conf.eth0.arp_accept = 0net.ipv4.conf.eth0.arp_ignore = 0net.ipv4.conf.eth0.arp_announce = 0net.ipv4.conf.eth0.arp_filter = 0net.ipv4.conf.eth0.proxy_arp = 0net.ipv4.conf.lo.arp_accept = 0net.ipv4.conf.lo.arp_ignore = 0net.ipv4.conf.lo.arp_announce = 0net.ipv4.conf.lo.arp_filter = 0net.ipv4.conf.lo.proxy_arp = 0net.ipv4.conf.default.arp_accept = 0net.ipv4.conf.default.arp_ignore = 0net.ipv4.conf.default.arp_announce = 0net.ipv4.conf.default.arp_filter = 0net.ipv4.conf.default.proxy_arp = 0net.ipv4.conf.all.arp_accept = 0net.ipv4.conf.all.arp_ignore = 1net.ipv4.conf.all.arp_announce = 1net.ipv4.conf.all.arp_filter = 0net.ipv4.conf.all.proxy_arp = 0</pre> Now we repeat the following:ping command, after the arp cache has been cleared. 
<pre>
arping 00:0c:29:d2:c7:aa > ff:ff:ff:ff:ff:ff, ARP, length 60: arp who-c3 has 192.168.18.10tell 192.168.18.129arping 00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp reply 192.168.18.10 is-c3 at 00:0c:29:b3:a2:5400:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.10: ICMP echo request, id 32066, seq 1, length 6400:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 1, length 6400:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, IPv4, length 98: 192.168.18.129 > 192.168.18.1110: ICMP echo request, id 32066, seq 2, length 64apring 00:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, IPv4, length 98: 192.168.18.10 > 192.168.18.129: ICMP echo reply, id 32066, seq 2, length 6400:0c:29:b3:a2:54 > 00:0c:29:d2:c7:aa, ARP, length 60: arp who-c3 has 192.168.18.129 tell 192.168.18.121000:0c:29:d2:c7:aa > 00:0c:29:b3:a2:54, ARP, length 60: arp reply 192.168.18.129 is-at 00:0c:29:d2:c7:aa
</pre>
Each of these should only generate responses from the network The desired affect has been achieved. Only interface associated with that IP address eth0 on the HNresponds to the ARP message and the other interfaces are silent.
=== Adding some VE's ===
Setup two Now let's add some VE's to the HN as follows:  {| align="center" border="1" cellpadding=5! VEID || IP|-| 101 || 192.168.18.101|-| 102 || 192.168.18.102|} From another system on the network you should be able to ping both. However, looking at the ARP traffic with tcpdump you'll see that once again the physical address associated with each VE will be subject to ARP flux, drifting between all three IP addresses over time. The reasons for this can be found from executing the following command on the HN. <pre>arp -an</pre> <pre>? (192.168.18.129) at 00:0C:29:D2:C7:AA [ether] on eth0? (192.168.18.102) at <from_interface> PERM PUB on eth3? (192.168.18.102) at <from_interface> PERM PUB on eth4? (192.168.18.102) at <from_interface> PERM PUB on eth0? (192.168.18.101) at <from_interface> PERM PUB on eth3? (192.168.18.101) at <from_interface> PERM PUB on eth4? (192.168.18.101) at <from_interface> PERM PUB on eth0</pre> Another view is obtained from the following command on the HN. <pre>cat /proc/net/arp</pre> 
<pre>
VEID = 101 IP = address HW type Flags HW address Mask Device192.168.18.101102 0x1 0xc 00:00:00:00:00:00 * eth3VEID = 192.168.18.102 IP = 0x1 0xc 00:00:00:00:00:00 * eth4192.168.18.102 0x1 0xc 00:00:00:00:00:00 * eth0192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth3192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth4192.168.18.101 0x1 0xc 00:00:00:00:00:00 * eth0
</pre>
From another system on the network you should be able to ping both. However, looking at the ARP traffic with tcpdump or using arping youWhat this shows is that each VE'll see that once again the physical s IP address is associated with each VE HN's interface. Therefore each interface will be subject respond to any ARP flux, drifting between all three IP addresses over time"who has" query.
TODO: Discuss approach of <code>ip rule ...</code> and <code>ip route ...</code>.
TODO: Add examples of ARP flux before fix is applied, and after.
[[Category:HOWTO]]
[[Category:Networking]]
9
edits

Navigation menu