9
edits
Changes
no edit summary
== Overview ==This page discusses how to setup a HN working with multiple network interfaces on the same physical Hardware Node (HN). ==The Simple Case==In the simple case you have multiple network and interfaces on the HN, all with IP addresses in the same subnet. Each of your Virtual Environments (VE's) also have IP networkaddresses in the same subnet. Then how to setup multiple You don't care which interfaces your VE's to use . So, no action is required. Everything just works. Setup OpenVZ normally. The only one downside is '''ARP flux'''. This describes the usually harmless condition where the network address (layer 3) drifts between multiple hardware addresses (layer 2). While this may cause some confusion to anyone trouble shooting, or generate alarms on network monitoring systems, it doesn't interrupt network traffic. For an example of these what this may look like, see the example and tcpdump captures below. ==A More Complex Case==Let's say you have three network interfaceson the HN, all with IP addresses on the same subnet. Each of your VE's also have IP addresses on the same subnet. But now you ''do'' care which interface your VE's use.
For example, you want some of your VE's to always use eth3, and some to use eth4. But none of the VE traffic should use eth0, which is reserved for use by the HN only. This makes sense if you have VE's that may generate or receive a lot of traffic and you don't want your remote administration of the server over eth0 to degrade or get blocked because of this.
===Example Network Setup===
To make this clear we'll use the following HN configuration. We'll also have another system to act as the client.
|}
=== HN ARP Flux ===The first issue is fixing the '''ARP flux''' noted above. Any client on the network broadcasting an ARP "who has" message for any of these the HN addresses will receive replies from all three interfaces. This results in IP addresses that float between three MAC addresses, depending on which response a client accepts first.
====Example One - HN ARP Flux====For example, the following is a tcpdump capture from executing <precode>ping -c2 192.168.18.10</precode> from another the client system on the network.
<pre>
</pre>
The ARP "who has" message generated replies from all three MAC addresses on the HN. In this case the client took the MAC address for eth4. The three ICMP messages are then sent to eth4, but all the replies com come from eth0. Normally this behavior isn't a problem, though it may generate some false alarms for a network monitor as it appears someone could be executing a man in the middle attack.
The following output is from executing this command on the HN.
</pre>
====A Simple Fix That May Work====
If all three network interfaces are on different IP networks (such as 10.x.x.x, 172.16.x.x, 192.168.x.x) then executing the following will work:
<codepre>sysctl -w net.ipv4.conf.all.arp_filter=1</codepre>
However, if they are all on the same IP network, which is the case here, then this won't achieve the desired results. ====A More Effective Solution====The following solution will work. This can be added to your /etc/sysctl.conf file once you've tested it.
<pre>
</pre>
====Example Two - HN ARP Flux Corrected====Now we repeat the ping command, after the arp cache on the client has been cleared.
<pre>
The desired affect has been achieved. Only interface eth0 on the HN responds to the ARP message and the other interfaces are silent.
=== Adding some VE's === Now that the HN is behaving as expected, let's add some VE's to the HN as follows:and see what happens.
====VE Network Setup====
The case we're addressing is when the VE's are on the same subnet as the HN. So we create two new VE's and assign the addresses as follows.
{| align="center" border="1" cellpadding=5
|}
====Example Three - VE ARP Flux====From another the client system on the network you should be able to ping bothVE's. However, looking at the ARP traffic with tcpdump you'll see that once again the physical network address associated with each VE will be subject to ARP flux, drifting between all three IP link layer addresses over time.
<pre>
</pre>
====The ARP Cache====The reasons for this can be found from executing the following command on the HNto display the ARP cache.
<pre>arp -an</pre>
What this shows is that each VE's IP address is associated with each HN's interface. Therefore each interface will respond to any ARP "who has" query.
====The Cause====
These entries are created by the vzarp function in the vps_functions script, which are called by vps-net_add, vps-net_del and vps-stop. The result of this function in our case is to execute the following commands:
What we want is to only add the IP addresses of our VE's to specific devices, not to all devices. This will prevent the ARP flux problem for our VE's.
====The Quick Fix====
Unfortunately this involves editing the OpenVZ scripts. The only case we really care about is vps-net_add, as the others execute <code>ip neigh del proxy</code>.
