Source based routing
Sometimes you have more than one router in your network, and want different containers to use different routers. Other times you may have a single HN with IP addresses on different networks and want to assign containers addresses from those networks.
Lets say you have a HN with an IP address in network 192.168.100.0/24 (192.168.100.10) and an IP address in 192.168.200.0 (192.168.200.10). Maybe those addresses are on different VLANs. Maybe one is an internal network and the other faces the wider internet. Maybe you have 10 different networks assigned to the HN. It does not matter as long as there is a gateway on each of those networks. In our example we will assume the gateways are 192.168.100.1 and 192.168.200.1. You want any container assigned an address in the 192.168.100.0/24 network to use 192.168.100.1 and any container assigned an address in the 192.168.200.0/24 network to use 192.168.200.1.
By default the network traffic coming from a container will use the default gateway on the HN to reach the rest of the world. If we want our containers to use the gateways on their respective networks we need to configure source based routing. This involves creating an additional routing table to redirect the traffic.
For example:
# /sbin/ip rule add from 192.168.100.0/24 table 10000 # /sbin/ip route add throw 192.168.100.0/24 table 10000 # /sbin/ip route add default via 192.168.100.1 table 10000
The first line adds a routing rule. This rule tells the system to use an alternate routing table when trying to route packets from a certain source. In this case we are telling the system that if a packet originates from a 192.168.100.0/24 address we should use routing table 10000. The table number is unique and simply must be an unused table number from your system. I tend to start at 10000, but you can start your number wherever is convenient. To see a list of tables in use you can use:
# /sbin/ip rule list
Next we add two routing rules to table 10000. The first one is a throw rule. A throw rule merely tells the system to stop processing the current table if the destination address matches the criteria provided. This will allow the host system and the VPSs to continue to reach other systems on our 192.168.100.0/24 network without trying to use the default gateway we provide. And the second rule provides that default gateway.
Now all we need to do is repeat this for our second network:
# /sbin/ip rule add from 192.168.200.0/24 table 10001 # /sbin/ip route add throw 192.168.200.0/24 table 10001 # /sbin/ip route add default via 192.168.200.1 table 10001
Here we have changed the networks in the rule and routes and used a different table number. Everything else stays the same. You can, of course, as as many complex routes to a particular table as you like. If you want to allow a container in the 192.168.100.0/24 network to reach the 192.168.200.0/24 network without using the gateway, you can add another throw rule and allow the HN's default routing table to take effect:
# /sbin/ip route add throw 192.168.200.0/24 table 10000
A previous version of this page suggested adding an additional route in order to allow the HN to contact the container. Indeed this would be required if we did not provide the throw rule, but maintaining such a configuration requires adding new rules for every container. Using vzctl set <ctid> --ipadd <ip> adds these rules to the main routing table by default, but not our custom routing table. The configuration here only requires rules to be modified when changes are made to the networks, not each container.
For more details on routing rules, see man ip.
