Traffic accounting with iptables

Suppose you need to know how much traffic your VEs eat. It can be easily done using iptables.

Situation description

Let's consider the very simple situation: one VE with one IP address on the Hardware Node with only one network interface. To be more exact, assume that VE ID is 200, the IP address of the HN is 192.168.0.56, the network interface name is eth0, and the IP address of the VE is 192.168.0.117.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules on HN now. All these assumption are only for clarity!

Solution

Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in VE0, thus we add such rules:

# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117

It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted. To obtain current traffic usage of VE you can issue the command:

# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
 pkts bytes target     prot opt in     out     source               destination
    8   832            all  --  *      *       192.168.0.117        0.0.0.0/0
   15  1052            all  --  *      *       0.0.0.0/0            192.168.0.117

Bytes column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting, it remains right. But if you restart your hardware node, all the rules and consequently statistics are dropped. So it is recommended to

• run some cron job that dumps statistics to some file
• add init script that creates iptables rules on HN start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful then changing VE IP addresses, otherwise you'll get mess of results.

By saying almost any traffic I mean that traffic between a VE and VE0 is not accounted by rules above. Not sure if it can be useful for anybody, but to account such traffic these rules are needed:

iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117

To observe results:

# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
 pkts bytes target     prot opt in     out     source               destination
   35  4533            all  --  venet0 *       0.0.0.0/0            192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
 pkts bytes target     prot opt in     out     source               destination
   48  4724            all  --  *      venet0  192.168.0.117        0.0.0.0/0

If you need to zero counters this command works:

# iptables -Z

The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need, you can just replace the rule with the same rule:

# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
 pkts bytes target     prot opt in     out     source               destination
   44  5151            all  --  *      *       192.168.0.117        0.0.0.0/0
   57  5564            all  --  *      *       0.0.0.0/0            192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0            all  --  *      *       192.168.0.117        0.0.0.0/0
   57  5564            all  --  *      *       0.0.0.0/0            192.168.0.117

More complicated cases

Well, now, when we know how to work in the easiest case, we'll try to understand what to do in more complicated situations.

More than one VE on the node

Just add the rules like above for each VE IP.

More than one IP per VE.

For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

More interfaces on the HN.

Nothing to do! :)

Scripting

Here are some scripting ideas

first a small script to get all vz id's for later on

host2:~/bin# cat vz-all-running
vzlist -H -oveid

second a small script witch get all ip's of running vz's

host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip

and a small script to set up all needed iptable rules

host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`;  do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`;  do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1

a small script to generate a traffic.log

host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
 do
  echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
  echo -n " $i " >> $trafficlog
  echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
 done
 iptables -Z

You can use crontab to run this script once per hour or day to collect your traffic statistics.

As you see this way can be time-consuming in case of big number of VEs. So if anybody has scripts that automate all the process — you are welcome!