Traffic accounting with iptables
Suppose you need to know how much traffic your containers eat. It can be easily done using iptables.
Situation description
Let's consider the very simple situation: one container with one IP address on the Hardware Node with only one network interface. To be more exact, assume that container ID is 200, the IP address of the HN is 192.168.0.56, the network interface name is eth0, and the IP address of the container is 192.168.0.117.
You wish to know how many bytes container 200 eats. One more assumption is that there are no iptables rules on HN now. All these assumption are only for clarity!
Solution
Almost any traffic that goes to and from a container can be catched by FORWARD chain of iptables module in container0, thus we add such rules:
# iptables -A FORWARD -s 192.168.0.117 # iptables -A FORWARD -d 192.168.0.117
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted. To obtain current traffic usage of container you can issue the command:
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
Bytes column is the column we need. It's worth saying, that restarting a container doesn't affect accounting, it remains right. But if you restart your hardware node, all the rules and consequently statistics are dropped. So it is recommended to
- run some cron job that dumps statistics to some file
- add init script that creates iptables rules on HN start.
If you want to process the results with a script it is useful to use the "-x" or "--exact" option of iptables
# iptables -nvx -L FORWARD
You will get the exact value of the packet and byte counters, instead of only the rounded number in K’s (multiples of 1000) M’s (multiples of 1000K) or G’s (multiples of 1000M).
As is easy to see, it's not per-container statistic, but rather per-IP statistic. Thus you must be careful then changing container IP addresses, otherwise you'll get mess of results.
By saying almost any traffic I mean that traffic between a container and container0 is not accounted by rules above. Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
iptables -I INPUT 1 -i venet0 -d 192.168.0.117 iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
To observe results:
# iptables -nvx -L INPUT Chain INPUT (policy ACCEPT 542 packets, 63745 bytes) pkts bytes target prot opt in out source destination 35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117 # iptables -nvx -L OUTPUT Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes) pkts bytes target prot opt in out source destination 48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
If you need to zero counters this command works:
# iptables -Z
The disadvantage is that by doingit this way you zero all counters in all rules. If it is not what you need, you can just replace the rule with the same rule:
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
More complicated cases
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in more complicated situations.
- More than one container on the node
- Just add the rules like above for each container IP.
- More than one IP per container.
- For each IP add the rules like above. When counting the complete traffic of a container you have to summarize over all IPs that this container owns.
- More interfaces on the HN.
- Nothing to do! :)
Scripting
Here are some scripting ideas
Get CTIDs of all running containers
host2:~/bin# cat vz-all-running vzlist -H -oveid | sed 's/ //g;'
Get all IPs of running containers
host2:~/bin# cat vz-all-running-ip vzlist -H -o ip
Set up all needed iptables rules
host2:~/bin# cat vz-iptables-create-rules for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1 for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
Generate a traffic.log
Please use crontab to run this script once per hour or day to collect your traffic statistics.
(Warning, the counters can overflow if there is too much traffic within that period. Would recommend 15 minute intervals if you expect a lot of traffic)
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nvx -L FORWARD | grep " $i " | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the iptables rules if there is a any change in containers
./vz-iptables-create-rules
# copy the trafficlog file to a webserver where users can see their traffic
# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node (~/.ssh/id_rsa.pub)
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# in order for the below scp command to not ask for root password
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic
# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null
Sample php script to store the trafficlog in a database
Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";
mysql_connect($MySQL_Host,$MySQL_User,$MySQL_Passw);
$HN=trim(addslashes($_GET["HN"])); // Hardware Node
$handle = fopen ("tmp/{$HN}-traffic","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('{$ip}','{$date} {$time}','{$traffic}')");}
}
fclose($handle);
?>A SQL query to get the traffic for the last 30 days
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ipNotes
As you see this way can be time-consuming in case of a big number of containers.
So if anybody has scripts that automate all the process — you are welcome!
