Changes

Jump to: navigation, search

Traffic accounting with iptables

3,072 bytes added, 09:28, 15 December 2011
m
Reverted edits by TermPaperServices (talk) to last revision by Kir
Suppose you need to know how much traffic your [[VEcontainer]]s eat. It can be easily done
using iptables.
== Situation description ==
Let's consider the very simple situation: one VE container with one IP address on the [[Hardware Node]]with only one network interface. To be more exact, assume that [[VEcontainer]] ID is <tt>200</tt>, the IP address of the [[HN]]is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VEcontainer]] is <tt>192.168.0.117</tt>.
You wish to know how much many bytes VE container 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!
== Solution ==
Almost any traffic that goes to and from a VE container can be catched by FORWARD chain of iptables module in [[VE0container0]],
thus we add such rules:
<pre>
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE container you can issue the command:
<pre>
# iptables -nv -L FORWARD
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to saysaying, that restarting a VE container doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* add init script that creates iptables rules on [[HN]] start.
As If you want to process the results with a script it is easy useful to see, it's not peruse the "-x" or "--exact" option of iptables<pre># iptables -VE statistic, but rather pernvx -IP statistic. Thus you must be carefulL FORWARD</pre>then changing VE IP addressesYou will get the exact value of the packet and byte counters, otherwise you'll get mess instead of only the rounded number in K’s (multiples of 1000) M’s (multiples of 1000K) or G’s (multiples of results 1000M).
As is easy to see, it's not per-container statistic, but rather per-IP statistic. Thus you must be carefulthen changing container IP addresses, otherwise you'll get mess of results. By saying ''almost any traffic'' I mean that traffic between a [[VEcontainer]] and [[VE0container0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
To observe results:
<pre>
# iptables -nv nvx -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv nvx -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
# iptables -Z
</pre>
The disadvantage is that doing by doingit this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
more complicated situations.
; More than one VE container on the node: Just add the rules like above for each VE container IP.
; More than one IP per VEcontainer.: For each IP add the rules like above. When counting the complete traffic of a VE container you have to summarize over all IPs that this VE container owns.
; More interfaces on the HN.
== Scripting ==
As you see this way can be time-consuming in case of big number of VEs.
So if anybody has scripts that automate all the process - you are welcome!
 
Here are some scripting ideas
first a small script to get === Get CTIDs of all vz id's for later onrunning containers ===
<pre>
host2:~/bin# cat vz-all-running
vzlist |grep run|tr -s [:blank:]H -oveid |cut -dsed ' s/ //g;' -f2
</pre>
second a small script witch get === Get all ip's IPs of running vz'scontainers ===
<pre>
host2:~/bin# cat vz-all-running-ip
for i in `./vzvzlist -allH -running`; do vzctl exec $i ifconfig |grep 'venet0:0' -A1|tail -n1|cut -d':' -f2|cut -d' ' -f1; doneo ip
</pre>
and a small script to set === Set up all iptable needed iptables rules===
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>/dev/null&1for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1</pre> === Generate a traffic.log ===Please use crontab to run this script once per hour or day to collect your traffic statistics. (Warning, the counters can overflow if there is too much traffic within that period. Would recommend 15 minute intervals if you expect a lot of traffic)<pre>host2:~/bin# cat vz-generate-traffic-logtrafficlog="/var/log/vz-traffic.log"for i in `./vz-all-running-ip` ; do echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog echo -n " $i " >> $trafficlog echo `iptables -nvx -L FORWARD | grep " $i " | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog done # reset the counter iptables -Z # update the iptables rules if there is a any change in containers ./vz-iptables-create-rules  # copy the trafficlog file to a webserver where users can see their traffic  # please mind to use # ssh-keygen -t rsa # to generate ssh keys # and append the new public key from your hardware node (~/.ssh/id_rsa.pub) # to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS # in order for the below scp command to not ask for root password scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic # clear the copied trafficlog cp /dev/null $trafficlog # start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS # please mind to use .htaccess to secure this wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>
=== Sample php script to store the trafficlog in a database ===
 
Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<source lang="text">
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
</source>
<source lang="php">
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";
mysql_connect($MySQL_Host,$MySQL_User,$MySQL_Passw);
 
$HN=trim(addslashes($_GET["HN"])); // Hardware Node
 
$handle = fopen ("tmp/{$HN}-traffic","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('{$ip}','{$date} {$time}','{$traffic}')");}
}
fclose($handle);
?>
</source>
 
=== A SQL query to get the traffic for the last 30 days ===
<source lang="mysql">
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</source>
 
=== Notes ===
 
As you see this way can be time-consuming in case of a big number of containers.
So if anybody has scripts that automate all the process — you are welcome!
== See also ==
* [[Traffic accounting through proc]]
[[Category: HOWTO]]
[[Category: Networking]]
[[Category: Monitoring]]

Navigation menu