Difference between revisions of "Traffic accounting with iptables"

From OpenVZ Virtuozzo Containers Wiki
Jump to: navigation, search
(Scripting)
m (Reverted edits by TermPaperServices (talk) to last revision by Kir)
 
(28 intermediate revisions by 8 users not shown)
Line 1: Line 1:
Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
+
Suppose you need to know how much traffic your [[container]]s eat. It can be easily done
 
using iptables.
 
using iptables.
  
 
== Situation description ==
 
== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
+
Let's consider the very simple situation: one container with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
+
with only one network interface. To be more exact, assume that [[container]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.
+
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[container]] is <tt>192.168.0.117</tt>.
  
You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
+
You wish to know how many bytes container 200 eats. One more assumption is that there are no iptables rules
 
on HN now. All these assumption are only for clarity!
 
on HN now. All these assumption are only for clarity!
  
 
== Solution ==
 
== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
+
Almost any traffic that goes to and from a container can be catched by FORWARD chain of iptables module in [[container0]],
 
thus we add such rules:
 
thus we add such rules:
 
<pre>
 
<pre>
Line 18: Line 18:
 
</pre>
 
</pre>
 
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
 
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
+
To obtain current traffic usage of container you can issue the command:
 
<pre>
 
<pre>
 
# iptables -nv -L FORWARD
 
# iptables -nv -L FORWARD
Line 26: Line 26:
 
   15  1052            all  --  *      *      0.0.0.0/0            192.168.0.117
 
   15  1052            all  --  *      *      0.0.0.0/0            192.168.0.117
 
</pre>
 
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
+
'''Bytes''' column is the column we need. It's worth saying, that restarting a container doesn't affect accounting,
 
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
 
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
 
So it is recommended to
 
So it is recommended to
Line 32: Line 32:
 
* add init script that creates iptables rules on [[HN]] start.
 
* add init script that creates iptables rules on [[HN]] start.
  
As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
+
If you want to process the results with a script it is useful to use the "-x" or "--exact" option of iptables
then changing VE IP addresses, otherwise you'll get mess of results.
+
<pre>
 +
# iptables -nvx -L FORWARD
 +
</pre>
 +
You will get the exact value of the packet and byte counters, instead of only the rounded number in K’s (multiples of 1000) M’s (multiples of 1000K) or G’s (multiples  of  1000M).
 +
 
 +
As is easy to see, it's not per-container statistic, but rather per-IP statistic. Thus you must be careful
 +
then changing container IP addresses, otherwise you'll get mess of results.
  
By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
+
By saying ''almost any traffic'' I mean that traffic between a [[container]] and [[container0]] is not accounted by rules above.
 
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
 
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
 
<pre>
 
<pre>
Line 44: Line 50:
 
To observe results:
 
To observe results:
 
<pre>
 
<pre>
# iptables -nv -L INPUT
+
# iptables -nvx -L INPUT
 
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
 
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
 
  pkts bytes target    prot opt in    out    source              destination
 
  pkts bytes target    prot opt in    out    source              destination
 
   35  4533            all  --  venet0 *      0.0.0.0/0            192.168.0.117
 
   35  4533            all  --  venet0 *      0.0.0.0/0            192.168.0.117
# iptables -nv -L OUTPUT
+
# iptables -nvx -L OUTPUT
 
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
 
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
 
  pkts bytes target    prot opt in    out    source              destination
 
  pkts bytes target    prot opt in    out    source              destination
Line 57: Line 63:
 
# iptables -Z
 
# iptables -Z
 
</pre>
 
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
+
The disadvantage is that by doingit this way you zero all counters in all rules. If it is not what you need,
 
you can just replace the rule with the same rule:
 
you can just replace the rule with the same rule:
 
<pre>
 
<pre>
# iptables -nv -L FORWARD
+
# iptables -nvx -L FORWARD
 
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
 
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
 
  pkts bytes target    prot opt in    out    source              destination
 
  pkts bytes target    prot opt in    out    source              destination
Line 66: Line 72:
 
   57  5564            all  --  *      *      0.0.0.0/0            192.168.0.117
 
   57  5564            all  --  *      *      0.0.0.0/0            192.168.0.117
 
# iptables -R FORWARD 1 -s 192.168.0.117
 
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
+
# iptables -nvx -L FORWARD
 
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
 
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
 
  pkts bytes target    prot opt in    out    source              destination
 
  pkts bytes target    prot opt in    out    source              destination
Line 77: Line 83:
 
more complicated situations.
 
more complicated situations.
  
; More than one VE on the node
+
; More than one container on the node
: Just add the rules like above for each VE IP.
+
: Just add the rules like above for each container IP.
  
; More than one IP per VE.
+
; More than one IP per container.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.
+
: For each IP add the rules like above. When counting the complete traffic of a container you have to summarize over all IPs that this container owns.
  
 
; More interfaces on the HN.
 
; More interfaces on the HN.
Line 89: Line 95:
 
Here are some scripting ideas
 
Here are some scripting ideas
  
first a small script to get all vz id's for later on
+
=== Get CTIDs of all running containers ===
 
<pre>
 
<pre>
 
host2:~/bin# cat vz-all-running
 
host2:~/bin# cat vz-all-running
Line 95: Line 101:
 
</pre>
 
</pre>
  
second a small script witch get all ip's of running vz's
+
=== Get all IPs of running containers ===
 
<pre>
 
<pre>
 
host2:~/bin# cat vz-all-running-ip
 
host2:~/bin# cat vz-all-running-ip
Line 101: Line 107:
 
</pre>
 
</pre>
  
and a small script to set up all needed iptable rules
+
=== Set up all needed iptables rules ===
 
<pre>
 
<pre>
 
host2:~/bin# cat vz-iptables-create-rules
 
host2:~/bin# cat vz-iptables-create-rules
Line 108: Line 114:
 
</pre>
 
</pre>
  
a small script to generate a traffic.log
+
=== Generate a traffic.log ===
 +
Please use crontab to run this script once per hour or day to collect your traffic statistics.
 +
 
 +
(Warning, the counters can overflow if there is too much traffic within that period. Would recommend 15 minute intervals if you expect a lot of traffic)
 
<pre>
 
<pre>
 
host2:~/bin# cat vz-generate-traffic-log
 
host2:~/bin# cat vz-generate-traffic-log
Line 116: Line 125:
 
   echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
 
   echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
 
   echo -n " $i " >> $trafficlog
 
   echo -n " $i " >> $trafficlog
   echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
+
   echo `iptables -nvx -L FORWARD | grep " $i " | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
 
  done
 
  done
 
  # reset the counter
 
  # reset the counter
 
  iptables -Z
 
  iptables -Z
  # update the ip table rules if there is a any change in vz's
+
  # update the iptables rules if there is a any change in containers
 
  ./vz-iptables-create-rules
 
  ./vz-iptables-create-rules
  
 
   
 
   
  # copy the trafficlog file to a webserver where users can take their traffic
+
  # copy the trafficlog file to a webserver where users can see their traffic
 +
 
 
  # please mind to use
 
  # please mind to use
 
  # ssh-keygen -t rsa
 
  # ssh-keygen -t rsa
 
  # to generate ssh keys
 
  # to generate ssh keys
  # and append the new public key from your hardware node ~/.ssh/id_rsa.pub
+
  # and append the new public key from your hardware node (~/.ssh/id_rsa.pub)
 
  # to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
 
  # to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
 +
# in order for the below scp command to not ask for root password
 
  scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic
 
  scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic
 
   
 
   
Line 135: Line 146:
 
  cp /dev/null $trafficlog
 
  cp /dev/null $trafficlog
 
  # start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
 
  # start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
  wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/login/traffic-read.php?HN=$HOSTNAME -O /dev/null
+
# please mind to use .htaccess to secure this
 +
  wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null
 
</pre>
 
</pre>
  
a small sample php script to store the trafficlog in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
+
=== Sample php script to store the trafficlog in a database ===
<pre>
+
 
 +
Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS  
 +
<source lang="text">
 +
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
 +
</source>
 +
<source lang="php">
 
<?
 
<?
 
  $MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
 
  $MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
Line 145: Line 162:
 
  $MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";
 
  $MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";
 
   
 
   
  mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");
+
  mysql_connect($MySQL_Host,$MySQL_User,$MySQL_Passw);
  
 
  $HN=trim(addslashes($_GET["HN"])); // Hardware Node
 
  $HN=trim(addslashes($_GET["HN"])); // Hardware Node
  
  $handle = fopen ("tmp/$HN","r");
+
  $handle = fopen ("tmp/{$HN}-traffic","r");
 
  while (!feof($handle)) {
 
  while (!feof($handle)) {
 
   $line = fgets($handle, 4096);
 
   $line = fgets($handle, 4096);
 
   list($date,$time,$ip,$traffic)=explode(" ",$line);
 
   list($date,$time,$ip,$traffic)=explode(" ",$line);
   if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
+
   if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('{$ip}','{$date} {$time}','{$traffic}')");}
 
  }  
 
  }  
 
  fclose($handle);
 
  fclose($handle);
</pre>
+
?>
 +
</source>
  
SQL querry to get the traffic for the last 30 days
+
=== A SQL query to get the traffic for the last 30 days ===
<pre>
+
<source lang="mysql">
 
SELECT sum(bytes)
 
SELECT sum(bytes)
 
FROM Traffic
 
FROM Traffic
Line 165: Line 183:
 
AND measuringtime > ( now() - INTERVAL 1 MONTH)
 
AND measuringtime > ( now() - INTERVAL 1 MONTH)
 
GROUP BY ip
 
GROUP BY ip
</pre>
+
</source>
 +
 
 +
=== Notes ===
  
You can use crontab to run this script once per hour or day to collect your traffic statistics.
+
As you see this way can be time-consuming in case of a big number of containers.  
  
As you see this way can be time-consuming in case of big number of VEs.
 
 
So if anybody has scripts that automate all the process — you are welcome!
 
So if anybody has scripts that automate all the process — you are welcome!
  
[[Category: HOWTO]]
+
== See also ==
 +
* [[Traffic accounting through proc]]
 +
 
 
[[Category: Networking]]
 
[[Category: Networking]]
 +
[[Category: Monitoring]]

Latest revision as of 09:28, 15 December 2011

Suppose you need to know how much traffic your containers eat. It can be easily done using iptables.

Situation description[edit]

Let's consider the very simple situation: one container with one IP address on the Hardware Node with only one network interface. To be more exact, assume that container ID is 200, the IP address of the HN is 192.168.0.56, the network interface name is eth0, and the IP address of the container is 192.168.0.117.

You wish to know how many bytes container 200 eats. One more assumption is that there are no iptables rules on HN now. All these assumption are only for clarity!

Solution[edit]

Almost any traffic that goes to and from a container can be catched by FORWARD chain of iptables module in container0, thus we add such rules:

# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117

It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted. To obtain current traffic usage of container you can issue the command:

# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
 pkts bytes target     prot opt in     out     source               destination
    8   832            all  --  *      *       192.168.0.117        0.0.0.0/0
   15  1052            all  --  *      *       0.0.0.0/0            192.168.0.117

Bytes column is the column we need. It's worth saying, that restarting a container doesn't affect accounting, it remains right. But if you restart your hardware node, all the rules and consequently statistics are dropped. So it is recommended to

  • run some cron job that dumps statistics to some file
  • add init script that creates iptables rules on HN start.

If you want to process the results with a script it is useful to use the "-x" or "--exact" option of iptables

# iptables -nvx -L FORWARD

You will get the exact value of the packet and byte counters, instead of only the rounded number in K’s (multiples of 1000) M’s (multiples of 1000K) or G’s (multiples of 1000M).

As is easy to see, it's not per-container statistic, but rather per-IP statistic. Thus you must be careful then changing container IP addresses, otherwise you'll get mess of results.

By saying almost any traffic I mean that traffic between a container and container0 is not accounted by rules above. Not sure if it can be useful for anybody, but to account such traffic these rules are needed:

iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117

To observe results:

# iptables -nvx -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
 pkts bytes target     prot opt in     out     source               destination
   35  4533            all  --  venet0 *       0.0.0.0/0            192.168.0.117
# iptables -nvx -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
 pkts bytes target     prot opt in     out     source               destination
   48  4724            all  --  *      venet0  192.168.0.117        0.0.0.0/0

If you need to zero counters this command works:

# iptables -Z

The disadvantage is that by doingit this way you zero all counters in all rules. If it is not what you need, you can just replace the rule with the same rule:

# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
 pkts bytes target     prot opt in     out     source               destination
   44  5151            all  --  *      *       192.168.0.117        0.0.0.0/0
   57  5564            all  --  *      *       0.0.0.0/0            192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0            all  --  *      *       192.168.0.117        0.0.0.0/0
   57  5564            all  --  *      *       0.0.0.0/0            192.168.0.117

More complicated cases[edit]

Well, now, when we know how to work in the easiest case, we'll try to understand what to do in more complicated situations.

More than one container on the node
Just add the rules like above for each container IP.
More than one IP per container.
For each IP add the rules like above. When counting the complete traffic of a container you have to summarize over all IPs that this container owns.
More interfaces on the HN.
Nothing to do! :)

Scripting[edit]

Here are some scripting ideas

Get CTIDs of all running containers[edit]

host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'

Get all IPs of running containers[edit]

host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip

Set up all needed iptables rules[edit]

host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`;  do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`;  do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1

Generate a traffic.log[edit]

Please use crontab to run this script once per hour or day to collect your traffic statistics.

(Warning, the counters can overflow if there is too much traffic within that period. Would recommend 15 minute intervals if you expect a lot of traffic)

host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
 do
  echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
  echo -n " $i " >> $trafficlog
  echo `iptables -nvx -L FORWARD | grep " $i " | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
 done
 # reset the counter
 iptables -Z
 # update the iptables rules if there is a any change in containers
 ./vz-iptables-create-rules

 
 # copy the trafficlog file to a webserver where users can see their traffic

 # please mind to use
 # ssh-keygen -t rsa
 # to generate ssh keys
 # and append the new public key from your hardware node (~/.ssh/id_rsa.pub)
 # to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
 # in order for the below scp command to not ask for root password
 scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic
 
 # clear the copied trafficlog
 cp /dev/null $trafficlog
 # start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
 # please mind to use .htaccess to secure this 
 wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null

Sample php script to store the trafficlog in a database[edit]

Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS

HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
<?
 $MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
 $MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
 $MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";
 
 mysql_connect($MySQL_Host,$MySQL_User,$MySQL_Passw);

 $HN=trim(addslashes($_GET["HN"])); // Hardware Node

 $handle = fopen ("tmp/{$HN}-traffic","r");
 while (!feof($handle)) {
   $line = fgets($handle, 4096);
   list($date,$time,$ip,$traffic)=explode(" ",$line);
   if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('{$ip}','{$date} {$time}','{$traffic}')");}
 } 
 fclose($handle);
?>

A SQL query to get the traffic for the last 30 days[edit]

SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip

Notes[edit]

As you see this way can be time-consuming in case of a big number of containers.

So if anybody has scripts that automate all the process — you are welcome!

See also[edit]