OpenVZ Virtuozzo Containers Wiki - User contributions [en]

Traffic accounting with iptables

2006-12-19T13:58:01Z

Golbs: /* Generate a traffic.log */ bugfix for exact values

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

If you want to process the results with a script it is useful to use the "-x" or "--exact" option of iptables
<pre>
# iptables -nvx -L FORWARD
</pre>
You will get the exact value of the packet and byte counters, instead of only the rounded number in K’s (multiples of 1000) M’s (multiples of 1000K) or G’s (multiples of 1000M).

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nvx -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nvx -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

=== Get VEIDs of all running VEs ===
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

=== Get all IPs of running VEs ===
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

=== Set up all needed iptables rules ===
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

=== Generate a traffic.log ===
Please use crontab to run this script once per hour or day to collect your traffic statistics.
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nvx -L FORWARD | grep " $i " | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the iptables rules if there is a any change in VEs
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can see their traffic

# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node (~/.ssh/id_rsa.pub)
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# in order for the below scp command to not ask for root password
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

=== Sample php script to store the trafficlog in a database ===

Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN-traffic","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</pre>

=== A SQL query to get the traffic for the last 30 days ===
<pre>
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</pre>

=== Notes ===

As you see this way can be time-consuming in case of a big number of VEs.

So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-12-19T13:57:01Z

Golbs: /* Solution */ add a hint for exact value

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

If you want to process the results with a script it is useful to use the "-x" or "--exact" option of iptables
<pre>
# iptables -nvx -L FORWARD
</pre>
You will get the exact value of the packet and byte counters, instead of only the rounded number in K’s (multiples of 1000) M’s (multiples of 1000K) or G’s (multiples of 1000M).

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nvx -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nvx -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

=== Get VEIDs of all running VEs ===
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

=== Get all IPs of running VEs ===
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

=== Set up all needed iptables rules ===
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

=== Generate a traffic.log ===
Please use crontab to run this script once per hour or day to collect your traffic statistics.
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep " $i " | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the iptables rules if there is a any change in VEs
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can see their traffic

# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node (~/.ssh/id_rsa.pub)
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# in order for the below scp command to not ask for root password
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

=== Sample php script to store the trafficlog in a database ===

Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN-traffic","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</pre>

=== A SQL query to get the traffic for the last 30 days ===
<pre>
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</pre>

=== Notes ===

As you see this way can be time-consuming in case of a big number of VEs.

So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-12-16T00:47:25Z

Golbs: /* Generate a traffic.log */ Bug fix

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

=== Get VEIDs of all running VEs ===
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

=== Get all IPs of running VEs ===
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

=== Set up all needed iptables rules ===
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

=== Generate a traffic.log ===
Please use crontab to run this script once per hour or day to collect your traffic statistics.
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep " $i " | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the iptables rules if there is a any change in VEs
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can see their traffic

# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node (~/.ssh/id_rsa.pub)
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# in order for the below scp command to not ask for root password
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

=== Sample php script to store the trafficlog in a database ===

Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN-traffic","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</pre>

=== A SQL query to get the traffic for the last 30 days ===
<pre>
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</pre>

=== Notes ===

As you see this way can be time-consuming in case of a big number of VEs.

So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-12-06T14:39:35Z

Golbs: /* bug fix in Generate a traffic.log*/

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

=== Get VEIDs of all running VEs ===
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

=== Get all IPs of running VEs ===
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

=== Set up all needed iptables rules ===
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

=== Generate a traffic.log ===
Please use crontab to run this script once per hour or day to collect your traffic statistics.
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep ' $i ' | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the iptables rules if there is a any change in VEs
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can see their traffic

# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node (~/.ssh/id_rsa.pub)
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# in order for the below scp command to not ask for root password
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

=== Sample php script to store the trafficlog in a database ===

Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN-traffic","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</pre>

=== A SQL query to get the traffic for the last 30 days ===
<pre>
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</pre>

=== Notes ===

As you see this way can be time-consuming in case of a big number of VEs.

So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-25T20:49:24Z

Golbs: /* Generate a traffic.log */

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

=== Get VEIDs of all running VEs ===
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

=== Get all IPs of running VEs ===
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

=== Set up all needed iptables rules ===
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

=== Generate a traffic.log ===
Please use crontab to run this script once per hour or day to collect your traffic statistics.
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the iptables rules if there is a any change in VEs
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can see their traffic

# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node (~/.ssh/id_rsa.pub)
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# in order for the below scp command to not ask for root password
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

=== Sample php script to store the trafficlog in a database ===

Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN-traffic","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</pre>

=== A SQL query to get the traffic for the last 30 days ===
<pre>
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</pre>

=== Notes ===

As you see this way can be time-consuming in case of a big number of VEs.

So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-25T20:35:08Z

Golbs: /* Scripting */

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

first a small script to get all vz id's for later on
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

second a small script witch get all ip's of running vz's
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

and a small script to set up all needed iptable rules
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

a small script to generate a traffic.log
Please use crontab to run this script once per hour or day to collect your traffic statistics.
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the ip table rules if there is a any change in vz's
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can take their traffic
# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node ~/.ssh/id_rsa.pub
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/login/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

a small sample php script to store the trafficlog in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN-traffic","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</pre>

SQL querry to get the traffic for the last 30 days
<pre>
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</pre>

As you see this way can be time-consuming in case of big number of VEs.
So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-25T20:34:19Z

Golbs: /* Scripting */

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

first a small script to get all vz id's for later on
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

second a small script witch get all ip's of running vz's
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

and a small script to set up all needed iptable rules
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

a small script to generate a traffic.log
Please use crontab to run this script once per hour or day to collect your traffic statistics.
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the ip table rules if there is a any change in vz's
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can take their traffic
# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node ~/.ssh/id_rsa.pub
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/login/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

a small sample php script to store the trafficlog in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</pre>

SQL querry to get the traffic for the last 30 days
<pre>
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</pre>

As you see this way can be time-consuming in case of big number of VEs.
So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-25T20:32:48Z

Golbs: /* Scripting */

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

first a small script to get all vz id's for later on
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

second a small script witch get all ip's of running vz's
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

and a small script to set up all needed iptable rules
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

a small script to generate a traffic.log
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the ip table rules if there is a any change in vz's
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can take their traffic
# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node ~/.ssh/id_rsa.pub
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/login/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

a small sample php script to store the trafficlog in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</pre>

SQL querry to get the traffic for the last 30 days
<pre>
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</pre>

You can use crontab to run this script once per hour or day to collect your traffic statistics.

As you see this way can be time-consuming in case of big number of VEs.
So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-25T20:30:17Z

Golbs: /* Scripting */

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

first a small script to get all vz id's for later on
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

second a small script witch get all ip's of running vz's
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

and a small script to set up all needed iptable rules
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

a small script to generate a traffic.log
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the ip table rules if there is a any change in vz's
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can take their traffic
# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node ~/.ssh/id_rsa.pub
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/login/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

a small sample php script to store the trafficlog in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</pre>

SQL querry to get the traffic for the last 30 days
<pre>
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</pre>

You can use crontab to run this script once per hour or day to collect your traffic statistics.

As you see this way can be time-consuming in case of big number of VEs.
So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-25T20:27:55Z

Golbs: /* Scripting */

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

first a small script to get all vz id's for later on
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

second a small script witch get all ip's of running vz's
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

and a small script to set up all needed iptable rules
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

a small script to generate a traffic.log
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the ip table rules if there is a any change in vz's
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can take their traffic
# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node ~/.ssh/id_rsa.pub
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/login/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

a small sample php script to store the trafficlog in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
</pre>

SQL querry to get the traffic for the last 30 days
<pre>
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</pre>

You can use crontab to run this script once per hour or day to collect your traffic statistics.

As you see this way can be time-consuming in case of big number of VEs.
So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-25T20:14:19Z

Golbs: /* Scripting */

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

first a small script to get all vz id's for later on
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

second a small script witch get all ip's of running vz's
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

and a small script to set up all needed iptable rules
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

a small script to generate a traffic.log
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the ip table rules if there is a any change in vz's
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can take their traffic
# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node ~/.ssh/id_rsa.pub
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/login/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

a small sample php script to store the trafficlog in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<pre>
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
</pre>

You can use crontab to run this script once per hour or day to collect your traffic statistics.

As you see this way can be time-consuming in case of big number of VEs.
So if anybody has scripts that automate all the process — you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-25T19:42:42Z

Golbs: /* Scripting */

Traffic accounting with iptables

2006-11-19T19:59:30Z

Golbs: /* Scripting */

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

first a small script to get all vz id's for later on
<pre>
host2:~/bin# cat vz-all-running
vzlist |grep run|tr -s [:blank:]|cut -d' ' -f2
</pre>

second a small script witch get all ip's of running vz's
<pre>
host2:~/bin# cat vz-all-running-ip
for i in `./vz-all-running`; do vzctl exec $i ifconfig |grep 'venet0:0' -A1|tail -n1|cut -d':' -f2|cut -d' ' -f1; done
</pre>

and a small script to set up all needed iptable rules
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>/dev/null
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>/dev/null
</pre>

a small script to generate a traffic.log
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
iptables -Z
</pre>
You can use crontab to run this script once per hour or day to collect your traffic statistics

As you see this way can be time-consuming in case of big number of VEs.
So if anybody has scripts that automate all the process - you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-19T19:56:11Z

Golbs: /* Scripting */

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

first a small script to get all vz id's for later on
<pre>
host2:~/bin# cat vz-all-running
vzlist |grep run|tr -s [:blank:]|cut -d' ' -f2
</pre>

second a small script witch get all ip's of running vz's
<pre>
host2:~/bin# cat vz-all-running-ip
for i in `./vz-all-running`; do vzctl exec $i ifconfig |grep 'venet0:0' -A1|tail -n1|cut -d':' -f2|cut -d' ' -f1; done
</pre>

and a small script to set up all needed iptable rules
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>/dev/null
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>/dev/null
</pre>

a small script to generate a traffic.log
<pre>
host2:~/bin# cat vz-iptables-traffic
trafficlog="./traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
iptables -Z
</pre>
You can use crontab to run this script once per hour or day to collect your traffic statistics

As you see this way can be time-consuming in case of big number of VEs.
So if anybody has scripts that automate all the process - you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-19T19:54:56Z

Golbs: /* Scripting */

Suppose you need to know how much traffic your [[VE]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one VE with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[VE]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[VE]] is <tt>192.168.0.117</tt>.

You wish to know how much bytes VE 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a VE can be catched by FORWARD chain of iptables module in [[VE0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of VE you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth to say, that restarting a VE doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

As is easy to see, it's not per-VE statistic, but rather per-IP statistic. Thus you must be careful
then changing VE IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[VE]] and [[VE0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nv -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nv -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that doing this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one VE on the node
: Just add the rules like above for each VE IP.

; More than one IP per VE.
: For each IP add the rules like above. When counting the complete traffic of a VE you have to summarize over all IPs that this VE owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

first a small script to get all vz id's for later on
<pre>
host2:~/bin# cat vz-all-running
vzlist |grep run|tr -s [:blank:]|cut -d' ' -f2
</pre>

second a small script witch get all ip's of running vz's
<pre>
host2:~/bin# cat vz-all-running-ip
for i in `./vz-all-running`; do vzctl exec $i ifconfig |grep 'venet0:0' -A1|tail -n1|cut -d':' -f2|cut -d' ' -f1; done
</pre>

and a small script to set all iptable rules
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>/dev/null
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>/dev/null
</pre>

a small script to generate a traffic.log
<pre>
host2:~/bin# cat vz-iptables-traffic
trafficlog="./traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nv -L FORWARD | grep $i | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
iptables -Z
</pre>
You can use crontab to run this script once per hour or day to collect your traffic statistics

As you see this way can be time-consuming in case of big number of VEs.
So if anybody has scripts that automate all the process - you are welcome!

[[Category: HOWTO]]
[[Category: Networking]]

Traffic accounting with iptables

2006-11-19T19:49:52Z

Golbs: /* Scripting */

Traffic accounting with iptables

2006-11-19T18:27:14Z

Golbs: /* Scripting */

Traffic accounting with iptables

2006-11-19T18:26:10Z

Golbs: /* Scripting */

Traffic accounting with iptables

2006-11-19T18:22:01Z

Golbs: /* Scripting */

Installation on Debian/old

2006-09-21T18:16:36Z

Golbs: /* Edit apt settings */

= Stable =

== Edit apt settings ==

add to your "/etc/apt/sources.list"

<pre>
deb http://debian.systs.org/ stable openvz
</pre>

and get the new package lists

<pre>
# apt-get update
</pre>

== Packages at debian.systs.org (dso) ==

Debian Packages used for OpenVZ (i386):

kernel(s): Version 022stab078.14
<pre>
kernel-image-2.6.8-stable-ovz
kernel-image-2.6.8-stable-ovz-smp
kernel-image-2.6.8-stable-ovz-entnosplit
kernel-image-2.6.8-stable-ovz-enterprise
</pre>

tool(s):
<pre>
vzctl
vzquota
</pre>

template(s):
<pre>
vzctl-template-debian
</pre>

== Installing the utilities and kernels ==

Example:

# aptitude install kernel-image-2.6.8-stable-ovz vzctl vzquota vzctl-template-debian

Maybe you need to update you "linux-loader" like lilo or grub:

for the "GRUB":
# /sbin/grub-update

Reboot in your new Debian Stable OpenVZ System

# reboot

That's all :-)

=Unstable=

OpenVZ is now a part of Debian Sid (a.k.a. "unstable") repository. This article describes how to install OpenVZ on a Debian Sid system.

== Installing the utilities and kernel patch ==

To install the OpenVZ kernel patch and utilities, run the following:
<pre>
apt-get update
apt-get install kernel-patch-openvz vzctl vzquota
</pre>

== Creating the kernel package ==

To create a kernel package, you need to download and unpack 2.6.16 “vanilla” kernel first:
<pre>
cd /usr/src/
wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.tar.bz2
tar xjf linux-2.6.16.tar.bz2
cd linux-2.6.16
</pre>

(Note that you do need a vanilla kernel for this, because the OpenVZ kernel patch doesn't apply cleanly to the Debian linux-source-2.6.16 package; see [http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=377707 Debian bug #377707].)

Next, get the proper kernel config from [http://download.openvz.org/kernel/devel/026test015.1/configs/ download.openvz.org]. Below is the example of using smp config for i686:

<pre>
wget http://download.openvz.org/kernel/devel/026test015.1/configs/kernel-2.6.16-026test015-i686-smp.config.ovz
mv kernel-2.6.16-026test015-i686-smp.config.ovz .config
</pre>

{{Note|This example uses a config file for the 026test015 kernel patch. If the kernel-patch-openvz package you installed is a different version, download a config file that corresponds with your version}}

Now you can apply openvz kernel patch and compile the kernel:

<pre>
make-kpkg --added_patches openvz
</pre>

[[Category: HOWTO]]
[[Category: Installation]]

Installation on Debian/old

2006-09-21T18:14:07Z

Golbs: /* Edit apt settings */

= Stable =

== Edit apt settings ==

add to your "/etc/apt/sources.list"

<pre>
deb http://debian.systs.org/ stable openvz
</pre>

get the new package lists

<pre>
# apt-get update
</pre>

== Packages at debian.systs.org (dso) ==

Debian Packages used for OpenVZ (i386):

kernel(s): Version 022stab078.14
<pre>
kernel-image-2.6.8-stable-ovz
kernel-image-2.6.8-stable-ovz-smp
kernel-image-2.6.8-stable-ovz-entnosplit
kernel-image-2.6.8-stable-ovz-enterprise
</pre>

tool(s):
<pre>
vzctl
vzquota
</pre>

template(s):
<pre>
vzctl-template-debian
</pre>

== Installing the utilities and kernels ==

Example:

# aptitude install kernel-image-2.6.8-stable-ovz vzctl vzquota vzctl-template-debian

Maybe you need to update you "linux-loader" like lilo or grub:

for the "GRUB":
# /sbin/grub-update

Reboot in your new Debian Stable OpenVZ System

# reboot

That's all :-)

=Unstable=

OpenVZ is now a part of Debian Sid (a.k.a. "unstable") repository. This article describes how to install OpenVZ on a Debian Sid system.

== Installing the utilities and kernel patch ==

To install the OpenVZ kernel patch and utilities, run the following:
<pre>
apt-get update
apt-get install kernel-patch-openvz vzctl vzquota
</pre>

== Creating the kernel package ==

To create a kernel package, you need to download and unpack 2.6.16 “vanilla” kernel first:
<pre>
cd /usr/src/
wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.tar.bz2
tar xjf linux-2.6.16.tar.bz2
cd linux-2.6.16
</pre>

(Note that you do need a vanilla kernel for this, because the OpenVZ kernel patch doesn't apply cleanly to the Debian linux-source-2.6.16 package; see [http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=377707 Debian bug #377707].)

Next, get the proper kernel config from [http://download.openvz.org/kernel/devel/026test015.1/configs/ download.openvz.org]. Below is the example of using smp config for i686:

<pre>
wget http://download.openvz.org/kernel/devel/026test015.1/configs/kernel-2.6.16-026test015-i686-smp.config.ovz
mv kernel-2.6.16-026test015-i686-smp.config.ovz .config
</pre>

{{Note|This example uses a config file for the 026test015 kernel patch. If the kernel-patch-openvz package you installed is a different version, download a config file that corresponds with your version}}

Now you can apply openvz kernel patch and compile the kernel:

<pre>
make-kpkg --added_patches openvz
</pre>

[[Category: HOWTO]]
[[Category: Installation]]

Installation on Debian/old

2006-09-21T18:12:18Z

Golbs: /* Edit apt settings */

= Stable =

== Edit apt settings ==

add to your "/etc/apt/sources.list"

<pre>
deb http://debian.systs.org/ stable openvz
</pre>

apt-get update

== Packages at debian.systs.org (dso) ==

Debian Packages used for OpenVZ (i386):

kernel(s): Version 022stab078.14
<pre>
kernel-image-2.6.8-stable-ovz
kernel-image-2.6.8-stable-ovz-smp
kernel-image-2.6.8-stable-ovz-entnosplit
kernel-image-2.6.8-stable-ovz-enterprise
</pre>

tool(s):
<pre>
vzctl
vzquota
</pre>

template(s):
<pre>
vzctl-template-debian
</pre>

== Installing the utilities and kernels ==

Example:

# aptitude install kernel-image-2.6.8-stable-ovz vzctl vzquota vzctl-template-debian

Maybe you need to update you "linux-loader" like lilo or grub:

for the "GRUB":
# /sbin/grub-update

Reboot in your new Debian Stable OpenVZ System

# reboot

That's all :-)

=Unstable=

OpenVZ is now a part of Debian Sid (a.k.a. "unstable") repository. This article describes how to install OpenVZ on a Debian Sid system.

== Installing the utilities and kernel patch ==

To install the OpenVZ kernel patch and utilities, run the following:
<pre>
apt-get update
apt-get install kernel-patch-openvz vzctl vzquota
</pre>

== Creating the kernel package ==

To create a kernel package, you need to download and unpack 2.6.16 “vanilla” kernel first:
<pre>
cd /usr/src/
wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.tar.bz2
tar xjf linux-2.6.16.tar.bz2
cd linux-2.6.16
</pre>

(Note that you do need a vanilla kernel for this, because the OpenVZ kernel patch doesn't apply cleanly to the Debian linux-source-2.6.16 package; see [http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=377707 Debian bug #377707].)

Next, get the proper kernel config from [http://download.openvz.org/kernel/devel/026test015.1/configs/ download.openvz.org]. Below is the example of using smp config for i686:

<pre>
wget http://download.openvz.org/kernel/devel/026test015.1/configs/kernel-2.6.16-026test015-i686-smp.config.ovz
mv kernel-2.6.16-026test015-i686-smp.config.ovz .config
</pre>

{{Note|This example uses a config file for the 026test015 kernel patch. If the kernel-patch-openvz package you installed is a different version, download a config file that corresponds with your version}}

Now you can apply openvz kernel patch and compile the kernel:

<pre>
make-kpkg --added_patches openvz
</pre>

[[Category: HOWTO]]
[[Category: Installation]]