OpenVZ Virtuozzo Containers Wiki - User contributions [en]

Talk:Setting up an iptables firewall

2013-06-14T11:16:34Z

Robferrer: /* Firewall rules in the wrong order? */ new section

The directions on this page for Container based firewalling didn't work for me at all.
However the Article at the Parallels Virtuozzo Knowledgebase regarding this issue worked perfectly.
URL: http://kb.parallels.com/en/746

== Hardware Firewall mode not working ==

I have tried step by step to enable a hardware lie firewall but i have big issues with existing virtual server that acts as a mailserver.
Basically with firewall activated the transaction time is around 48 seconds, with firewall deactivated about 0,700 seconds.
What on earth could possible be the cause? The FORWARD rule is that bad on VZ?

== Firewall rules in the wrong order? ==

I've been using the firewall script for a while and it works great. That is until I tried to ban an IP address, and it didn't work.

I'm not an iptables expert, so was a bit wary about messing around too much, but my theory is all the iptables -I (insert) should be iptables -A (append), which has the effect of running the rules in the oposite order to intended. This means the source I wanted to block was matching an OKPORT before getting to the BANNED section.

In fact to fix my problem I just moved the BANNED section between the DMZS and OKPORTS, which had the desired effect.

I'd love to see anyone's comments. [[User:Robferrer|Robferrer]] ([[User talk:Robferrer|talk]]) 07:16, 14 June 2013 (EDT)

Fair-share scheduling

2010-05-07T15:35:14Z

Robferrer: spelling (example)

Fair-share scheduling is a method of allotting CPU time to multiple processes in a defined way. When multiple processes need a CPU, operating systems frequently allot CPU time slices on a round robin, equal time basis for processes with the same priority. An example is that if there are 4 processes wanting the CPU then each process will get 1/4 of a second every second. This is not to say that each process will get 1/4 of a second all at one time. Most operating systems define a time slice, something like 50 msec (arbitrary number that doesn't necessarily represent any OS) and will give each process a time slice. So in this completely made up case, Each process will get 5 time slices every second.

Long ago someone said that we need a way of giving people with legitimate requirements more of the time than less needy people. "Need" often reflects what level of service the user wants to pay for. The fair-share scheduler was born. There have been a number of implementations over the years, but on of the most common was to simply assign weights to users such that one user may get twice as many time slices in a given time period as others. This has been expanded often to groups so that processes belonging to a group of users would be given the same number of time slices as another group.

For example if group A had 8 processes running and group B had 4, then each group would get half the time, meaning that each process in the A group would only get 6.25% of the time (8*6.5 = 50) and each process in group B would get 12.5% of the time.

There are a number of different implementations of the fair-share scheduler other than the ones described here. I know of one that actually kept track of the accumulated time over a time period for a user/group and would use that information in it's calculations. The theory was that if you'd been a cpu hog in the past, you might not get as much time right now. This calculation also was also time weighted such that your share of time in the last second was much more important than a second a few minutes ago.

Talk:Traffic accounting with iptables

2008-05-14T11:08:54Z

Robferrer: New page: == IPTables bytecounter overflow == I added a warning about the cron period. Our iptables bytecounters overflowed within an hour when we had constant download at 60Mbps. We calculated (ba...

== IPTables bytecounter overflow ==
I added a warning about the cron period. Our iptables bytecounters overflowed within an hour when we had constant download at 60Mbps. We calculated (based on the maxiumum value we had seen accounted and the 100Mbps network connection) that we should run the cron every 15 mins. Should consider even more frequently if using a Gbit connection.[[User:Robferrer|Robferrer]] 07:08, 14 May 2008 (EDT)

Traffic accounting with iptables

2008-05-14T11:05:34Z

Robferrer: /* Generate a traffic.log */

Suppose you need to know how much traffic your [[container]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one container with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[container]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[container]] is <tt>192.168.0.117</tt>.

You wish to know how many bytes container 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a container can be catched by FORWARD chain of iptables module in [[container0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of container you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth saying, that restarting a container doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

If you want to process the results with a script it is useful to use the "-x" or "--exact" option of iptables
<pre>
# iptables -nvx -L FORWARD
</pre>
You will get the exact value of the packet and byte counters, instead of only the rounded number in K’s (multiples of 1000) M’s (multiples of 1000K) or G’s (multiples of 1000M).

As is easy to see, it's not per-container statistic, but rather per-IP statistic. Thus you must be careful
then changing container IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[container]] and [[container0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nvx -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nvx -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that by doingit this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one container on the node
: Just add the rules like above for each container IP.

; More than one IP per container.
: For each IP add the rules like above. When counting the complete traffic of a container you have to summarize over all IPs that this container owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

=== Get CTIDs of all running containers ===
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

=== Get all IPs of running containers ===
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

=== Set up all needed iptables rules ===
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

=== Generate a traffic.log ===
Please use crontab to run this script once per hour or day to collect your traffic statistics.

(Warning, the counters can overflow if there is too much traffic within that period. Would recommend 15 minute intervals if you expect a lot of traffic)
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nvx -L FORWARD | grep " $i " | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the iptables rules if there is a any change in containers
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can see their traffic

# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node (~/.ssh/id_rsa.pub)
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# in order for the below scp command to not ask for root password
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

=== Sample php script to store the trafficlog in a database ===

Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<source lang="text">
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
</source>
<source lang="php">
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN-traffic","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</source>

=== A SQL query to get the traffic for the last 30 days ===
<source lang="mysql">
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</source>

=== Notes ===

As you see this way can be time-consuming in case of a big number of containers.

So if anybody has scripts that automate all the process — you are welcome!

== See also ==
* [[Traffic accounting through proc]]

[[Category: Networking]]
[[Category: Monitoring]]

Traffic accounting with iptables

2008-05-14T11:05:20Z

Robferrer: /* Generate a traffic.log */ - cron interval warning

Suppose you need to know how much traffic your [[container]]s eat. It can be easily done
using iptables.

== Situation description ==
Let's consider the very simple situation: one container with one IP address on the [[Hardware Node]]
with only one network interface. To be more exact, assume that [[container]] ID is <tt>200</tt>, the IP address of the [[HN]]
is <tt>192.168.0.56</tt>, the network interface name is <tt>eth0</tt>, and the IP address of the [[container]] is <tt>192.168.0.117</tt>.

You wish to know how many bytes container 200 eats. One more assumption is that there are no iptables rules
on HN now. All these assumption are only for clarity!

== Solution ==
Almost any traffic that goes to and from a container can be catched by FORWARD chain of iptables module in [[container0]],
thus we add such rules:
<pre>
# iptables -A FORWARD -s 192.168.0.117
# iptables -A FORWARD -d 192.168.0.117
</pre>
It means that all traffic forwarded to IP 192.168.0.117 and from IP 192.168.0.117 will be accounted.
To obtain current traffic usage of container you can issue the command:
<pre>
# iptables -nv -L FORWARD
Chain FORWARD (policy ACCEPT 243 packets, 28089 bytes)
pkts bytes target prot opt in out source destination
8 832 all -- * * 192.168.0.117 0.0.0.0/0
15 1052 all -- * * 0.0.0.0/0 192.168.0.117
</pre>
'''Bytes''' column is the column we need. It's worth saying, that restarting a container doesn't affect accounting,
it remains right. But if you restart your [[hardware node]], all the rules and consequently statistics are dropped.
So it is recommended to
* run some cron job that dumps statistics to some file
* add init script that creates iptables rules on [[HN]] start.

If you want to process the results with a script it is useful to use the "-x" or "--exact" option of iptables
<pre>
# iptables -nvx -L FORWARD
</pre>
You will get the exact value of the packet and byte counters, instead of only the rounded number in K’s (multiples of 1000) M’s (multiples of 1000K) or G’s (multiples of 1000M).

As is easy to see, it's not per-container statistic, but rather per-IP statistic. Thus you must be careful
then changing container IP addresses, otherwise you'll get mess of results.

By saying ''almost any traffic'' I mean that traffic between a [[container]] and [[container0]] is not accounted by rules above.
Not sure if it can be useful for anybody, but to account such traffic these rules are needed:
<pre>
iptables -I INPUT 1 -i venet0 -d 192.168.0.117
iptables -I OUTPUT 1 -o venet0 -s 192.168.0.117
</pre>

To observe results:
<pre>
# iptables -nvx -L INPUT
Chain INPUT (policy ACCEPT 542 packets, 63745 bytes)
pkts bytes target prot opt in out source destination
35 4533 all -- venet0 * 0.0.0.0/0 192.168.0.117
# iptables -nvx -L OUTPUT
Chain OUTPUT (policy ACCEPT 247 packets, 27847 bytes)
pkts bytes target prot opt in out source destination
48 4724 all -- * venet0 192.168.0.117 0.0.0.0/0
</pre>
If you need to zero counters this command works:
<pre>
# iptables -Z
</pre>
The disadvantage is that by doingit this way you zero all counters in all rules. If it is not what you need,
you can just replace the rule with the same rule:
<pre>
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
44 5151 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
# iptables -R FORWARD 1 -s 192.168.0.117
# iptables -nvx -L FORWARD
Chain FORWARD (policy ACCEPT 101 packets, 10715 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.0.117 0.0.0.0/0
57 5564 all -- * * 0.0.0.0/0 192.168.0.117
</pre>

== More complicated cases ==
Well, now, when we know how to work in the easiest case, we'll try to understand what to do in
more complicated situations.

; More than one container on the node
: Just add the rules like above for each container IP.

; More than one IP per container.
: For each IP add the rules like above. When counting the complete traffic of a container you have to summarize over all IPs that this container owns.

; More interfaces on the HN.
: Nothing to do! :)

== Scripting ==
Here are some scripting ideas

=== Get CTIDs of all running containers ===
<pre>
host2:~/bin# cat vz-all-running
vzlist -H -oveid | sed 's/ //g;'
</pre>

=== Get all IPs of running containers ===
<pre>
host2:~/bin# cat vz-all-running-ip
vzlist -H -o ip
</pre>

=== Set up all needed iptables rules ===
<pre>
host2:~/bin# cat vz-iptables-create-rules
for i in `./vz-all-running-ip`; do iptables -D FORWARD -s $i; iptables -D FORWARD -d $i; done >/dev/null 2>&1
for i in `./vz-all-running-ip`; do iptables -A FORWARD -s $i; iptables -A FORWARD -d $i; done >/dev/null 2>&1
</pre>

=== Generate a traffic.log ===
Please use crontab to run this script once per hour or day to collect your traffic statistics.
(Warning, the counters can overflow if there is too much traffic within that period. Would recommend 15 minute intervals if you expect a lot of traffic)
<pre>
host2:~/bin# cat vz-generate-traffic-log
trafficlog="/var/log/vz-traffic.log"
for i in `./vz-all-running-ip` ;
do
echo -n `date "+%Y-%m-%d %H:%M:%S"` >> $trafficlog
echo -n " $i " >> $trafficlog
echo `iptables -nvx -L FORWARD | grep " $i " | tr -s [:blank:] |cut -d' ' -f3| awk '{sum+=$1} END {print sum;}'` >> $trafficlog
done
# reset the counter
iptables -Z
# update the iptables rules if there is a any change in containers
./vz-iptables-create-rules

# copy the trafficlog file to a webserver where users can see their traffic

# please mind to use
# ssh-keygen -t rsa
# to generate ssh keys
# and append the new public key from your hardware node (~/.ssh/id_rsa.pub)
# to ~/.ssh/authorized_keys2 on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# in order for the below scp command to not ask for root password
scp $trafficlog USER@HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE/tmp/$HOSTNAME-traffic

# clear the copied trafficlog
cp /dev/null $trafficlog
# start a php script to store the traffic in a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
# please mind to use .htaccess to secure this
wget -q http://HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS/traffic-read.php?HN=$HOSTNAME -O /dev/null
</pre>

=== Sample php script to store the trafficlog in a database ===

Below script will process traffic.log and store the data into a MySQL Database on the HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS
<source lang="text">
HOST-TO-SHOW-THE-TRAFFIC-TO-THE-USERS:/var/www/OPENVZ-CONTROL-WEB-SITE# cat traffic-read.php
</source>
<source lang="php">
<?
$MySQL_Host="INSERT-YOUR-MYSQL-HOST-HERE";
$MySQL_User="INSERT-YOUR-MYSQL-USER-HERE";
$MySQL_Passw="INSERT-YOUR-MYSQL-PASSWORD-HERE";

mysql_connect("$MySQL_Host","$MySQL_User","$MySQL_Passw");

$HN=trim(addslashes($_GET["HN"])); // Hardware Node

$handle = fopen ("tmp/$HN-traffic","r");
while (!feof($handle)) {
$line = fgets($handle, 4096);
list($date,$time,$ip,$traffic)=explode(" ",$line);
if($traffic>0) {mysql($db,"insert into Traffic (ip,measuringtime,bytes) values('$ip','$date $time','$traffic')");}
}
fclose($handle);
?>
</source>

=== A SQL query to get the traffic for the last 30 days ===
<source lang="mysql">
SELECT sum(bytes)
FROM Traffic
WHERE ip = 'INSERT-YOUR-IP-HERE'
AND measuringtime > ( now() - INTERVAL 1 MONTH)
GROUP BY ip
</source>

=== Notes ===

As you see this way can be time-consuming in case of a big number of containers.

So if anybody has scripts that automate all the process — you are welcome!

== See also ==
* [[Traffic accounting through proc]]

[[Category: Networking]]
[[Category: Monitoring]]