Traffic shaping with tc
Sometimes it's necessary to limit traffic bandwidth from and to a container.
You can do it using ordinary tc
tool.
Contents
Packet routes
First of all, a few words about how packets travel from and to a container. Suppose we have Hardware Node (HN) with a container (CT) on it, and this container talks to some Remote Host (RH). HN has one "real" network interface eth0 and, thanks to OpenVZ, there is also "virtual" network interface venet0. Inside the container we have interface venet0:0.
venet0:0 venet0 eth0 CT >------------->-------------> HN >--------->--------> RH venet0:0 venet0 eth0 CT <-------------<-------------< HN <---------<--------< RH
Limiting outgoing bandwidth
We can limit container outgoing bandwidth by setting the tc filter on eth0.
DEV=eth0 tc qdisc del dev $DEV root tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 100mbit tc class add dev $DEV parent 1: classid 1:1 cbq rate 256kbit allot 1500 prio 5 bounded isolated tc filter add dev $DEV parent 1: protocol ip prio 16 u32 match ip src X.X.X.X flowid 1:1 tc qdisc add dev $DEV parent 1:1 sfq perturb 10
X.X.X.X is an IP address of container.
Limiting incoming bandwidth
This can be done by setting the tc
filter on venet0
:
DEV=venet0 tc qdisc del dev $DEV root tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 100mbit tc class add dev $DEV parent 1: classid 1:1 cbq rate 256kbit allot 1500 prio 5 bounded isolated tc filter add dev $DEV parent 1: protocol ip prio 16 u32 match ip dst X.X.X.X flowid 1:1 tc qdisc add dev $DEV parent 1:1 sfq perturb 10
Note that X.X.X.X
is an IP address of container.
Limiting CT to HN talks
As you can see, two filters above don't limit container to HN talks. I mean a container can emit as much traffic as it wishes. To make such a limitation from the HN, it is necessary to use tc police on venet0:
DEV=venet0 tc filter add dev $DEV parent 1: protocol ip prio 20 u32 match u32 1 0x0000 police rate 2kbit buffer 10k drop flowid :1
Limiting packets per second rate from container
To prevent dos atacks from the container you can limit packets per second rate using iptables.
DEV=eth0
iptables -I FORWARD 1 -o $DEV -s X.X.X.X -m limit --limit 200/sec -j ACCEPT
iptables -I FORWARD 2 -o $DEV -s X.X.X.X -j DROP
Here X.X.X.X
is an IP address of container.
An alternate approach using HTB
For details refer to the HTB Home Page
#!/bin/sh
#
# Incoming traffic control
#
CT_IP1=$1
CT_IP2=$2
DEV=venet0
#
tc qdisc del dev $DEV root
#
tc qdisc add dev $DEV root handle 1: htb default 10
#
tc class add dev $DEV parent 1: classid 1:1 htb rate 100mbit burst 15k
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 10mbit ceil 10mbit burst 15k
tc class add dev $DEV parent 1:1 classid 1:20 htb rate 20mbit ceil 20mbit burst 15k
tc class add dev $DEV parent 1:1 classid 1:30 htb rate 30mbit ceil 30mbit burst 15k
#
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10
#
if [ ! -z $CT_IP1 ]; then
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dst "$CT_IP1" flowid 1:20
fi
if [ ! -z $CT_IP2 ]; then
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dst "$CT_IP2" flowid 1:30
fi
#
echo;echo "tc configuration for $DEV:"
tc qdisc show dev $DEV
tc class show dev $DEV
tc filter show dev $DEV
#
# Outgoing traffic control
#
DEV=eth0
#
tc qdisc del dev $DEV root
#
tc qdisc add dev $DEV root handle 1: htb default 10
#
tc class add dev $DEV parent 1: classid 1:1 htb rate 100mbit burst 15k
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 10mbit ceil 10mbit burst 15k
tc class add dev $DEV parent 1:1 classid 1:20 htb rate 20mbit ceil 20mbit burst 15k
tc class add dev $DEV parent 1:1 classid 1:30 htb rate 30mbit ceil 30mbit burst 15k
#
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10
#
if [ ! -z $CT_IP1 ]; then
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip src "$CT_IP1" flowid 1:20
fi
if [ ! -z $CT_IP2 ]; then
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip src "$CT_IP2" flowid 1:30
fi
#
echo;echo "tc configuration for $DEV:"
tc qdisc show dev $DEV
tc class show dev $DEV
tc filter show dev $DEV
Autogen rules for CT
#!/usr/bin/env python
# -*- coding: utf-8 -*-
######################################
## by VoidLess, lepus.su
######################################
import fnmatch
import os
######################################
## НАСТРОЙКИ СКРИПТА
confdir = "/etc/vz/conf/"
rulefile = "/etc/vz/shaper/rules.sh"
# шейпятся только эти тарифы
table_band = {
#ORIGIN_SAMPLE : канал
"SSD8": "100mbit",
"SSD7": "100mbit",
"SSD6": "100mbit",
"SSD5": "100mbit",
"SSD4": "100mbit",
"SSD3": "100mbit",
"SSD2": "100mbit",
"SSD1": "100mbit"
}
######################################
## ШАБЛОН СКРИПТА НАСТРОЙКИ ШЕЙПЕРА
fileheader = """#!/bin/bash
# shaper configuration file
##### ЧИСТКА ОТ СТАРЫХ ПРАВИЛ #####
tc qdisc del dev eth0 root
tc -s qdisc ls dev eth0
tc qdisc del dev venet0 root
tc -s qdisc ls dev venet0
##### ЧИСТКА ОТ СТАРЫХ ПРАВИЛ #####
"""
# eth0
eth0header = """
DEV=eth0
tc qdisc del dev $DEV root 2>/dev/null
tc qdisc add dev $DEV root handle 1: htb default 1 r2q 3000
tc class add dev $DEV parent 1: classid 1:1 htb rate 1000mbit burst 10mb
"""
eth0template = """
# vzid {VPSID} tarif {BANDWIDTH} ip {IP}
tc class add dev $DEV parent 1:1 classid 1:{CLASS} htb rate {BANDWIDTH} ceil {BANDWIDTH} burst 1mb
tc qdisc add dev $DEV parent 1:{CLASS} handle {CLASS}: sfq perturb 5 #quantum 5000b
"""
eth0match = """
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip src {IP} flowid 1:{CLASS}
"""
# venet0
venet0header = """
DEV=venet0
tc qdisc del dev $DEV root 2>/dev/null
tc qdisc add dev $DEV root handle 1: htb default 1 r2q 3000
tc class add dev $DEV parent 1: classid 1:1 htb rate 1000mbit burst 10mb
"""
venet0template = """
# vzid {VPSID} tarif {BANDWIDTH} ip {IP}
tc class add dev $DEV parent 1:1 classid 1:{CLASS} htb rate {BANDWIDTH} ceil {BANDWIDTH} burst 1mb
tc qdisc add dev $DEV parent 1:{CLASS} handle {CLASS}: sfq perturb 5 #quantum 5000b
"""
venet0match = """
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dst {IP} flowid 1:{CLASS}
"""
######################################
## КОД СКРИПТА
# получает список и настройки виртуалок из папки конфигов OpenVZ
def getvz():
vzlist = []
for file in os.listdir(confdir):
if fnmatch.fnmatch(file, '*.conf') and file != "0.conf":
#берем список конфигов
vzid = file.split(".",1)[0]
vztarif = None
vzip = None
#открываем файл, узнаем название тарифа
with open(confdir + file) as f:
for line in f:
if line[0]=="#":
continue
ar = line.split("=")
if ar[0] == "ORIGIN_SAMPLE":
vztarif = ar[1].replace('"', '').strip()
continue
if ar[0] == "IP_ADDRESS":
vzip = ar[1].replace('"', '').strip().split(" ")
continue
#если в файле не написан тариф, или он левый, ничего не делаем
if vztarif == None or vzip == None or vztarif not in table_band:
continue
# в список записываем инфу про все vps
vzlist.append((vzid, table_band[vztarif], vzip))
return vzlist
# записывает команды настройки шейпера в sh файл
def genconf(vzlist):
with open(rulefile,"w+") as f:
# file header
f.write(fileheader)
# eth0 rules
f.write(eth0header)
shaper_id = 100
for vz in vzlist:
f.write(eth0template
.replace("{VPSID}",vz[0])
.replace("{CLASS}",str(shaper_id))
.replace("{BANDWIDTH}",vz[1])
.replace("{IP}",str(vz[2])))
for ip in vz[2]:
f.write(eth0match
.replace("{VPSID}",vz[0])
.replace("{CLASS}",str(shaper_id))
.replace("{IP}",ip))
shaper_id += 1
# venet0 rules
f.write(venet0header)
shaper_id = 100
for vz in vzlist:
f.write(venet0template
.replace("{VPSID}",vz[0])
.replace("{CLASS}",str(shaper_id))
.replace("{BANDWIDTH}",vz[1])
.replace("{IP}",str(vz[2])))
for ip in vz[2]:
f.write(venet0match
.replace("{VPSID}",vz[0])
.replace("{CLASS}",str(shaper_id))
.replace("{IP}",ip))
shaper_id += 1
# make in runnable
os.chmod(rulefile, 0755)
## основной код скрипта. выполнение функций
# получили список VPSок
vzlist = getvz()
# сгенерировали скрипт с правилами
genconf(vzlist)
# применили правила
os.system(rulefile)