Monitoring openvz resources using rrdtool

In this setup we're using rrdtool to collect data from several openvz hardware nodes to a central rrd server.

Contents

Setting up rrd serverEdit

Create a directory /var/rrd on central server.

mkdir /var/rrd

In /etc/xinetd.d create a file rrdsrv and adjust addresses and paths according to local settings.

service rrdsrv
{
  disable         = no
  socket_type     = stream
  protocol        = tcp
  wait            = no
  user            = root
  only_from       = 192.168.2.0/24
  server          = /usr/bin/rrdtool
  server_args     = - /var/rrd
}

Add 'rrdsrv' to /etc/services

rrdsrv                13900/tcp

Collect scriptEdit

Put the following script on the openvz hardware nodes (ubc_rrd_collect.sh):

#!/bin/sh

PATH="/bin:/sbin:/usr/bin:/usr/sbin"
SENDCOMMAND='netcat 192.168.2.1 13900'
HEARTBEAT=600

function error {
        echo $1
        exit 1
}

test -r /proc/bc/0/resources || error "Not openvz kernel?"

command='cat'
init=0
update=0
send=0

while [ ! -z "$1" ]; do
        case "$1" in
                send)
                        send=1
                        command=$SENDCOMMAND
                        ;;
                init)
                        init=1
                        ;;
                update)
                        update=1
                        ;;
        esac
        shift
done

VPSIDS=$(vzlist -H -o veid)

PARAMS=$(cat /proc/bc/0/resources | awk '{ print $1 }'| sort -u | xargs)
if [ "$send" -eq 1 ]; then
        # to spread the load we sleep for a random while (between 0 and 60 sec)
        number=$RANDOM
        let "number %= 60"
        sleep $number
fi

for VPSID in $VPSIDS; do
       RRDFILENAME="ubc_${VPSID}.rrd" 

        if [ "$init" -eq 1 ]; then
                exists=0
                if [ "$send" -eq 1 ]; then
                        echo "info $RRDFILENAME" | $command | grep -q "No such file or directory" || exists=1
                fi      
                
                if [ "$exists" -eq 0 ]; then
                (
                        echo -n "create ${RRDFILENAME} "
                        for param in $PARAMS; do
                                echo -n "DS:${param}_f:DERIVE:$HEARTBEAT:0:U "
                                echo -n "DS:${param}_h:GAUGE:$HEARTBEAT:0:U "
                                echo -n "DS:${param}_b:GAUGE:$HEARTBEAT:0:U "
                        done
                        echo -n "RRA:AVERAGE:0.5:1:600 "
                        echo -n "RRA:AVERAGE:0.5:6:700 "
                        echo -n "RRA:AVERAGE:0.5:24:775 "
                        echo -n "RRA:AVERAGE:0.5:288:797 "
                        echo -n "RRA:MAX:0.5:1:600 "
                        echo -n "RRA:MAX:0.5:6:700 "
                        echo -n "RRA:MAX:0.5:24:775 "
                        echo "RRA:MAX:0.5:288:797"
                ) | $command
                fi
        fi
 
        if [ "$update" -eq 1 ]; then
                (
                        cat /proc/bc/$VPSID/resources | awk -v rrdfile=$RRDFILENAME '
                        BEGIN { 
                                ORS=""; 
                                getline
                                keys=$1 "_f"; values=$6;
                                keys=keys ":" $1 "_h"; values=values ":" $2;
                                keys=keys ":" $1 "_b"; values=values ":" $4;
                        } 
                        {
                                keys=keys ":" $1 "_f"; values=values ":" $6;
                                keys=keys ":" $1 "_h"; values=values ":" $2;
                                keys=keys ":" $1 "_b"; values=values ":" $4;
                        }
                        END { print "update " rrdfile " -t " keys " N:" values "\n"; } '
                ) | $command
        fi
done

UsageEdit

ubc_rrd_collect.sh [update] [init] [send]

  update - run this to update rrd files
  init   - scan system and create rrd files
  send   - use this to send commands to rddtool

Migration and new containerEdit

When you have this script running in cron on every hardware node and you're migrating a container from one node to the other the script will automatically detect this and pick it up on the other node.

When you create a new container you have to run ubc_rrd_collect.sh init send and it will ping rddtool to create a new rrd database. It won't overwrite existing databases because it first checks if a database already exists.

Setting up cronEdit

Run update every 5 minutes:

*/5     *       *       *       *       /path/to/ubc_rrd_collect.sh update send | grep -v '^OK'

Run init every hour (or run it manually when needed):

0       *       *       *       *       /path/to/ubc_rrd_collect.sh init send | grep -v '^OK'