Difference between revisions of "Monitoring openvz resources using munin"

From OpenVZ Virtuozzo Containers Wiki
Jump to: navigation, search
(munin plugin: Added plugin from Jan (http://forum.openvz.org/index.php?t=msg&goto=15122))
(Wikified (at least some more), added reference to another plugin, which uses vzlist)
Line 1: Line 1:
 
[[Category: Monitoring]]
 
[[Category: Monitoring]]
  
{{wikify}}
+
There are several plugins available on this page to monitor beancounter values in [http://munin.projects.linpro.no/ Munin].
 +
The third one tries to combine the other two.
  
== munin plugin ==
+
= "Simple" munin plugin =
  
 
The plugin listed below grabs all the beancounters' values.
 
The plugin listed below grabs all the beancounters' values.
Line 29: Line 30:
  
 
if [ "$1" = "suggest" ]; then
 
if [ "$1" = "suggest" ]; then
 
 
         if [ -r /proc/bc/0/resources ]; then
 
         if [ -r /proc/bc/0/resources ]; then
 
             cat /proc/bc/0/resources |
 
             cat /proc/bc/0/resources |
 
             while read str; do
 
             while read str; do
 
 
                 vals=($str)
 
                 vals=($str)
 
                 echo ${vals[0]}
 
                 echo ${vals[0]}
 
             done
 
             done
 
 
                 exit 0
 
                 exit 0
 
         else
 
         else
Line 43: Line 41:
 
         fi
 
         fi
 
fi
 
fi
 
  
 
if [ "$1" = "config" ]; then
 
if [ "$1" = "config" ]; then
 
 
#      echo "graph_order down up"
 
#      echo "graph_order down up"
 
         echo "graph_title $ATTR beancounter for containers"
 
         echo "graph_title $ATTR beancounter for containers"
 
         echo 'graph_category system'
 
         echo 'graph_category system'
 
         echo "graph_info 'Containers beancounters info'"
 
         echo "graph_info 'Containers beancounters info'"
 
  
 
         for CTID in `ls -d1 /proc/bc/???`; do
 
         for CTID in `ls -d1 /proc/bc/???`; do
Line 69: Line 64:
 
fi;
 
fi;
  
 
+
for CTID in `ls -d1 /proc/bc/???`; do
        for CTID in `ls -d1 /proc/bc/???`; do
+
    id=`basename $CTID`
            id=`basename $CTID`
+
    grep $ATTR $CTID/resources |
            grep $ATTR $CTID/resources |
+
    while read str; do
            while read str; do
+
        vals=($str)
 
+
        name=${vals[0]}
                vals=($str)
+
        echo "$id.value ${vals[1]}"
                name=${vals[0]}
+
    done
                echo "$id.value ${vals[1]}"
+
done
            done
 
        done
 
 
 
 
</source>
 
</source>
  
Please note, you have to configure plugin to run as root:
+
= Extended Version for old system using user_beancounter =
 
 
<code>
 
[vebc*]
 
user root
 
</code>
 
 
 
at /etc/munin/plugin-conf.d/ somewhere.
 
 
 
 
 
Extended Version for old system using user_beancounter.
 
  
 
Put it with the munin plugins and make a link for every
 
Put it with the munin plugins and make a link for every
Line 125: Line 107:
  
 
if [ "$1" = "suggest" ]; then
 
if [ "$1" = "suggest" ]; then
 
 
         if [ -r /proc/bc/0/resources ]; then
 
         if [ -r /proc/bc/0/resources ]; then
 
             cat /proc/bc/0/resources |
 
             cat /proc/bc/0/resources |
 
             while read str; do
 
             while read str; do
 
 
                 vals=($str)
 
                 vals=($str)
 
                 echo ${vals[0]}
 
                 echo ${vals[0]}
 
             done
 
             done
 
 
                 exit 0
 
                 exit 0
 
         else
 
         else
Line 141: Line 120:
  
 
if [ "$1" = "config" ]; then
 
if [ "$1" = "config" ]; then
 
 
#      echo "graph_order down up"
 
#      echo "graph_order down up"
 
         echo "graph_title beancounter for CT$ATTR: $STATS"
 
         echo "graph_title beancounter for CT$ATTR: $STATS"
Line 149: Line 127:
 
         readme="false"
 
         readme="false"
 
         cat /proc/user_beancounters | while read myid stuff; do
 
         cat /proc/user_beancounters | while read myid stuff; do
 
 
                 line=""
 
                 line=""
 
 
                 if [ "$myid" == "$ATTR:" ]; then
 
                 if [ "$myid" == "$ATTR:" ]; then
 
 
                         readme="true"
 
                         readme="true"
 
                         line="$stuff"
 
                         line="$stuff"
 
                         echo $line
 
                         echo $line
 
                 else
 
                 else
 
 
                         loid=`echo $myid | sed -e 's/.*:/:/'`
 
                         loid=`echo $myid | sed -e 's/.*:/:/'`
 
 
                         if [ "$loid" == ":" ]; then
 
                         if [ "$loid" == ":" ]; then
 
 
                                 readme="false"
 
                                 readme="false"
 
                         fi
 
                         fi
 
                         if [ "$readme" == "true" ]; then
 
                         if [ "$readme" == "true" ]; then
 
 
                                 line="$myid $stuff"
 
                                 line="$myid $stuff"
 
                         echo $line
 
                         echo $line
 
                         fi
 
                         fi
 
                 fi
 
                 fi
 
 
         done | while read name value top warn max; do
 
         done | while read name value top warn max; do
 
 
                 okname="dummy"
 
                 okname="dummy"
 
 
                 for statname in $STATS; do
 
                 for statname in $STATS; do
 
 
                         if [ "$name" == "$statname" ]; then
 
                         if [ "$name" == "$statname" ]; then
 
 
                                 okname=$name
 
                                 okname=$name
 
                         fi
 
                         fi
 
 
                 done
 
                 done
  
 
                 if [ "$okname" != "dummy" ]; then
 
                 if [ "$okname" != "dummy" ]; then
 
 
                         echo $okname.label $name
 
                         echo $okname.label $name
 
                         echo $okname.warning $warn
 
                         echo $okname.warning $warn
Line 192: Line 156:
 
                 fi
 
                 fi
 
         done
 
         done
 
 
         exit 0
 
         exit 0
 
fi;
 
fi;
 
  
 
readme="false"
 
readme="false"
 
cat /proc/user_beancounters | while read myid stuff; do
 
cat /proc/user_beancounters | while read myid stuff; do
 
 
         line=""
 
         line=""
 
 
         if [ "$myid" == "$ATTR:" ]; then
 
         if [ "$myid" == "$ATTR:" ]; then
 
 
                 readme="true"
 
                 readme="true"
 
                 line="$stuff"
 
                 line="$stuff"
 
                 echo $line
 
                 echo $line
 
         else
 
         else
 
 
                 loid=`echo $myid | sed -e 's/.*:/:/'`
 
                 loid=`echo $myid | sed -e 's/.*:/:/'`
 
 
                 if [ "$loid" == ":" ]; then
 
                 if [ "$loid" == ":" ]; then
 
 
                         readme="false"
 
                         readme="false"
 
                 fi
 
                 fi
 
 
                 if [ "$readme" == "true" ]; then
 
                 if [ "$readme" == "true" ]; then
 
 
                         line="$myid $stuff"
 
                         line="$myid $stuff"
 
                 echo $line
 
                 echo $line
Line 223: Line 177:
 
         fi
 
         fi
 
done | while read name value x; do
 
done | while read name value x; do
 
 
         okname="dummy"
 
         okname="dummy"
 
 
         for statname in $STATS; do
 
         for statname in $STATS; do
 
 
                 if [ "$name" == "$statname" ]; then
 
                 if [ "$name" == "$statname" ]; then
 
 
                         okname=$name
 
                         okname=$name
 
                 fi
 
                 fi
 
         done
 
         done
 
 
         if [ "$okname" != "dummy" ]; then
 
         if [ "$okname" != "dummy" ]; then
 
 
                 echo $okname.value $value
 
                 echo $okname.value $value
 
         fi
 
         fi
  
 
done
 
done
 
 
</source>
 
</source>
  
Line 247: Line 194:
  
  
== Another plugin from Jan Tomasek ==
+
= Extended plugin from Jan Tomasek =
 
Jan has posted another plugin on http://forum.openvz.org/index.php?t=msg&goto=15122, where I've fixed two things:
 
Jan has posted another plugin on http://forum.openvz.org/index.php?t=msg&goto=15122, where I've fixed two things:
1. "exit 0" in the "config" block
+
# "exit 0" in the "config" block
2. Replaced "vals=($str); echo ${vals[0]}" with "echo ${str%% *}" (the former was causing problems I don't remember anymore)
+
# Replaced "vals=($str); echo ${vals[0]}" with "echo ${str%% *}" (the former was causing problems I don't remember anymore)
  
 
<source lang="bash">
 
<source lang="bash">
Line 341: Line 288:
 
</source>
 
</source>
  
 +
 +
= Munin plugin setup =
 +
== Run as root ==
 +
Please note, you have to configure plugin to run as root. Therefore, add the following to /etc/munin/plugin-conf.d/ somewhere:
 +
  [vebc*]
 +
  user root
 +
 +
== Installing the plugins ==
 +
There's a single plugin file, which can be installed several times and can put several values into the same graph.
 +
You should install the plugin from above to e.g. <code>/usr/local/share/munin/plugins/vebc_</code> and then put symlinks to there from /etc/munin/plugins.
 +
The following script allows you to handle this easily:
 
To install this, you can use the following script:
 
To install this, you can use the following script:
 
<source lang="bash">
 
<source lang="bash">
Line 376: Line 334:
 
rm $FILE
 
rm $FILE
 
</source>
 
</source>
 +
 +
 +
= Notes =
 +
== Another plugin, using vzlist ==
 +
There is another plugin available at http://muninexchange.projects.linpro.no/?view&phid=249 - using vzlist and PHP.
 +
Especially using "vzlist" to only gather the requested fields might be a faster approach.

Revision as of 23:55, 12 May 2008


There are several plugins available on this page to monitor beancounter values in Munin. The third one tries to combine the other two.

"Simple" munin plugin

The plugin listed below grabs all the beancounters' values.

#!/bin/sh
#
# plugin to monitor OpenVZ bean counters.
#
#
#%# family=auto
#%# capabilities=autoconf suggest

ATTR=`basename $0 | sed 's/^vebc_//g'`

if [ "$1" = "autoconf" ]; then
        if [ -r /proc/bc/0/resources ]; then
                echo yes
                exit 0
        else
                echo "no (/proc/bc/0/resources not found)"
                exit 1
        fi
fi

if [ "$1" = "suggest" ]; then
        if [ -r /proc/bc/0/resources ]; then
            cat /proc/bc/0/resources |
            while read str; do
                vals=($str)
                echo ${vals[0]}
            done
                exit 0
        else
                exit 1
        fi
fi

if [ "$1" = "config" ]; then
#       echo "graph_order down up"
        echo "graph_title $ATTR beancounter for containers"
        echo 'graph_category system'
        echo "graph_info 'Containers beancounters info'"

        for CTID in `ls -d1 /proc/bc/???`; do
            id=`basename $CTID`
            grep $ATTR $CTID/resources |
            while read str; do

                vals=($str)
                name=${vals[0]}
                echo ${id}.label $id
                echo "${id}.warning  ${vals[3]}"
                echo "${id}.critical  ${vals[4]}"
            done
        done

        exit 0
fi;

for CTID in `ls -d1 /proc/bc/???`; do
    id=`basename $CTID`
    grep $ATTR $CTID/resources |
    while read str; do
        vals=($str)
        name=${vals[0]}
        echo "$id.value ${vals[1]}"
    done
done

Extended Version for old system using user_beancounter

Put it with the munin plugins and make a link for every graph which should be produced named like:

vebc_VALUENAME1_VALUENAME2_..._CTID

e.g.: vebc_numflock_numpty_numsiginfo_101

#!/bin/sh
#
# plugin to monitor OpenVZ bean counters.
#
#
#%# family=auto
#%# capabilities=autoconf suggest

ATTR=`basename $0 | sed -e 's/^vebc_.*_//'`
STATS=`basename $0 | sed -e 's/^vebc_//' -e 's/_[0-9]*$//' -e 's/_/ /g'`

if [ "$1" = "autoconf" ]; then
        if [ -r /proc/bc/0/resources ]; then
                echo yes
                exit 0
        else
                echo "no (/proc/bc/0/resources not found)"
                exit 1
        fi
fi

if [ "$1" = "suggest" ]; then
        if [ -r /proc/bc/0/resources ]; then
            cat /proc/bc/0/resources |
            while read str; do
                vals=($str)
                echo ${vals[0]}
            done
                exit 0
        else
                exit 1
        fi
fi

if [ "$1" = "config" ]; then
#       echo "graph_order down up"
        echo "graph_title beancounter for CT$ATTR: $STATS"
        echo "graph_category CT$ATTR"
        echo "graph_info 'Container bean counters info'"

        readme="false"
        cat /proc/user_beancounters | while read myid stuff; do
                line=""
                if [ "$myid" == "$ATTR:" ]; then
                        readme="true"
                        line="$stuff"
                        echo $line
                else
                        loid=`echo $myid | sed -e 's/.*:/:/'`
                        if [ "$loid" == ":" ]; then
                                readme="false"
                        fi
                        if [ "$readme" == "true" ]; then
                                line="$myid $stuff"
                        echo $line
                        fi
                fi
        done | while read name value top warn max; do
                okname="dummy"
                for statname in $STATS; do
                        if [ "$name" == "$statname" ]; then
                                okname=$name
                        fi
                done

                if [ "$okname" != "dummy" ]; then
                        echo $okname.label $name
                        echo $okname.warning $warn
                        echo $okname.critical $max
                fi
        done
        exit 0
fi;

readme="false"
cat /proc/user_beancounters | while read myid stuff; do
        line=""
        if [ "$myid" == "$ATTR:" ]; then
                readme="true"
                line="$stuff"
                echo $line
        else
                loid=`echo $myid | sed -e 's/.*:/:/'`
                if [ "$loid" == ":" ]; then
                        readme="false"
                fi
                if [ "$readme" == "true" ]; then
                        line="$myid $stuff"
                echo $line
                fi
        fi
done | while read name value x; do
        okname="dummy"
        for statname in $STATS; do
                if [ "$name" == "$statname" ]; then
                        okname=$name
                fi
        done
        if [ "$okname" != "dummy" ]; then
                echo $okname.value $value
        fi

done

This is not too performant but should do and the graphs are much more readable then the first solution.


Extended plugin from Jan Tomasek

Jan has posted another plugin on http://forum.openvz.org/index.php?t=msg&goto=15122, where I've fixed two things:

  1. "exit 0" in the "config" block
  2. Replaced "vals=($str); echo ${vals[0]}" with "echo ${str%% *}" (the former was causing problems I don't remember anymore)
#!/bin/sh
#
# Munin's plugin to monitor OpenVZ bean counters.
#
# $Log$
# Revision 1.3 2007/07/19 12:57:00 Jan Tomasek <jan@tomasek.cz>
# * rewrited to work with /proc/bc/<VEID>/resources instead of
# /proc/user_beancounters, that simplified code and result
# is also bit faster.
# * added references to OpenVZ wiki
# Revision 1.3.1 2008/05/13 01:26:00 Daniel Hahler <http://daniel.hahler.de/>
# * Minor fixes
#   - "exit 0" in "config" block
#   - Use "echo ${str%% *}" in "suggest", instead of "vals=($str); echo ${vals[0]}"
#
# Original revision taken from:
# http://wiki.openvz.org/Monitoring_openvz_resources_using_munin
#
#%# family=auto
#%# capabilities=autoconf suggest

VEID=`basename $0 | sed -e 's/^vebc_.*_//'`;
STATS=`basename $0 | sed -e 's/^vebc_//' -e 's/_[0-9]*$//' -e 's/_/ /g'`

if [ "$1" = "autoconf" ]; then
    if [ -r /proc/bc/0/resources ]; then
        echo yes
        exit 0
    else
        echo "no (/proc/bc/0/resources not found)"
        exit 1
    fi
fi

if [ "$1" = "suggest" ]; then
    if [ -r /proc/bc/0/resources ]; then
        cat /proc/bc/0/resources | while read str; do
        # Print everything before " "
        echo ${str%% *}
        done
        exit 0
    else
        exit 1
    fi
fi

if [ ! -f /proc/bc/$VEID/resources ]; then
    exit 0;
fi

if [ "$1" = "config" ]; then
    #echo "graph_order down up"
    echo "graph_title VE$VEID: $STATS"
    echo "graph_vlabel bean counters"
    echo "graph_category VE$VEID"
    # Note on URLs. General graph info is by munin version 1.2.5
    # accepted even with HTML code. But for value.info it escapes URL,
    # I expect that authors of munin will note that in future and put
    # escaping even for graph.info.
    echo "graph_info VE bean counters info. Documentation of the OpenVZ resource management is located at <a href=\"
http://wiki.openvz.org/UBC\">http://wiki.openvz.org/UBC</a>."

    cat /proc/bc/$VEID/resources | while read name value top warn max stuff ; do
        for statname in $STATS; do
        if [ "$name" = "$statname" ]; then
            URL="http://wiki.openvz.org/$name"
            if [ "$warn" = "0" ]; then
                warn=$max
            fi
            echo $name.label $name
            echo $name.warning $warn
            echo $name.critical $max
            echo $name.info Description of this resource is located at $URL
        fi
        done
    done
    exit 0
fi;

cat /proc/bc/$VEID/resources | while read name value stuff ; do
    for statname in $STATS; do
    if [ "$name" = "$statname" ]; then
        echo $name".value "$value;
    fi
    done
done


Munin plugin setup

Run as root

Please note, you have to configure plugin to run as root. Therefore, add the following to /etc/munin/plugin-conf.d/ somewhere:

 [vebc*]
 user root

Installing the plugins

There's a single plugin file, which can be installed several times and can put several values into the same graph. You should install the plugin from above to e.g. /usr/local/share/munin/plugins/vebc_ and then put symlinks to there from /etc/munin/plugins. The following script allows you to handle this easily: To install this, you can use the following script:

#!/bin/bash

FILE=`mktemp /tmp/ln-vebc-XXXXXX`

cd /etc/munin/plugins

for resources in kmemsize \
    lockedpages_privvmpages_shmpages_physpages_vmguarpages_oomguarpages \
    numproc \
    numtcpsock_numflock_numpty_numsiginfo_numothersock_numiptent \
    tcpsndbuf_tcprcvbuf_othersockbuf_dgramrcvbuf \
    dcachesize \
    numfile
do
    for VE in 0 `/usr/sbin/vzlist | sed "s/^ *//" |grep '^[0-9]' | cut -f 1 -d " "` ; do
        ln -sf /usr/local/share/munin/plugins/vebc_ "vebc_"$resources"_"$VE
        echo "vebc_"$resources"_"$VE >> $FILE
    done
done

# remove no longer deserved links (ie. links pointing to machines
# which were destroyed or stoped)

find -type l -name vebc_\* | sed "s/\.\///" | while read LN; do
    if grep ^$LN$ $FILE >/dev/null; then
        true
    else
        rm $LN
    fi
done

rm $FILE


Notes

Another plugin, using vzlist

There is another plugin available at http://muninexchange.projects.linpro.no/?view&phid=249 - using vzlist and PHP. Especially using "vzlist" to only gather the requested fields might be a faster approach.