Monitoring openvz resources using nagios and snmp

From OpenVZ Virtuozzo Containers Wiki
Revision as of 07:19, 7 April 2009 by Kako (talk | contribs) (→‎check_vzquota Without SNMP: Hint to sudo problems)
Jump to: navigation, search

snmpd configuration

Debian Etch example: 

apt-get install snmpd

edit /etc/default/snmpd : remove -u snmp and replace 127.0.0.1 with your ip (ie : 207.46.250.119), Full/etc/default/snmpd example: 

export MIBDIRS=/usr/share/snmp/mibs
SNMPDRUN=yes
SNMPDOPTS='-Lsd -Lf /dev/null  -I -smux -p /var/run/snmpd.pid 207.46.250.119'
TRAPDRUN=no
TRAPDOPTS='-Lsd -p /var/run/snmptrapd.pid'

For Debian 4.x: 

export MIBDIRS=/usr/share/snmp/mibs
SNMPDRUN=yes
SNMPDOPTS='-Lsd -Lf /dev/null  -I -smux -p /var/run/snmpd.pid'
TRAPDRUN=no
TRAPDOPTS='-Lsd -p /var/run/snmptrapd.pid'

Create user(my_username) and add new mib. Password need a min. of 8 charactes. Username only characters: 

/etc/init.d/snmpd stop
echo rouser my_username priv >> /etc/snmp/snmpd.conf
echo "extend  .1.3.6.1.4.1.2021.51  beancounters  /bin/cat /proc/user_beancounters" >> /etc/snmp/snmpd.conf
echo "extend  .1.3.6.1.4.1.2021.52  vzquota  /bin/cat /proc/vz/vzquota" >> /etc/snmp/snmpd.conf
echo  createUser my_username MD5 my_password DES >> /var/lib/snmp/snmpd.conf
/etc/init.d/snmpd start

(Note that the createUser command goes into a separate file. On Centos5 this file is located in /var/net-snmp/snmpd.conf. Make sure you stop snmpd before putting the createUser command there!).

Testing snmp: 

snmpwalk   -v 3  -u my_username -l authPriv   -a MD5 -A my_password -x DES -X my_password  $(hostname -i)

Warning: the minimum pass phrase length is 8 characters.

nagios configuration

example nagios configuration

add to configuration: 

define command {
command_name check_snmp_openvz_on_port
# command_line /usr/local/bin/check_snmp_openvz.sh  $HOSTADDRESS$ PORT    USER    PASSWORD
command_line /usr/local/bin/check_snmp_openvz.sh  $HOSTADDRESS$ $ARG1$  $ARG2$  $ARG3$
}

define host {
        host_name   openvz-server
        alias       Serwer Openvz
        address     207.46.250.119
        use         generic-host
        contact_groups  admins
        }

define service{
        use                             generic-service
        host_name                       openvz-server
        service_description             Virtual Machines Limits
        check_command                   check_snmp_openvz_on_port!161!my_username!my_password
        max_check_attempts              1
        }

nagios plugin

It is shell script: 

# cat /usr/local/bin/check_snmp_openvz.sh
#!/bin/bash
HOST=$1
PORT=$2
USER=$3
PASS=$4
export FILE=/tmp/$HOST.beancounters
RET=0

DATA_TMP=`snmpwalk   -v 3  -u $USER -l authPriv   -a MD5 -A $PASS -x DES -X $PASS $HOST:$PORT .1.3.6.1.4.1.2021.51.4`
if [ "$?" != "0" ]; then
        echo "Unknown snmp error"
        exit 1
fi

DATA=`echo "$DATA_TMP"| perl -ne '/"(.*)"/ ; print "$1\n" ;'`

if [ -f $FILE ]; then
echo "$DATA" | perl  -n -e'
use Data::Dumper;
my $file=$ENV{"FILE"};
my $ret=0 ;
my $vid ;
my $resource ;
my $held ;
my $maxheld ;
my $barrier ;
my $limit ;
my $failcnt ;
my %beancounters ;
my %beancounters_old ;
while(<STDIN>){
        my %vmachine;
        if ( /\D*(\d+):.*/ ){ $vid=$1; $beancounters{$vid}=\%vmachine ; }
        if ( /^[\W\d]+([a-z]+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+).*/ ) {
                $resource=$1 ;
                $held=$2 ;
                $maxheld=$3 ;
                $barrier=$4 ;
                $limit=$5 ;
                $failcnt=$6 ;
                ${beancounters{$vid}}{$resource}=[$held , $maxheld , $barrier , $limit ,$failcnt ];
                if ( ($held  > $barrier) && ($barrier != 0) ) {
                        print "WARNING: Limits on $vid: $resource  held->$held , barrier->$barrier ( limit->$limit ) " ;
                        $ret=1;
                }
        }
}

# read and parse old data
open(MYINPUTFILE, "<$file");
while(<MYINPUTFILE>){
        my %vmachine;
        if ( /\D*(\d+):.*/ ){ $vid=$1; $beancounters_old{$vid}=\%vmachine ; }
        if ( /^[\W\d]+([a-z]+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+).*/ ) {
                $resource=$1 ;
                $held=$2 ;
                $maxheld=$3 ;
                $barrier=$4 ;
                $limit=$5 ;
                $failcnt=$6 ;
                ${beancounters_old{$vid}}{$resource}=[$held , $maxheld , $barrier , $limit ,$failcnt ];
        }
}

foreach my $vmachine_id (keys %beancounters) {
        foreach my $resource (keys %{$beancounters{$vmachine_id}} ) {
                if ( defined($beancounters{$vmachine_id}{$resource}[4]) && defined($beancounters_old{$vmachine_id}{$resource}[4]) ){
                        my $failcnt=$beancounters{$vmachine_id}{$resource}[4];
                        my $failcnt_old=$beancounters_old{$vmachine_id}{$resource}[4];
                        my $held=$beancounters{$vmachine_id}{$resource}[0];
                        my $maxheld=$beancounters{$vmachine_id}{$resource}[1];
                        my $barrier=$beancounters{$vmachine_id}{$resource}[2];
                        my $limit=$beancounters{$vmachine_id}{$resource}[3];
                        if ( $failcnt_old < $failcnt ){
                                print "CRITICAL: Incrased failcnt  $vmachine_id: $resource from $failcnt_old to $failcnt (held->$held , maxheld->$maxheld , barrier->$barrier , limit->$limit ) " ;
                                $ret=2;
                        }
                }
        }

}

# if ($ret == 0 ) { print "Ok. \n" ; }
# print Dumper(%beancounters_old) ;
# print "\n";
exit($ret);
'

RET1=$?
fi

echo "$DATA" > $FILE
#####################################################################################
######### quota check
#####################################################################################

DATA=`snmpwalk   -v 3  -u $USER -l authPriv   -a MD5 -A $PASS -x DES -X $PASS $HOST:$PORT .1.3.6.1.4.1.2021.52.4 \
|  perl -ne '/"(.*)"/ ; print "$1\n" ;'`

if [ "$?" != "0" ]; then
        echo "Unknown snmp error"
        exit 1
fi


echo "$DATA" | perl  -n -e'
my $vid ;
my $ret=0 ;
while(<STDIN>){
        my %vid;
        if ( /\D*(\d+):.*/ ){ $vid=$1; }
        if ( /\s*(\S+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+).*/ ){
                $resource=$1 ;
                $usage=$2 ;
                $softlimit=$3 ;
                $hardlimit=$4 ;
                $time=$5 ;
                $expire=$6 ;
                if ( $usage >= $softlimit ){
                        print "WARNING: VZquota limit exceeded on $vid: $resource  usage->$usage, softlimit->$softlimit, hardlimit->$hardlimit, time->$time, expire->$expire  " ;
                        $ret=1;
                }
        }
}
exit($ret);
'
RET2=$?

#####################################################################################
########### return
#####################################################################################

if [  $RET1  -gt $RET2  ]; then
        RET=$RET1
        else
        RET=$RET2
fi

if [  $RET  = 0  ]; then
        echo Ok.
fi
exit $RET

check_vzquota Without SNMP

#!/bin/bash
RET=0
DATA=`echo;sudo /usr/sbin/vzlist -1 | xargs -I {} bash -c "echo {}:;sudo /usr/sbin/vzquota stat {}"`
echo "$DATA" | perl  -n -e'
my $vid ;
my $ret=0 ;
while(<STDIN>){
        my %vid;
        if ( /^(\d+):.*/ ){ $vid=$1; }
        if ( /\D*(\d+):.*/ ){ $vid=$1; }
        if ( /\s*(\S+)\s+(\d+)\s+(\d+)\s+(\d+).*/ ){
                $resource=$1 ;
                $usage=$2 ;
                $softlimit=$3 ;
                $hardlimit=$4 ;
                if ( $usage >= $softlimit ){
                        print "WARNING: VZquota limit exceeded on $vid: $resource  usage->$usage, softlimit->$softlimit, hardlimit->$hardlimit, time->$time, expire->$expire  " ;
                        $ret=1;
                }
                                print "$vid:$resource $usage/$softlimit ";
        }
}
exit($ret);
'
RET=$?
echo
exit $RET

The script calls /usr/sbin/vzlist by sudo. When doing this it normally needs a password, which check_nrpe will not know. Because of this it is necessary that you append a line like the following to /etc/sudors (user name an path should be adapted to the right ones on your system): 

nagios   ALL=NOPASSWD: /usr/local/nagios/libexec/check_vzquota

Also don't forget to consider this on your nrpe.cfg, so that you call the script with sudo: 

command[check_ubc]=sudo /usr/local/nagios/libexec/check_vzquota

check_ubc Without SNMP

#!/bin/bash
export FILE=/tmp/check_ubc
RET=0

ubc_file='/proc/user_beancounters';
DATA='';
if [ -r $ubc_file ]; then
        DATA=`cat $ubc_file`
fi
if [ -z "$DATA" ]; then
        echo "CRITICAL - $ubc_file is not readable or empty. Maybe it is only readable for root and this script should be called by sudo.";
        exit 3;
fi

if [ -f $FILE ]; then
echo "$DATA" | perl  -n -e'
use Data::Dumper;
my $file=$ENV{"FILE"};
my $ret=0 ;
my $vid ;
my $resource ;
my $held ;
my $maxheld ;
my $barrier ;
my $limit ;
my $failcnt ;
my %beancounters ;
my %beancounters_old ;
while(<STDIN>){
        my %vmachine;
        if ( /\D*(\d+):.*/ ){ $vid=$1; $beancounters{$vid}=\%vmachine ; }
        if ( /^[\W\d]+([a-z]+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+).*/ ) {
                $resource=$1 ;
                $held=$2 ;
                $maxheld=$3 ;
                $barrier=$4 ;
                $limit=$5 ;
                $failcnt=$6 ;
                ${beancounters{$vid}}{$resource}=[$held , $maxheld , $barrier , $limit ,$failcnt ];
                if ( ($held  > $barrier) && ($barrier != 0) ) {
                        print "WARNING: Limits on $vid: $resource  held->$held , barrier->$barrier ( limit->$limit ) " ;
                        $ret=1;
                }
                                #print "$vid:$resource $held Barrier:$barrier ";
        }
}

# read and parse old data
open(MYINPUTFILE, "<$file");
while(<MYINPUTFILE>){
        my %vmachine;
        if ( /\D*(\d+):.*/ ){ $vid=$1; $beancounters_old{$vid}=\%vmachine ; }
        if ( /^[\W\d]+([a-z]+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+).*/ ) {
                $resource=$1 ;
                $held=$2 ;
                $maxheld=$3 ;
                $barrier=$4 ;
                $limit=$5 ;
                $failcnt=$6 ;
                ${beancounters_old{$vid}}{$resource}=[$held , $maxheld , $barrier , $limit ,$failcnt ];
        }
}

foreach my $vmachine_id (keys %beancounters) {
        foreach my $resource (keys %{$beancounters{$vmachine_id}} ) {
                if ( defined($beancounters{$vmachine_id}{$resource}[4]) && defined($beancounters_old{$vmachine_id}{$resource}[4]) ){
                        my $failcnt=$beancounters{$vmachine_id}{$resource}[4];
                        my $failcnt_old=$beancounters_old{$vmachine_id}{$resource}[4];
                        my $held=$beancounters{$vmachine_id}{$resource}[0];
                        my $maxheld=$beancounters{$vmachine_id}{$resource}[1];
                        my $barrier=$beancounters{$vmachine_id}{$resource}[2];
                        my $limit=$beancounters{$vmachine_id}{$resource}[3];
                        if ( $failcnt_old < $failcnt ){
                                print "CRITICAL: Incrased failcnt  $vmachine_id: $resource from $failcnt_old to $failcnt (held->$held , maxheld->$maxheld , barrier->$barrier , limit->$limit ) " ;
                                $ret=2;
                        }
                                                #print "$vmachine_id: Old_Failcnt: $failcnt_old Failcnt: $failcnt \n";
                }
        }

}

if ($ret == 0 ) { print "OK. \n" ; }
# print Dumper(%beancounters_old) ;
# print "\n";
exit($ret);
'

RET=$?
fi

echo "$DATA" > $FILE
exit $RET

The script needs to read the /proc/user_beancounters file. This is normally only readable for root. Because of this it is necessary that you append a line like the following to /etc/sudors (user name an path should be adapted to the right ones on your system): 

nagios   ALL=NOPASSWD: /usr/local/nagios/libexec/check_ubc

Also don't forget to consider this on your nrpe.cfg, so that you call the script with sudo: 

command[check_ubc]=sudo /usr/local/nagios/libexec/check_ubc
Retrieved from "https://wiki.openvz.org/index.php?title=Monitoring_openvz_resources_using_nagios_and_snmp&oldid=7196"