Resource shortage
Sometimes you see a strange fails from some programs inside your Virtual Environment. In some cases it means one of the resources controlled by OpenVZ hit the limit.
The first thing to do is to check the contents of /proc/user_beancounters file in your VE. The last column of output is fail counter. Each time a resource hit the limit, fail counter is increasing. So, if you see non-zero values in failcnt column that means something is wrong.
There are two ways to fix the situation: reconfugure (in some cases recompile) the application, or change the resource management settings.
UBC parameters
UBC stands for user beancounters — this is a set of limits and guarantees contolled per VE. You can see the current usage values and limits by examining the output of /proc/user_beancounters file.
Here is the example of /proc/user_beancounters inside VE with ID of 123:
# cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
123: kmemsize 836919 1005343 2752512 2936012 0
lockedpages 0 0 32 32 0
privvmpages 4587 7289 49152 53575 0
shmpages 39 39 8192 8192 0
dummy 0 0 0 0 0
numproc 20 26 65 65 0
physpages 2267 2399 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 2267 2399 6144 2147483647 0
numtcpsock 3 3 80 80 0
numflock 3 4 100 110 0
numpty 1 1 16 16 0
numsiginfo 0 1 256 256 0
tcpsndbuf 0 0 319488 524288 0
tcprcvbuf 0 0 319488 524288 0
othersockbuf 6684 7888 132096 336896 0
dgramrcvbuf 0 8372 132096 132096 0
numothersock 8 10 80 80 0
dcachesize 87672 92168 1048576 1097728 0
numfile 238 306 2048 2048 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 10 16 128 128 0
Note the last column (failcnt) — it shows a number of fails for this counter, i.e. a number of times the parameter hit the limit. Usually what you need to do is to increase the parameter in question. But you need to do it carefully, and here is how.
- Get the current values for the parameter's barrier and limit. For example, we want to increase kmemsize values from the above example. From /proc/user_beancounters we see that kmemsize barrier is 2752512, and its limit is 2936012.
- Increase the values. Say, we want to increase kmemsize by 1.5 times. This is how it can be done using built-in bash arithmetics:
# vzctl set 123 --kmemsize $((2752512*1.5)):$((2936012*1.5)) --save
By using --save flag, we denote we want to both apply the new settings to the running VE, and save them in the configuration file (from which they will be taken during next VE start).
- Check the new configuration. Issue the following command:
# vzcfgvalidate /etc/vz/conf/123.conf
If something is wrong, you need to fix it as suggested by the utility.
Disk quota
To check if your VE ran out of its disk quota, use the following commands (inside a VE):
# df Filesystem 1K-blocks Used Available Use% Mounted on simfs 1048576 327664 720912 32% / # df -i Filesystem Inodes IUsed IFree IUse% Mounted on simfs 200000 18857 181143 10% /
First command shows disk space usage, and second command shows the inodes usage (you can roughly see inodes as a number of files/directories on your system).
If one of the commands give you usage of 100%, that means you hit one of the disk quota limit.
You can increase the limit from the host system (VE0) only. This is how:
- Get the current values for disk quota:
# vzquota stat 123 resource usage softlimit hardlimit grace 1k-blocks 327664 1048576 1153434 inodes 18857 200000 220000 - To increase the disk space quota, use vzctl set --diskspace. For example, we want to increase it by a factor of 2:
vzctl set 123 --diskspace $(( 1048576*2 )):$(( 1153434*2 )) --save
- To increase the disk inodes quota, use vzctl set --diskinodes. For example, we want to increase it by a factor of 3:
vzctl set 123 --diskinodes $(( 200000*3 )):$(( 220000*3 )) --save
|
Note: shell does not support floating-point arithmetics, i.e. you can not use expressions like $(( 220000*1.5 )). To use floating point, try bc instead, something like this: $(echo 220000*1.5 | bc).
|
