Resource shortage

Sometimes you see a strange failures from some programs inside your Virtual Environment. In some cases it means one of the resources controlled by OpenVZ hit the limit.

The first thing to do is to check the contents of /proc/user_beancounters file in your VE. The last column of output is fail counter. Each time a resource hit the limit, fail counter is increasing. So, if you see non-zero values in failcnt column that means something is wrong.

There are two ways to fix the situation: reconfigure (in some cases recompile) the application, or change the resource management settings.

UBC parameters

Here is the example of current UBC values obtained from /proc/user_beancounters file in VE 123:

# cat /proc/user_beancounters
Version: 2.5
       uid  resource           held    maxheld    barrier      limit    failcnt
       123: kmemsize         836919    1005343    2752512    2936012          0
            lockedpages           0          0         32         32          0
            privvmpages        4587       7289      49152      53575          0
            shmpages             39         39       8192       8192          0
            dummy                 0          0          0          0          0
            numproc              20         26         65         65          0
            physpages          2267       2399          0 2147483647          0
            vmguarpages           0          0       6144 2147483647          0
            oomguarpages       2267       2399       6144 2147483647          0
            numtcpsock            3          3         80         80          0
            numflock              3          4        100        110          0
            numpty                1          1         16         16          0
            numsiginfo            0          1        256        256          0
            tcpsndbuf             0          0     319488     524288          0
            tcprcvbuf             0          0     319488     524288          0
            othersockbuf       6684       7888     132096     336896          0
            dgramrcvbuf           0       8372     132096     132096          0
            numothersock          8         10         80         80          0
            dcachesize        87672      92168    1048576    1097728          0
            numfile             238        306       2048       2048          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            numiptent            10         16        128        128          0

You can see if you hit the limits for some UBC parameters by analyzing the last column (named failcnt). It shows a number of fails for this counter, i.e. a number of times a parameter hit the limit. Usually what you need to do is to increase the parameter in question. But you need to do it carefully, and here is how.

1. Get the current values for the parameter's barrier and limit. For example, we want to increase kmemsize values from the example at . From /proc/user_beancounters we see that kmemsize barrier is 2752512, and its limit is 2936012.
2. Increase the values. Say, we want to increase kmemsize by 2 times. This is how it can be done using built-in bash arithmetics:

   # vzctl set 123 --kmemsize $((2752512*2)):$((2936012*2)) --save
   

   By using --save flag, we denote we want to both apply the new settings to the running VE, and save them in the configuration file (from which they will be taken during next VE start).

3. Check the new configuration. Issue the following command:

   # vzcfgvalidate /etc/vz/conf/123.conf
   

   If something is wrong, you need to fix it as suggested by the utility.

For more in-depth explanation of different parameters, their meaning and how to set them properly, see setting UBC parameters.

Disk quota

To check if your VE ran out of its disk quota, use the following commands (inside a VE):

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
simfs                  1048576    327664    720912  32% /
# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
simfs                 200000   18857  181143   10% /

First command shows disk space usage, and second command shows the inodes usage (you can roughly see inodes as a number of files/directories on your system).

If one of the commands give you usage of 100%, that means you hit one of the disk quota limit.

You can increase the limit from the host system (VE0) only. This is how:

1. Get the current values for disk quota:

   # vzquota stat 123
      resource          usage       softlimit      hardlimit    grace
     1k-blocks         327664         1048576        1153434
        inodes          18857          200000         220000
   

2. To increase the disk space quota, use vzctl set --diskspace. For example, we want to increase it by a factor of 2:

   vzctl set 123 --diskspace $(( 1048576*2 )):$(( 1153434*2 )) --save
   

3. To increase the disk inodes quota, use vzctl set --diskinodes. For example, we want to increase it by a factor of 3:

   vzctl set 123 --diskinodes $(( 200000*3 )):$(( 220000*3 )) --save
   

Note: shell does not support floating-point arithmetics, i.e. you can not use expressions like $(( 220000*1.5 )). To use floating point, try bc instead, something like this: $(echo 220000*1.5 | bc).