Resource shortage

Sometimes you see a strange fails from some programs inside your Virtual Environment. In some cases it means one of the resources controlled by OpenVZ hit the limit.

The first thing to do is to check the contents of /proc/user_beancounters file in your VE. The last column of output is fail counter. Each time a resource hit the limit, fail counter is increasing. So, if you see non-zero values in failcnt column that means something is wrong.

There are two ways to fix the situation: reconfugure (in some cases recompile) the application, or change the resource management settings.

UBC parameters

UBC stands for user beancounters — this is a set of limits and guarantees contolled per VE. You can see the current usage values and limits by examining /proc/user_beancounters file.

Here is the example of /proc/user_beancounters inside VE with ID of 123:

# cat /proc/user_beancounters
Version: 2.5
       uid  resource           held    maxheld    barrier      limit    failcnt
       123: kmemsize         836919    1005343    2752512    2936012          0
            lockedpages           0          0         32         32          0
            privvmpages        4587       7289      49152      53575          0
            shmpages             39         39       8192       8192          0
            dummy                 0          0          0          0          0
            numproc              20         26         65         65          0
            physpages          2267       2399          0 2147483647          0
            vmguarpages           0          0       6144 2147483647          0
            oomguarpages       2267       2399       6144 2147483647          0
            numtcpsock            3          3         80         80          0
            numflock              3          4        100        110          0
            numpty                1          1         16         16          0
            numsiginfo            0          1        256        256          0
            tcpsndbuf             0          0     319488     524288          0
            tcprcvbuf             0          0     319488     524288          0
            othersockbuf       6684       7888     132096     336896          0
            dgramrcvbuf           0       8372     132096     132096          0
            numothersock          8         10         80         80          0
            dcachesize        87672      92168    1048576    1097728          0
            numfile             238        306       2048       2048          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            numiptent            10         16        128        128          0

Note the last column (failcnt) — it shows a number of fails for this counter, i.e. a number of times the parameter hit the limit. Usually what you need to do is to increase the parameter in question. But you need to do it carefully, and here is how.

1. Get the current values for the parameter's barrier and limit. For example, we want to increase kmemsize values from the above example. From /proc/user_beancounters we see that kmemsize barrier is 2752512, and its limit is 2936012.
2. Increase the values. Say, we want to increase kmemsize by 1.5 times. This is how it can be done using built-in bash arithmetics:

   # vzctl set 123 --kmemsize $((2752512*1.5)):$((2936012*1.5)) --save
   

   By using --save flag, we denote we want to both apply the new settings to the running VE, and save them in the configuration file (from which they will be taken during next VE start).

3. Check the new configuration. Issue the following command:

   # vzcfgvalidate /etc/vz/123.conf
   

   If something is wrong, you need to fix it as suggested by the utility.

FIXME more