Difference between revisions of "Resource shortage"

From OpenVZ Virtuozzo Containers Wiki
Jump to: navigation, search
(Cleaning up punctuation and grammer.)
Line 1: Line 1:
Sometimes you see a strange failures from some programs inside your [[container]]. In some cases it means one of the resources controlled by OpenVZ hit the limit.
+
Sometimes you see strange failures from some programs inside your [[container]]. In some cases it means one of the resources controlled by OpenVZ has hit the limit.
  
The first thing to do is to check the contents of /proc/user_beancounters file in your [[container]]. The last column of output is fail counter. Each time a resource hits the limit, the fail counter is increasing. So, if you see non-zero values in failcnt column that means something is wrong.
+
The first thing to do is to check the contents of the /proc/user_beancounters file in your [[container]]. The last column of output is the fail counter. Each time a resource hits the limit, the fail counter is incremented. So, if you see non-zero values in the failcnt column that means something is wrong.
  
 
There are two ways to fix the situation: reconfigure (in some cases recompile) the application, or change the resource management settings.
 
There are two ways to fix the situation: reconfigure (in some cases recompile) the application, or change the resource management settings.
Line 7: Line 7:
 
== [[UBC]] parameters ==
 
== [[UBC]] parameters ==
  
Here is the example of current [[UBC]] values obtained from <code>/proc/user_beancounters</code> file in container 123:
+
Here is an example of current [[UBC]] values obtained from <code>/proc/user_beancounters</code> file in container 123:
 
<pre>
 
<pre>
 
# cat /proc/user_beancounters
 
# cat /proc/user_beancounters
Line 38: Line 38:
 
</pre>
 
</pre>
  
You can see if you hit the limits for some [[UBC]] parameters by analyzing the last column (named <code>failcnt</code>). It shows a number of fails for this counter, i.e. a number of times a parameter hit the limit. Usually what you need to do is to increase the parameter in question. But you need to do it carefully, and here is how.
+
You can see if you hit the limit for some [[UBC]] parameters by analyzing the last column (named <code>failcnt</code>). It shows a number of failures for this counter, i.e. a number of times a parameter hit the limit. Usually what you need to do is to increase the parameter in question. But you need to do it carefully, and here is how.
  
 
<ol>
 
<ol>
Line 44: Line 44:
 
</li>
 
</li>
  
<li>Increase the values. Say, we want to increase <code>kmemsize</code> by 2 times. This is how it can be done using built-in bash arithmetics:
+
<li>Increase the values. Say, we want to double <code>kmemsize</code>. This is how it can be done using built-in bash arithmetics:
 
<pre>
 
<pre>
 
# vzctl set 123 --kmemsize $((2752512*2)):$((2936012*2)) --save
 
# vzctl set 123 --kmemsize $((2752512*2)):$((2936012*2)) --save
 
</pre>
 
</pre>
By using <code>--save</code> flag, we denote we want to both apply the new settings to the running container, and save them in the configuration file (from which they will be taken during next container start).
+
By using the <code>--save</code> flag, we indicate we want to apply the new setting to the running container and save it in the configuration file (from which the settings will be taken during next container start).
 
</li>
 
</li>
  
Line 62: Line 62:
  
 
== Disk quota ==
 
== Disk quota ==
To check if your [[container]] ran out of its disk quota, use the following commands (inside a container):
+
To check if your [[container]] exceeded its disk quota, use the following commands (inside a container):
 
<pre>
 
<pre>
 
# df
 
# df
Line 72: Line 72:
 
</pre>
 
</pre>
  
First command shows disk space usage, and second command shows the inodes usage (you can roughly see inodes as a number of files/directories on your system).
+
The first command shows disk space usage and the second command shows the inodes usage (you can roughly use the inodes count as a number of files/directories on your system).
  
If one of the commands give you usage of 100%, that means you hit one of the disk quota limit.  
+
If one of the commands shows a usage of 100% you have exceeded one of the disk quota limits.  
  
 
You can increase the limit from the host system ([[CT0]]) only. This is how:
 
You can increase the limit from the host system ([[CT0]]) only. This is how:
Line 99: Line 99:
 
</ol>
 
</ol>
  
{{Note|shell does not support floating-point arithmetics, i.e. you can not use expressions like <code>$((&nbsp;220000*1.5&nbsp;))</code>. To use floating point, try <code>bc</code> instead, something like this: <code><nowiki>$(echo&nbsp;220000*1.5&nbsp;|&nbsp;bc)</nowiki></code>.}}
+
{{Note|shell does not support floating-point arithmetic, i.e. you can not use expressions like <code>$((&nbsp;220000*1.5&nbsp;))</code>. To use floating point, try <code>bc</code> instead, something like this: <code><nowiki>$(echo&nbsp;220000*1.5&nbsp;|&nbsp;bc)</nowiki></code>.}}
  
 
== CPU ==
 
== CPU ==

Revision as of 11:32, 22 May 2008

Sometimes you see strange failures from some programs inside your container. In some cases it means one of the resources controlled by OpenVZ has hit the limit.

The first thing to do is to check the contents of the /proc/user_beancounters file in your container. The last column of output is the fail counter. Each time a resource hits the limit, the fail counter is incremented. So, if you see non-zero values in the failcnt column that means something is wrong.

There are two ways to fix the situation: reconfigure (in some cases recompile) the application, or change the resource management settings.

UBC parameters

Here is an example of current UBC values obtained from /proc/user_beancounters file in container 123:

# cat /proc/user_beancounters
Version: 2.5
       uid  resource           held    maxheld    barrier      limit    failcnt
       123: kmemsize         836919    1005343    2752512    2936012          0
            lockedpages           0          0         32         32          0
            privvmpages        4587       7289      49152      53575          0
            shmpages             39         39       8192       8192          0
            dummy                 0          0          0          0          0
            numproc              20         26         65         65          0
            physpages          2267       2399          0 2147483647          0
            vmguarpages           0          0       6144 2147483647          0
            oomguarpages       2267       2399       6144 2147483647          0
            numtcpsock            3          3         80         80          0
            numflock              3          4        100        110          0
            numpty                1          1         16         16          0
            numsiginfo            0          1        256        256          0
            tcpsndbuf             0          0     319488     524288          0
            tcprcvbuf             0          0     319488     524288          0
            othersockbuf       6684       7888     132096     336896          0
            dgramrcvbuf           0       8372     132096     132096          0
            numothersock          8         10         80         80          0
            dcachesize        87672      92168    1048576    1097728          0
            numfile             238        306       2048       2048          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            numiptent            10         16        128        128          0

You can see if you hit the limit for some UBC parameters by analyzing the last column (named failcnt). It shows a number of failures for this counter, i.e. a number of times a parameter hit the limit. Usually what you need to do is to increase the parameter in question. But you need to do it carefully, and here is how.

  1. Get the current values for the parameter's barrier and limit. For example, we want to increase kmemsize values from the example at . From /proc/user_beancounters we see that kmemsize barrier is 2752512, and its limit is 2936012.
  2. Increase the values. Say, we want to double kmemsize. This is how it can be done using built-in bash arithmetics:
    # vzctl set 123 --kmemsize $((2752512*2)):$((2936012*2)) --save
    

    By using the --save flag, we indicate we want to apply the new setting to the running container and save it in the configuration file (from which the settings will be taken during next container start).

  3. Check the new configuration. Issue the following command:
    # vzcfgvalidate /etc/vz/conf/123.conf
    

    If something is wrong, you need to fix it as suggested by the utility.

For more in-depth explanation of different parameters, their meaning and how to set them properly, see setting UBC parameters.

Disk quota

To check if your container exceeded its disk quota, use the following commands (inside a container):

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
simfs                  1048576    327664    720912  32% /
# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
simfs                 200000   18857  181143   10% /

The first command shows disk space usage and the second command shows the inodes usage (you can roughly use the inodes count as a number of files/directories on your system).

If one of the commands shows a usage of 100% you have exceeded one of the disk quota limits.

You can increase the limit from the host system (CT0) only. This is how:

  1. Get the current values for disk quota:
    # vzquota stat 123
       resource          usage       softlimit      hardlimit    grace
      1k-blocks         327664         1048576        1153434
         inodes          18857          200000         220000
    
  2. To increase the disk space quota, use vzctl set --diskspace. For example, we want to increase it by a factor of 2:
    vzctl set 123 --diskspace $(( 1048576*2 )):$(( 1153434*2 )) --save
    
  3. To increase the disk inodes quota, use vzctl set --diskinodes. For example, we want to increase it by a factor of 3:
    vzctl set 123 --diskinodes $(( 200000*3 )):$(( 220000*3 )) --save
    
Yellowpin.svg Note: shell does not support floating-point arithmetic, i.e. you can not use expressions like $(( 220000*1.5 )). To use floating point, try bc instead, something like this: $(echo 220000*1.5 | bc).

CPU

There are two parameters controlling fair CPU scheduler in OpenVZ: cpuunits and cpulimit.