OpenVZ Virtuozzo Containers Wiki - User contributions [en]

Proc/user beancounters

2011-08-15T23:28:21Z

Nathanhaigh: /* Reading */ fixed a typo

{{UBC toc}}
{{DISPLAYTITLE:/proc/user_beancounters}}
The proc filesystem entry showing resource control information is <code>/proc/user_beancounters</code> file. An example content of <code>/proc/user_beancounters</code> is shown below.

{{Note|This article describes an old interface. This is kept for compatibility and [[BC proc entries|new one]] will be used since <code>028test008</code> kernel}}

== Example ==
<pre>
# cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
123: kmemsize 836919 1005343 2752512 2936012 0
lockedpages 0 0 32 32 0
privvmpages 4587 7289 49152 53575 0
shmpages 39 39 8192 8192 0
dummy 0 0 0 0 0
numproc 20 26 65 65 0
physpages 2267 2399 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 2267 2399 6144 2147483647 0
numtcpsock 3 3 80 80 0
numflock 3 4 100 110 0
numpty 1 1 16 16 0
numsiginfo 0 1 256 256 0
tcpsndbuf 0 0 319488 524288 0
tcprcvbuf 0 0 319488 524288 0
othersockbuf 6684 7888 132096 336896 0
dgramrcvbuf 0 8372 132096 132096 0
numothersock 8 10 80 80 0
dcachesize 87672 92168 1048576 1097728 0
numfile 238 306 2048 2048 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 10 16 128 128 0
</pre>

== Fields ==

The field '''<code>uid</code>''' is a numeric identifier of the container<ref>The name “uid” is related to the fact that OpenVZ Resource Management can be used without virtualization and containers.</ref>.

For accountable parameters, the field '''<code>held</code>''' shows the current counter for the container (resource “usage”), and the field '''<code>maxheld</code>''' shows the counter's maximum for the last accounting period. The accounting period is usually the lifetime of the container<ref>The lifetime of a container is the period from the start of the container till the time all processes in the container exited and all resources used by these processes are freed. Usually, the lifetime is just the time between the start and the stop of the container.</ref>.

The field '''<code>failcnt</code>''' shows the number of refused “resource allocations” for the whole lifetime of the process group.

The '''<code>barrier</code>''' and '''<code>limit</code>''' fields are resource control settings. For some
parameters, only one of them may be used, for some parameters — both.
These fields may specify limits or guarantees, and the exact meaning of
them is parameter-specific. Description of each parameter in [[UBC parameters]]
contains information about the difference between the <code>barrier</code> and
the <code>limit</code> for the parameter.

== Units ==
{{Main|UBC parameter units}}

== Settings ==

General notes about parameters and their barrier and limit settings
are provided in [[UBC parameters]]. Parameters not having limit setting have
[[LONG_MAX]] in the corresponding field.

== Reading ==

{{Note|It's easier to use [[BC proc entries|the new <code>/proc/bc</code> interface]], i.e. <code>cat /proc/bc/$CTID/resources</code>.}}

If you want to print UBC values for a specific container, the following construct can be helpful

<nowiki>egrep -A23 "${CTID}:" /proc/user_beancounters</nowiki>

Or with headers, though slightly harder to remember:

<nowiki>sed -nr "1,2p;/${CTID}:/,+23p" /proc/user_beancounters</nowiki>

Here <code>${CTID}</code> is the ID of the container you are interested in.

In case you want to print UBC values for a few containers at once, set CTID to something like <code>(101|102|103)</code>.

== Notes ==
<references/>

Proc/user beancounters

2011-08-15T23:24:45Z

Nathanhaigh: /* Units */ Link directly to the main article for ease of maintenance

{{UBC toc}}
{{DISPLAYTITLE:/proc/user_beancounters}}
The proc filesystem entry showing resource control information is <code>/proc/user_beancounters</code> file. An example content of <code>/proc/user_beancounters</code> is shown below.

{{Note|This article describes an old interface. This is kept for compatibility and [[BC proc entries|new one]] will be used since <code>028test008</code> kernel}}

== Example ==
<pre>
# cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
123: kmemsize 836919 1005343 2752512 2936012 0
lockedpages 0 0 32 32 0
privvmpages 4587 7289 49152 53575 0
shmpages 39 39 8192 8192 0
dummy 0 0 0 0 0
numproc 20 26 65 65 0
physpages 2267 2399 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 2267 2399 6144 2147483647 0
numtcpsock 3 3 80 80 0
numflock 3 4 100 110 0
numpty 1 1 16 16 0
numsiginfo 0 1 256 256 0
tcpsndbuf 0 0 319488 524288 0
tcprcvbuf 0 0 319488 524288 0
othersockbuf 6684 7888 132096 336896 0
dgramrcvbuf 0 8372 132096 132096 0
numothersock 8 10 80 80 0
dcachesize 87672 92168 1048576 1097728 0
numfile 238 306 2048 2048 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 10 16 128 128 0
</pre>

== Fields ==

The field '''<code>uid</code>''' is a numeric identifier of the container<ref>The name “uid” is related to the fact that OpenVZ Resource Management can be used without virtualization and containers.</ref>.

For accountable parameters, the field '''<code>held</code>''' shows the current counter for the container (resource “usage”), and the field '''<code>maxheld</code>''' shows the counter's maximum for the last accounting period. The accounting period is usually the lifetime of the container<ref>The lifetime of a container is the period from the start of the container till the time all processes in the container exited and all resources used by these processes are freed. Usually, the lifetime is just the time between the start and the stop of the container.</ref>.

The field '''<code>failcnt</code>''' shows the number of refused “resource allocations” for the whole lifetime of the process group.

The '''<code>barrier</code>''' and '''<code>limit</code>''' fields are resource control settings. For some
parameters, only one of them may be used, for some parameters — both.
These fields may specify limits or guarantees, and the exact meaning of
them is parameter-specific. Description of each parameter in [[UBC parameters]]
contains information about the difference between the <code>barrier</code> and
the <code>limit</code> for the parameter.

== Units ==
{{Main|UBC parameter units}}

== Settings ==

General notes about parameters and their barrier and limit settings
are provided in [[UBC parameters]]. Parameters not having limit setting have
[[LONG_MAX]] in the corresponding field.

== Reading ==

{{Note|It's easier to use [[BC proc entries|the new <code>/proc/bc</code> interface]], i.e. <code>cat /proc/bc/$CTID/resources</code>.}}

If you want to print UBC values for a specific container, the following construct can be helpful

<nowiki>egrep -A23 "${CTID}:" /proc/user_beancounters</nowiki>

Or with headers, though slightly harder to remember:

<nowiki>sed -nr "1,2p;/${CTID}:/,+23p' /proc/user_beancounters</nowiki>

Here <code>${CTID}</code> is the ID of the container you are interested in.

In case you want to print UBC values for a few containers at once, set CTID to something like <code>(101|102|103)</code>.

== Notes ==
<references/>

UBC auxiliary parameters

2011-08-15T23:13:19Z

Nathanhaigh: /* Units */

{{UBC toc}}

Configuration of primary and secondary resource control parameters is
important for security and stability of the whole system. Auxiliary
parameters differ much from primary and secondary parameters in this respect.

The primary functions of auxiliary parameters are the following.
<ul>
<li>These parameters improve application's handling of errors and resource
consumption limitations.

Without these auxiliary parameters, possible bugs in applications (such
as forgetting to unlock locked files or forgetting to collect signals) will
cause slowdown and, after some time, killing of the applications because
of memory exhaustion. In presence of these parameters, applications
will notice the problem (because, for example, attempts to create new
file locks start to fail) and show an appropriate message helping to
debug the problem.

Another example. Each object such as opened file or established network
connection consume certain resources. When the container
is close to exhaustion of the resources allowed to him, it is
usually better to refuse creation of new object than to allow it but deny
memory allocation or terminate (in case of complete exhaustion of the
resources) an already running application.
</li>

<li>
These parameters improve fault isolation between applications in the
same container. Failures or misbehavior of one application
inside a container is more likely to cause hitting a
limit on some auxiliary parameter and normal termination of this mis-
behaving application, rather than abnormal termination of some other
long-running application inside the same container.
</li>

<li>
These parameters may be used to impose some administrative limits
on the container (for example, to not allow the user to run
database servers by limiting the amount of [[shmpages]], or limiting the
number of simultaneous shell sessions through [[numpty]]).
</li>
</ul>

So, auxiliary parameters play a role similar to limits imposed by
<code>setrlimit(2)</code> interface and limits configurable by
<code>sysctl(8)</code> in standard
Linux installations.

Because of this helper role in resource control, system management software
may show auxiliary parameters only in advanced mode for experienced
administrators and hide them in “basic” management modes.

== UBC Auxiliary Parameters ==

=== lockedpages ===
Process pages not allowed to be swapped out (pages locked by <code>mlock(2)</code>).

The size of these pages is also accounted into <code>[[kmemsize]]</code>.
The <code>barrier</code> may be set equal to the <code>limit</code> or may allow
some gap between the <code>barrier</code> and the <code>limit</code>, depending
on the nature of applications using memory locking features.

Note that typical server applications like Web, FTP, mail servers
do not use memory locking features.

The configuration of this parameter doesn't affect security and
stability of the whole system or isolation between containers.
Its configuration affects functionality and resource shortage reaction
of applications in the given container only.

=== shmpages ===
The total size of shared memory (IPC, shared anonymous mappings and
<code>tmpfs</code> objects).

These pages are also accounted into <code>[[privvmpages]]</code>.

The <code>barrier</code> should be set equal to the <code>limit</code>.
The configuration of this parameter doesn't affect security and
stability of the whole system or isolation between containers.
Its configuration affects functionality and resource shortage reaction
of applications in the given container only.

=== physpages ===
Total number of RAM pages used by processes in a container.

For memory pages used by several different containers (mappings of
shared libraries, for example), only a fraction of a page is charged to each
container.
The sum of the <code>physpages</code> usage for all containers
corresponds to the total number of pages used in the system by all
containers.

For [[VSwap]]-enabled kernels, the <code>barrier</code> should be set to 0,
and the <code>limit</code> limits the total size of RAM used by a container.

For older kernels, <code>physpages</code> is an accounting-only parameter.
The <code>barrier</code> should be set to <code>0</code> and the
<code>limit</code> to 'unlimited' ([[LONG_MAX]]).

=== numfile ===
Number of open files.

The <code>barrier</code> should be set equal to the <code>limit</code>.
The configuration of this parameter doesn't affect security and
stability of the whole system or isolation between containers.
Its configuration affects functionality and resource shortage reaction
of applications in the given container only.

Note: actually currently adjusting the <code>barrier</code> will change the kernel behaviour on "pre-charging" the numfile resource. If you change one you will most likely not notice any changes in container behaviour at all. This ability was added for researching purposes purely.

=== numflock ===
Number of file locks.

The configuration of this parameter should have a
gap between the <code>barrier</code> and the <code>limit</code>, as illustrated in
[[UBC configuration examples]].

Very high limits on <code>numflock</code> parameters and the big number
of file locks in the system may cause certain slowdown of
the whole system (but not fatal).
So, the limits on this parameter should be reasonable, depending
on the real requirements of the applications.

=== numpty ===
Number of pseudo-terminals.

This parameter is usually used to limit the number of simultaneous shell
sessions.
The <code>barrier</code> should be set equal to the <code>limit</code>.
The configuration of this parameter doesn't affect security and
stability of the whole system or isolation between containers.
Its configuration affects functionality and resource shortage reaction
of applications in the given container only.
However, in OpenVZ systems, the actual number of pseudo-terminals allowed
for one container is limited to <code>256</code>.

=== numsiginfo ===
Number of <code>siginfo</code> structures.

The size of the structure is also accounted into <code>[[kmemsize]]</code>.
The default installations of stand-alone Linux systems limit this number
to <code>1024</code> for the whole system.
In OpenVZ installations, <code>numsiginfo</code> limit applies to each
container individually.

The <code>barrier</code> should be set equal to the <code>limit</code>.
Very high settings of the <code>limit</code> of this parameter may reduce
responsiveness of the system.
It is unlikely that any container will need the limit greater than
the Linux default — <code>1024</code>.

=== dcachesize ===
The total size of <code>dentry</code> and <code>inode</code> structures locked in memory.

<code>Dcachesize</code> parameter controls filesystem-related caches, such as
directory entry (<code>dentry</code>) and inode caches.
The value accounted into <code>dcachesize</code> is also included into
<code>[[kmemsize]]</code>.

<code>Dcachesize</code> exists as a separate parameter to impose a limit causing
file operations to sense memory shortage and return an error to applications,
protecting from memory shortages during critical operations that shouldn't
fail.

The configuration of this parameter should have a
gap between the <code>barrier</code> and the <code>limit</code>, as illustrated in
[[UBC configuration examples]].
The configuration of this parameter doesn't affect security and
stability of the whole system or isolation between containers.
Its configuration affects functionality and resource shortage reaction
of applications in the given container only.

=== numiptent ===
The number of NETFILTER (IP packet filtering) entries.

The <code>barrier</code> should be set equal to the <code>limit</code>.
There is a restriction on the total number of <code>numiptent</code>.
It depends on the amount of other allocations in so called “vmalloc”
memory area and constitutes about <code>250000</code> entries.
Violation of this restriction may cause failures of operations with
IP packet filter tables (execution of <code>iptables(8)</code>)
in any container or the host system,
or failures of container starts.

Also, large <code>numiptent</code> cause considerable slowdown of processing
of network packets. It is not recommended to allow containers
to create more than 200–300 <code>numiptent</code>.

=== swappages ===

The amount of swap space to show in container.

{{Note|this parameter is only available in RHEL5-based kernel since version 028stab060.2, in 2.6.27 since kiprensky.}}

The configuration of this parameter doesn't affect security and
stability of the whole system or isolation between containers.
Its configuration only affects the way OpenVZ kernel reports about
available swap in a container. This is needed for some applications
which refuse to run inside a container unless the kernel
report that no less than some specific amount of swap is available.

If <code>limit</code> is set, its value is reported as the amount
of total swap space in a container.

If the <code>limit</code> is set to [[LONG_MAX]] (which is the
in-kernel default for this parameter), all the swap space values
parameters (total, used, free) are reported as 0.

The value of <code>barrier</code> for this beancounter is ignored.

The value of <code>held</code> shows how much swap space
is currently being used for this container.

== Units ==
{{Main|UBC parameter units}}

UBC secondary parameters

2011-08-15T23:13:03Z

Nathanhaigh: /* Units */

{{UBC toc}}

'''Secondary (dependant) UBC parameters''' are directly connected
to the [[UBC primary parameters|primary ones]] and can't be configured arbitrarily.

== UBC Secondary Parameters ==
=== kmemsize ===
Size of unswappable memory in bytes, allocated by the operating system kernel.

It includes all the kernel internal data structures associated with the
container's processes, except the network buffers discussed below.
These data structures reside in the first gigabyte of the computer's RAM,
so called [[UBC systemwide configuration#“Low memory”|“low memory”]].

This parameter is related to the number of processes ([[numproc]]).
Each process consumes certain amount of kernel memory —
24 kilobytes at minimum, 30–60 KB typically.
Very large processes may consume much more than that.

It is important to have a certain safety gap between the <code>barrier</code> and
the <code>limit</code> of the <code>kmemsize</code> parameter
(for example, 10%, as in [[UBC configuration examples]]). Equal <code>barrier</code> and <code>limit</code> of
the <code>kmemsize</code> parameter may lead to the situation where the kernel will
need to kill container's applications to keep the <code>kmemsize</code>
usage under the limit.

<code>Kmemsize</code> limits can't be set arbitrarily high.
The total amount of <code>kmemsize</code> consumable by all containers
in the system plus the socket buffer space (see below) is limited by the
hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

=== tcpsndbuf ===
The total size of buffers used to send data over TCP network connections.
These socket buffers reside in [[UBC systemwide configuration#“Low memory”|“low memory”]].

<code>Tcpsndbuf</code> parameter depends on number of TCP
sockets ([[numtcpsock]]) and should allow for some minimal amount of
socket buffer memory for each socket, as discussed in [[UBC consistency check]]:

<math>tcpsndbuf_{lim} - tcpsndbuf_{bar} \ge 2.5KB \cdot numtcpsock \rm.</math>

If this restriction is not satisfied, some network connections
may silently stall, being unable to transmit data.

Setting high values for <code>tcpsndbuf</code> parameter
may, but doesn't necessarily, increase performance of network communications.
Note that, unlike most other parameters, hitting <code>tcpsndbuf</code>
limits and failed socket buffer allocations
do not have strong negative effect on the applications, but just reduce
performance of network communications.

If you use rtorrent in a container, a low value for <code>tcpsndbuf</code> may cause rtorrent to take unusual amount of cpu. In this case, you must put a higher value. Also watch the number of failcnt in /proc/user_beancounters.

<code>Tcpsndbuf</code> limits can't be set arbitrarily high.
The total amount of <code>tcpsndbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers is limited
by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

=== tcprcvbuf ===
The total size of buffers used to temporary store the data coming from TCP network connections.
These socket buffers also reside in [[UBC systemwide configuration#“Low memory”|“low memory”]].

<code>Tcprcvbuf</code> parameter depends on number of TCP
sockets ([[numtcpsock]]) and should allow for some minimal amount of
socket buffer memory for each socket, as discussed in [[UBC consistency check]]:

<math>tcprcvbuf_{lim} - tcprcvbuf_{bar} \ge 2.5KB \cdot numtcpsock \rm.</math>

If this restriction is not satisfied, some network connections
may stall, being unable to receive data, and will be terminated
after a couple of minutes.

Similarly to <code>tcpsndbuf</code>, setting high values for <code>tcprcvbuf</code>
parameter may, but doesn't necessarily, increase performance of network
communications.
Hitting <code>tcprcvbuf</code> limits and failed socket buffer allocations
do not have strong negative effect on the applications, but just reduce
performance of network communications.
However, staying above the <code>barrier</code> of <code>tcprcvbuf</code> parameter
for a long time is less harmless than for <code>tcpsndbuf</code>.
Long periods of exceeding the <code>barrier</code> may cause termination
of some connections.

<code>Tcprcvbuf</code> limits can't be set arbitrarily high.
The total amount of <code>tcprcvbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers is limited
by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

=== othersockbuf ===
The total size of buffers used by local (UNIX-domain) connections between
processes inside the system (such as connections to a local database server)
and send buffers of UDP and other datagram protocols.

<code>Othersockbuf</code> parameter depends on number of non-TCP sockets (<code>[[numothersock]]</code>).

<code>Othersockbuf</code> configuration should satisfy

<math>othersockbuf_{lim} - othersockbuf_{bar} \ge 2.5KB \cdot numothersock.</math>

Increased limit for <code>othersockbuf</code> is necessary for high performance of
communications through local (UNIX-domain) sockets.
However, similarly to <code>tcpsndbuf</code>, hitting <code>othersockbuf</code> affects
the communication performance only and does not affect the functionality.

<code>Othersockbuf</code> limits can't be set arbitrarily high.
The total amount of <code>othersockbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers
is limited by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

=== dgramrcvbuf ===
The total size of buffers used to temporary store the incoming packets of UDP and
other datagram protocols.

<code>Dgramrcvbuf</code> parameters depend on number of
non-TCP sockets (<code>[[numothersock]]</code>).

<code>Dgramrcvbuf</code> limits usually don't need to be high.
Only if the containers needs to send and receive very large
datagrams, the <code>barrier</code>s for both <code>othersockbuf</code> and
<code>dgramrcvbuf</code> parameters should be raised.

Hitting <code>dgramrcvbuf</code> means that some datagrams are dropped, which may
or may not be important for application functionality.
UDP is a protocol with not guaranteed delivery, so even if the buffers
permit, the datagrams may be as well dropped later on any stage of the
processing, and applications should be prepared for it.

Unlike other socket buffer parameters, for <code>dgramrcvbuf</code>
the <code>barrier</code> should be set to the <code>limit</code>.

<code>Dgramrcvbuf</code> limits can't be set arbitrarily high.
The total amount of <code>dgramrcvbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers
is limited by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

=== oomguarpages ===
The guaranteed amount of memory for the case the memory is “over-booked”
(out-of-memory kill guarantee).

<code>Oomguarpages</code> parameter is related to <code>[[vmguarpages]]</code>.
If applications start to consume more memory than the computer has,
the system faces an out-of-memory condition.
In this case the operating system will start to kill container's
processes to free some memory and prevent the total death
of the system. Although it happens very rarely in typical system loads,
killing processes in out-of-memory situations is a normal reaction of the
system, and it is built into every Linux kernel<ref>The possible reasons of out-of-memory situations are the excess of total <code>[[vmguarpages]]</code> guarantees the available physical resources or high memory consumption by system processes. Also, the kernel might allow some containers to allocate memory above their <code>[[vmguarpages]]</code> guarantees when the system had a lot of free memory, and later, when other containers claim their guarantees, the system will experience the memory shortage.</ref>.

<code>[[Oomguarpages]]</code> parameter accounts the total amount of
memory and swap space used by the processes of a particular
container.
The <code>barrier</code> of the <code>oomguarpages</code> parameter is the out-of-memory
guarantee.

If the current usage of memory and swap space
(the value of <code>oomguarpages</code>) plus the amount of used kernel memory
(<code>[[kmemsize]]</code>) and socket buffers is below the <code>barrier</code>,
processes in this container are guaranteed not to be killed in
out-of-memory situations.
If the system is in out-of-memory situation and there are several
containers with <code>oomguarpages</code> excess, applications in the
container with the biggest excess will be killed first.
The <code>failcnt</code> counter of <code>oomguarpages</code> parameter
increases when a process in this container is killed because
of out-of-memory situation.

If the administrator needs to make sure that some application won't be
forcedly killed regardless of the application's behavior,
setting the <code>[[privvmpages]]</code> limit to a value not greater than the
<code>oomguarpages</code> guarantee significantly reduce the likelihood of
the application being killed,
and setting it to a half of the <code>oomguarpages</code> guarantee completely
prevents it.
Such configurations are not popular because they significantly reduce
the utilization of the hardware.

The meaning of the <code>limit</code> for the <code>oomguarpages</code> parameter is
unspecified in the current version.

The total out-of-memory guarantees given to the containers should
not exceed the physical capacity of the computer, as discussed in [[UBC systemwide configuration#Memory and swap space]].
If guarantees are given for more than the system has, in out-of-memory
situations applications in containers with guaranteed level of
service and system daemons may be killed.

=== privvmpages ===
Memory allocation limit in [[Memory_page|pages]] (which are typically 4096 bytes in size).

<code>Privvmpages</code> parameter
allows controlling the amount of memory allocated by applications.

The <code>barrier</code> and the <code>limit</code> of <code>privvmpages</code> parameter
control the upper boundary of the total size of allocated memory.
Note that this upper boundary doesn't guarantee that the container
will be able to allocate that much memory, neither does it guarantee that
other containers will be able to allocate their fair share of
memory.
The primary mechanism to control memory allocation is the <code>[[vmguarpages]]</code>
guarantee.

<code>Privvmpages</code> parameter accounts allocated (but, possibly,
not used yet) memory.
The accounted value is an estimation how much memory will be really consumed
when the container's applications start to use the allocated
memory.
Consumed memory is accounted into <code>[[oomguarpages]]</code> parameter.

Since the memory accounted into <code>privvmpages</code> may not be actually used,
the sum of current <code>privvmpages</code> values for all containers
may exceed the RAM and swap size of the computer.

There should be a safety gap between the <code>barrier</code> and the <code>limit</code>
for <code>privvmpages</code> parameter to reduce the number of memory allocation
failures that the application is unable to handle.
This gap will be used for “high-priority” memory allocations, such
as process stack expansion.
Normal priority allocations will fail when the <code>barrier</code> of
<code>privvmpages</code> is reached.

Total <code>privvmpages</code> should correlate with the physical resources of the
computer.
Also, it is important not to allow any container to allocate a
significant portion of all system RAM to avoid serious service level
degradation for other containers.
Both these configuration requirements are discussed in [[UBC systemwide configuration#Allocated memory]].

There's also an article describing how [[user pages accounting]] works.

== Units ==
{{Main|UBC parameter units}}

== System-wide limits ==
All secondary parameters are related to memory.
Total limits on memory-related parameters must not exceed the physical
resources of the computer.
The restrictions on the configuration of memory-related parameters are listed
in [[UBC systemwide configuration]].
Those restrictions are very important, because their violation may
allow any container cause the whole system to hang.

== Notes ==
<references/>

UBC primary parameters

2011-08-15T23:12:31Z

Nathanhaigh: /* Units */

{{UBC toc}}

The most important parameters determining the resources available to
container are explained below. The meaning of the parameters
is illustrated assuming that the container runs some network
server applications.

== UBC Primary Parameters ==

=== numproc ===
Maximum number of processes and kernel-level threads allowed for this container.

Many server applications (like Apache Web server, FTP and mail servers)
spawn a process to handle each client, so the limit on number of processes
defines how many clients the application will be able to handle in parallel.
However, the number of processes doesn't limit how “heavy” the application is
and whether the server will be able to serve heavy requests from clients.

Configuring resource control system, it is important to estimate both
the maximum number of processes and the average number of processes
(referred to as <code>avnumproc</code> later). Other (dependent) resource
control parameters may depend both on the limit and the average number (see [[UBC consistency check]]).

The <code>barrier</code> of <code>numproc</code> doesn't provide additional control and should be set equal to the <code>limit</code>.

There is a restriction on the total number of processes in the system.
More than about 16000 processes start to cause poor responsiveness of
the system, worsening when the number grows. Total number of processes
exceeding 32000 is very likely to cause hang of the system.

Note that in practice the number of processes is usually less. Each
process consumes some memory, and the available memory and the
"low memory" (see [[UBC systemwide configuration#“Low memory”|“Low memory”]]) limit the number of processes to lower
values. With typical processes, it is normal to be able to run only up
to 8000 processes in a system.

=== numtcpsock ===
Maximum number of TCP sockets.

This parameter limits the
number of TCP connections and, thus, the number of clients the server
application can handle in parallel.

The <code>barrier</code> of this parameter should be set equal to the <code>limit</code>.

If each container has it's own set of IP addresses (which is
the only way a OpenVZ system can be configured), there are no direct
limits on the total number of TCP sockets in the system. The number
of sockets needs to be controlled because each socket needs certain
amount of memory for receive and transmit buffers (see descriptions
of <code>[[tcpsndbuf]]</code> and <code>[[tcprcvbuf]]</code>), and
the memory is a limited resource.

=== numothersock ===
Maximum number of non-TCP sockets (local sockets, UDP and other types of sockets).

Local (UNIX-domain) sockets are used for communications inside the
system. Multi-tier applications (for example, a Web application with a
database server as a back-end) may need one or multiple local sockets
to serve each client. Straightforward applications (for example, most
mail servers, with the exception of Postfix) do not use local sockets.

UDP sockets are used for Domain Name Service (DNS) queries, but
the number of such sockets opened simultaneously is low. UDP and
other sockets may also be used in some very special applications
(SNMP agents and others).

The <code>barrier</code> of this parameter should be set equal to the <code>limit</code>.
The number of local sockets in a system is not limited. The number of
UDP sockets in a system, similarly to TCP sockets, is not limited in
OpenVZ systems.

Similarly to <code>numtcpsock</code> parameter discussed above, the number of
non-TCP sockets needs to be controlled because each socket needs certain
amount of memory for its buffers, and the memory is a limited
resource.

=== vmguarpages ===
Memory allocation guarantee.

This parameter controls how much memory is available to the Virtual
Environment (i.e. how much memory its applications can allocate by
<code>malloc(3)</code> or other standard Linux memory allocation mechanisms).
The more clients are served or the more “heavy” the application is, the
more memory it needs.

The amount of memory that container's applications are
guaranteed to be able to allocate is specified as the <code>barrier</code> of
<code>vmguarpages</code> parameter. The current amount of allocated memory space
is accounted into <code>privvmpages</code> parameter, and <code>vmguarpages</code>
parameter does not have its own accounting. The <code>barrier</code> and the <code>limit</code> of
<code>privvmpages</code> parameter impose an upper limit on the memory allocations
(see [[privvmpages]]). The meaning of the <code>limit</code> for the
<code>vmguarpages</code> parameter is unspecified in the current version and should be set
to the maximal allowed value ([[LONG_MAX]]).

If the current amount of allocated memory space does not exceed the
guaranteed amount (the <code>barrier</code> of <code>vmguarpages</code>),
memory allocations of container's applications always succeed.
If the current amount of allocated memory space exceeds the guarantee but below
the <code>barrier</code> of <code>privvmpages</code>, allocations may or may not succeed,
depending on the total amount of available memory in the system.

Starting from the <code>barrier</code> of <code>privvmpages</code>,
normal priority allocations and, starting from the <code>limit</code>
of <code>privvmpages</code>, all memory allocations
made by the applications fail.
The memory allocation guarantee (<code>vmguarpages</code>) is a primary tool for
controlling the memory available to containers, because
it allows administrators to provide Service Level Agreements — agreements
guaranteeing certain quality of service, certain amount of resources
and general availability of the service. The unit of measurement
of vmguarpages values is memory pages (4KB on x86 and x86_64 processors).
The total memory allocation guarantees given to containers
are limited by the physical resources of the computer — the size of RAM
and the swap space — as discussed in [[UBC systemwide configuration]].

There is a ''pseudo-graphical'' tool - <code>[http://en.dklab.ru/lib/dklab_vzmem/ vzmem]</code> - which allows you to distribute physical memory among all VEs consistently. It shows all physical memory blocks graphically in <code>/etc/vz/conf/MEM-MAP</code> text file and lets you to move these blocks from one VE to another to redistribute the memory. Also you may specify "additional" memory personally for each VE: such memory will be obtained from system's free memory or swap (it is reflected as modifying of <code>privvmpages</code> parameter).

== Units ==
{{Main|UBC parameter units}}

UBC auxiliary parameters

2011-08-15T22:44:42Z

Nathanhaigh: Indented parameters by another heading level so toc is easier to follow. linked units subheading to the UBC parameter units page

UBC primary parameters

2011-08-15T22:40:32Z

Nathanhaigh: Indented parameters by another heading level so toc is easier to follow. linked units subheading to the UBC parameter units page

UBC secondary parameters

2011-08-15T22:37:43Z

Nathanhaigh: Indented parameters by another heading level so toc is easier to follow

{{UBC toc}}

'''Secondary (dependant) UBC parameters''' are directly connected
to the [[UBC primary parameters|primary ones]] and can't be configured arbitrarily.

== UBC Secondary Parameters ==
=== kmemsize ===
Size of unswappable memory in bytes, allocated by the operating system kernel.

It includes all the kernel internal data structures associated with the
container's processes, except the network buffers discussed below.
These data structures reside in the first gigabyte of the computer's RAM,
so called [[UBC systemwide configuration#“Low memory”|“low memory”]].

This parameter is related to the number of processes ([[numproc]]).
Each process consumes certain amount of kernel memory —
24 kilobytes at minimum, 30–60 KB typically.
Very large processes may consume much more than that.

It is important to have a certain safety gap between the <code>barrier</code> and
the <code>limit</code> of the <code>kmemsize</code> parameter
(for example, 10%, as in [[UBC configuration examples]]). Equal <code>barrier</code> and <code>limit</code> of
the <code>kmemsize</code> parameter may lead to the situation where the kernel will
need to kill container's applications to keep the <code>kmemsize</code>
usage under the limit.

<code>Kmemsize</code> limits can't be set arbitrarily high.
The total amount of <code>kmemsize</code> consumable by all containers
in the system plus the socket buffer space (see below) is limited by the
hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

=== tcpsndbuf ===
The total size of buffers used to send data over TCP network connections.
These socket buffers reside in [[UBC systemwide configuration#“Low memory”|“low memory”]].

<code>Tcpsndbuf</code> parameter depends on number of TCP
sockets ([[numtcpsock]]) and should allow for some minimal amount of
socket buffer memory for each socket, as discussed in [[UBC consistency check]]:

<math>tcpsndbuf_{lim} - tcpsndbuf_{bar} \ge 2.5KB \cdot numtcpsock \rm.</math>

If this restriction is not satisfied, some network connections
may silently stall, being unable to transmit data.

Setting high values for <code>tcpsndbuf</code> parameter
may, but doesn't necessarily, increase performance of network communications.
Note that, unlike most other parameters, hitting <code>tcpsndbuf</code>
limits and failed socket buffer allocations
do not have strong negative effect on the applications, but just reduce
performance of network communications.

If you use rtorrent in a container, a low value for <code>tcpsndbuf</code> may cause rtorrent to take unusual amount of cpu. In this case, you must put a higher value. Also watch the number of failcnt in /proc/user_beancounters.

<code>Tcpsndbuf</code> limits can't be set arbitrarily high.
The total amount of <code>tcpsndbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers is limited
by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

=== tcprcvbuf ===
The total size of buffers used to temporary store the data coming from TCP network connections.
These socket buffers also reside in [[UBC systemwide configuration#“Low memory”|“low memory”]].

<code>Tcprcvbuf</code> parameter depends on number of TCP
sockets ([[numtcpsock]]) and should allow for some minimal amount of
socket buffer memory for each socket, as discussed in [[UBC consistency check]]:

<math>tcprcvbuf_{lim} - tcprcvbuf_{bar} \ge 2.5KB \cdot numtcpsock \rm.</math>

If this restriction is not satisfied, some network connections
may stall, being unable to receive data, and will be terminated
after a couple of minutes.

Similarly to <code>tcpsndbuf</code>, setting high values for <code>tcprcvbuf</code>
parameter may, but doesn't necessarily, increase performance of network
communications.
Hitting <code>tcprcvbuf</code> limits and failed socket buffer allocations
do not have strong negative effect on the applications, but just reduce
performance of network communications.
However, staying above the <code>barrier</code> of <code>tcprcvbuf</code> parameter
for a long time is less harmless than for <code>tcpsndbuf</code>.
Long periods of exceeding the <code>barrier</code> may cause termination
of some connections.

<code>Tcprcvbuf</code> limits can't be set arbitrarily high.
The total amount of <code>tcprcvbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers is limited
by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

=== othersockbuf ===
The total size of buffers used by local (UNIX-domain) connections between
processes inside the system (such as connections to a local database server)
and send buffers of UDP and other datagram protocols.

<code>Othersockbuf</code> parameter depends on number of non-TCP sockets (<code>[[numothersock]]</code>).

<code>Othersockbuf</code> configuration should satisfy

<math>othersockbuf_{lim} - othersockbuf_{bar} \ge 2.5KB \cdot numothersock.</math>

Increased limit for <code>othersockbuf</code> is necessary for high performance of
communications through local (UNIX-domain) sockets.
However, similarly to <code>tcpsndbuf</code>, hitting <code>othersockbuf</code> affects
the communication performance only and does not affect the functionality.

<code>Othersockbuf</code> limits can't be set arbitrarily high.
The total amount of <code>othersockbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers
is limited by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

=== dgramrcvbuf ===
The total size of buffers used to temporary store the incoming packets of UDP and
other datagram protocols.

<code>Dgramrcvbuf</code> parameters depend on number of
non-TCP sockets (<code>[[numothersock]]</code>).

<code>Dgramrcvbuf</code> limits usually don't need to be high.
Only if the containers needs to send and receive very large
datagrams, the <code>barrier</code>s for both <code>othersockbuf</code> and
<code>dgramrcvbuf</code> parameters should be raised.

Hitting <code>dgramrcvbuf</code> means that some datagrams are dropped, which may
or may not be important for application functionality.
UDP is a protocol with not guaranteed delivery, so even if the buffers
permit, the datagrams may be as well dropped later on any stage of the
processing, and applications should be prepared for it.

Unlike other socket buffer parameters, for <code>dgramrcvbuf</code>
the <code>barrier</code> should be set to the <code>limit</code>.

<code>Dgramrcvbuf</code> limits can't be set arbitrarily high.
The total amount of <code>dgramrcvbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers
is limited by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

=== oomguarpages ===
The guaranteed amount of memory for the case the memory is “over-booked”
(out-of-memory kill guarantee).

<code>Oomguarpages</code> parameter is related to <code>[[vmguarpages]]</code>.
If applications start to consume more memory than the computer has,
the system faces an out-of-memory condition.
In this case the operating system will start to kill container's
processes to free some memory and prevent the total death
of the system. Although it happens very rarely in typical system loads,
killing processes in out-of-memory situations is a normal reaction of the
system, and it is built into every Linux kernel<ref>The possible reasons of out-of-memory situations are the excess of total <code>[[vmguarpages]]</code> guarantees the available physical resources or high memory consumption by system processes. Also, the kernel might allow some containers to allocate memory above their <code>[[vmguarpages]]</code> guarantees when the system had a lot of free memory, and later, when other containers claim their guarantees, the system will experience the memory shortage.</ref>.

<code>[[Oomguarpages]]</code> parameter accounts the total amount of
memory and swap space used by the processes of a particular
container.
The <code>barrier</code> of the <code>oomguarpages</code> parameter is the out-of-memory
guarantee.

If the current usage of memory and swap space
(the value of <code>oomguarpages</code>) plus the amount of used kernel memory
(<code>[[kmemsize]]</code>) and socket buffers is below the <code>barrier</code>,
processes in this container are guaranteed not to be killed in
out-of-memory situations.
If the system is in out-of-memory situation and there are several
containers with <code>oomguarpages</code> excess, applications in the
container with the biggest excess will be killed first.
The <code>failcnt</code> counter of <code>oomguarpages</code> parameter
increases when a process in this container is killed because
of out-of-memory situation.

If the administrator needs to make sure that some application won't be
forcedly killed regardless of the application's behavior,
setting the <code>[[privvmpages]]</code> limit to a value not greater than the
<code>oomguarpages</code> guarantee significantly reduce the likelihood of
the application being killed,
and setting it to a half of the <code>oomguarpages</code> guarantee completely
prevents it.
Such configurations are not popular because they significantly reduce
the utilization of the hardware.

The meaning of the <code>limit</code> for the <code>oomguarpages</code> parameter is
unspecified in the current version.

The total out-of-memory guarantees given to the containers should
not exceed the physical capacity of the computer, as discussed in [[UBC systemwide configuration#Memory and swap space]].
If guarantees are given for more than the system has, in out-of-memory
situations applications in containers with guaranteed level of
service and system daemons may be killed.

=== privvmpages ===
Memory allocation limit in [[Memory_page|pages]] (which are typically 4096 bytes in size).

<code>Privvmpages</code> parameter
allows controlling the amount of memory allocated by applications.

The <code>barrier</code> and the <code>limit</code> of <code>privvmpages</code> parameter
control the upper boundary of the total size of allocated memory.
Note that this upper boundary doesn't guarantee that the container
will be able to allocate that much memory, neither does it guarantee that
other containers will be able to allocate their fair share of
memory.
The primary mechanism to control memory allocation is the <code>[[vmguarpages]]</code>
guarantee.

<code>Privvmpages</code> parameter accounts allocated (but, possibly,
not used yet) memory.
The accounted value is an estimation how much memory will be really consumed
when the container's applications start to use the allocated
memory.
Consumed memory is accounted into <code>[[oomguarpages]]</code> parameter.

Since the memory accounted into <code>privvmpages</code> may not be actually used,
the sum of current <code>privvmpages</code> values for all containers
may exceed the RAM and swap size of the computer.

There should be a safety gap between the <code>barrier</code> and the <code>limit</code>
for <code>privvmpages</code> parameter to reduce the number of memory allocation
failures that the application is unable to handle.
This gap will be used for “high-priority” memory allocations, such
as process stack expansion.
Normal priority allocations will fail when the <code>barrier</code> of
<code>privvmpages</code> is reached.

Total <code>privvmpages</code> should correlate with the physical resources of the
computer.
Also, it is important not to allow any container to allocate a
significant portion of all system RAM to avoid serious service level
degradation for other containers.
Both these configuration requirements are discussed in [[UBC systemwide configuration#Allocated memory]].

There's also an article describing how [[user pages accounting]] works.

== Units ==

For full article, see: [[UBC parameter units]]

== System-wide limits ==
All secondary parameters are related to memory.
Total limits on memory-related parameters must not exceed the physical
resources of the computer.
The restrictions on the configuration of memory-related parameters are listed
in [[UBC systemwide configuration]].
Those restrictions are very important, because their violation may
allow any container cause the whole system to hang.

== Notes ==
<references/>

UBC secondary parameters

2011-08-15T22:34:10Z

Nathanhaigh: /* Units */ linked units subheading to the UBC parameter units page

{{UBC toc}}

'''Secondary (dependant) UBC parameters''' are directly connected
to the [[UBC primary parameters|primary ones]] and can't be configured arbitrarily.

== kmemsize ==
Size of unswappable memory in bytes, allocated by the operating system kernel.

It includes all the kernel internal data structures associated with the
container's processes, except the network buffers discussed below.
These data structures reside in the first gigabyte of the computer's RAM,
so called [[UBC systemwide configuration#“Low memory”|“low memory”]].

This parameter is related to the number of processes ([[numproc]]).
Each process consumes certain amount of kernel memory —
24 kilobytes at minimum, 30–60 KB typically.
Very large processes may consume much more than that.

It is important to have a certain safety gap between the <code>barrier</code> and
the <code>limit</code> of the <code>kmemsize</code> parameter
(for example, 10%, as in [[UBC configuration examples]]). Equal <code>barrier</code> and <code>limit</code> of
the <code>kmemsize</code> parameter may lead to the situation where the kernel will
need to kill container's applications to keep the <code>kmemsize</code>
usage under the limit.

<code>Kmemsize</code> limits can't be set arbitrarily high.
The total amount of <code>kmemsize</code> consumable by all containers
in the system plus the socket buffer space (see below) is limited by the
hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== tcpsndbuf ==
The total size of buffers used to send data over TCP network connections.
These socket buffers reside in [[UBC systemwide configuration#“Low memory”|“low memory”]].

<code>Tcpsndbuf</code> parameter depends on number of TCP
sockets ([[numtcpsock]]) and should allow for some minimal amount of
socket buffer memory for each socket, as discussed in [[UBC consistency check]]:

<math>tcpsndbuf_{lim} - tcpsndbuf_{bar} \ge 2.5KB \cdot numtcpsock \rm.</math>

If this restriction is not satisfied, some network connections
may silently stall, being unable to transmit data.

Setting high values for <code>tcpsndbuf</code> parameter
may, but doesn't necessarily, increase performance of network communications.
Note that, unlike most other parameters, hitting <code>tcpsndbuf</code>
limits and failed socket buffer allocations
do not have strong negative effect on the applications, but just reduce
performance of network communications.

If you use rtorrent in a container, a low value for <code>tcpsndbuf</code> may cause rtorrent to take unusual amount of cpu. In this case, you must put a higher value. Also watch the number of failcnt in /proc/user_beancounters.

<code>Tcpsndbuf</code> limits can't be set arbitrarily high.
The total amount of <code>tcpsndbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers is limited
by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== tcprcvbuf ==
The total size of buffers used to temporary store the data coming from TCP network connections.
These socket buffers also reside in [[UBC systemwide configuration#“Low memory”|“low memory”]].

<code>Tcprcvbuf</code> parameter depends on number of TCP
sockets ([[numtcpsock]]) and should allow for some minimal amount of
socket buffer memory for each socket, as discussed in [[UBC consistency check]]:

<math>tcprcvbuf_{lim} - tcprcvbuf_{bar} \ge 2.5KB \cdot numtcpsock \rm.</math>

If this restriction is not satisfied, some network connections
may stall, being unable to receive data, and will be terminated
after a couple of minutes.

Similarly to <code>tcpsndbuf</code>, setting high values for <code>tcprcvbuf</code>
parameter may, but doesn't necessarily, increase performance of network
communications.
Hitting <code>tcprcvbuf</code> limits and failed socket buffer allocations
do not have strong negative effect on the applications, but just reduce
performance of network communications.
However, staying above the <code>barrier</code> of <code>tcprcvbuf</code> parameter
for a long time is less harmless than for <code>tcpsndbuf</code>.
Long periods of exceeding the <code>barrier</code> may cause termination
of some connections.

<code>Tcprcvbuf</code> limits can't be set arbitrarily high.
The total amount of <code>tcprcvbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers is limited
by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== othersockbuf ==
The total size of buffers used by local (UNIX-domain) connections between
processes inside the system (such as connections to a local database server)
and send buffers of UDP and other datagram protocols.

<code>Othersockbuf</code> parameter depends on number of non-TCP sockets (<code>[[numothersock]]</code>).

<code>Othersockbuf</code> configuration should satisfy

<math>othersockbuf_{lim} - othersockbuf_{bar} \ge 2.5KB \cdot numothersock.</math>

Increased limit for <code>othersockbuf</code> is necessary for high performance of
communications through local (UNIX-domain) sockets.
However, similarly to <code>tcpsndbuf</code>, hitting <code>othersockbuf</code> affects
the communication performance only and does not affect the functionality.

<code>Othersockbuf</code> limits can't be set arbitrarily high.
The total amount of <code>othersockbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers
is limited by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== dgramrcvbuf ==
The total size of buffers used to temporary store the incoming packets of UDP and
other datagram protocols.

<code>Dgramrcvbuf</code> parameters depend on number of
non-TCP sockets (<code>[[numothersock]]</code>).

<code>Dgramrcvbuf</code> limits usually don't need to be high.
Only if the containers needs to send and receive very large
datagrams, the <code>barrier</code>s for both <code>othersockbuf</code> and
<code>dgramrcvbuf</code> parameters should be raised.

Hitting <code>dgramrcvbuf</code> means that some datagrams are dropped, which may
or may not be important for application functionality.
UDP is a protocol with not guaranteed delivery, so even if the buffers
permit, the datagrams may be as well dropped later on any stage of the
processing, and applications should be prepared for it.

Unlike other socket buffer parameters, for <code>dgramrcvbuf</code>
the <code>barrier</code> should be set to the <code>limit</code>.

<code>Dgramrcvbuf</code> limits can't be set arbitrarily high.
The total amount of <code>dgramrcvbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers
is limited by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== oomguarpages ==
The guaranteed amount of memory for the case the memory is “over-booked”
(out-of-memory kill guarantee).

<code>Oomguarpages</code> parameter is related to <code>[[vmguarpages]]</code>.
If applications start to consume more memory than the computer has,
the system faces an out-of-memory condition.
In this case the operating system will start to kill container's
processes to free some memory and prevent the total death
of the system. Although it happens very rarely in typical system loads,
killing processes in out-of-memory situations is a normal reaction of the
system, and it is built into every Linux kernel<ref>The possible reasons of out-of-memory situations are the excess of total <code>[[vmguarpages]]</code> guarantees the available physical resources or high memory consumption by system processes. Also, the kernel might allow some containers to allocate memory above their <code>[[vmguarpages]]</code> guarantees when the system had a lot of free memory, and later, when other containers claim their guarantees, the system will experience the memory shortage.</ref>.

<code>[[Oomguarpages]]</code> parameter accounts the total amount of
memory and swap space used by the processes of a particular
container.
The <code>barrier</code> of the <code>oomguarpages</code> parameter is the out-of-memory
guarantee.

If the current usage of memory and swap space
(the value of <code>oomguarpages</code>) plus the amount of used kernel memory
(<code>[[kmemsize]]</code>) and socket buffers is below the <code>barrier</code>,
processes in this container are guaranteed not to be killed in
out-of-memory situations.
If the system is in out-of-memory situation and there are several
containers with <code>oomguarpages</code> excess, applications in the
container with the biggest excess will be killed first.
The <code>failcnt</code> counter of <code>oomguarpages</code> parameter
increases when a process in this container is killed because
of out-of-memory situation.

If the administrator needs to make sure that some application won't be
forcedly killed regardless of the application's behavior,
setting the <code>[[privvmpages]]</code> limit to a value not greater than the
<code>oomguarpages</code> guarantee significantly reduce the likelihood of
the application being killed,
and setting it to a half of the <code>oomguarpages</code> guarantee completely
prevents it.
Such configurations are not popular because they significantly reduce
the utilization of the hardware.

The meaning of the <code>limit</code> for the <code>oomguarpages</code> parameter is
unspecified in the current version.

The total out-of-memory guarantees given to the containers should
not exceed the physical capacity of the computer, as discussed in [[UBC systemwide configuration#Memory and swap space]].
If guarantees are given for more than the system has, in out-of-memory
situations applications in containers with guaranteed level of
service and system daemons may be killed.

== privvmpages ==
Memory allocation limit in [[Memory_page|pages]] (which are typically 4096 bytes in size).

<code>Privvmpages</code> parameter
allows controlling the amount of memory allocated by applications.

The <code>barrier</code> and the <code>limit</code> of <code>privvmpages</code> parameter
control the upper boundary of the total size of allocated memory.
Note that this upper boundary doesn't guarantee that the container
will be able to allocate that much memory, neither does it guarantee that
other containers will be able to allocate their fair share of
memory.
The primary mechanism to control memory allocation is the <code>[[vmguarpages]]</code>
guarantee.

<code>Privvmpages</code> parameter accounts allocated (but, possibly,
not used yet) memory.
The accounted value is an estimation how much memory will be really consumed
when the container's applications start to use the allocated
memory.
Consumed memory is accounted into <code>[[oomguarpages]]</code> parameter.

Since the memory accounted into <code>privvmpages</code> may not be actually used,
the sum of current <code>privvmpages</code> values for all containers
may exceed the RAM and swap size of the computer.

There should be a safety gap between the <code>barrier</code> and the <code>limit</code>
for <code>privvmpages</code> parameter to reduce the number of memory allocation
failures that the application is unable to handle.
This gap will be used for “high-priority” memory allocations, such
as process stack expansion.
Normal priority allocations will fail when the <code>barrier</code> of
<code>privvmpages</code> is reached.

Total <code>privvmpages</code> should correlate with the physical resources of the
computer.
Also, it is important not to allow any container to allocate a
significant portion of all system RAM to avoid serious service level
degradation for other containers.
Both these configuration requirements are discussed in [[UBC systemwide configuration#Allocated memory]].

There's also an article describing how [[user pages accounting]] works.

== Units ==

For full article, see: [[UBC parameter units]]

== System-wide limits ==
All secondary parameters are related to memory.
Total limits on memory-related parameters must not exceed the physical
resources of the computer.
The restrictions on the configuration of memory-related parameters are listed
in [[UBC systemwide configuration]].
Those restrictions are very important, because their violation may
allow any container cause the whole system to hang.

== Notes ==
<references/>

Talk:Setting UBC parameters

2011-08-03T06:42:27Z

Nathanhaigh: Suggested refactoring of this page

I don't find summerising discussions/threads to be all that enlightening for a new user as myself. It's much better to summarise the main points in a succinct way. If you must use Q+A, couldn't they go into an FAQ. I came here thinking I might actually learn how to set the UBC parameters. Only after seeing vzctl on the [[Resource shortage]] page did I actually find what I need.

Unless there are screems of "NOoooooo", I'll refactor this page over the coming weeks. --[[User:Nathanhaigh|Nathanhaigh]] 06:42, 3 August 2011 (UTC)

UBC parameters table

2011-08-01T23:11:03Z

Nathanhaigh: Updated some descriptions to come in line with what's show on UBC_primary_parameters, UBC_secondary_parameters etc

{{UBC toc}}

{| class="wikitable"
! Name !! Description
|-
| colspan="2" align="center" | [[UBC primary parameters|Primary parameters]]
|-
| [[numproc]] || Maximum number of processes and kernel-level threads allowed for this container.
|-
| [[numtcpsock]] || Maximum number of TCP sockets.
|-
| [[numothersock]] || Maximum number of non-TCP sockets (local sockets, UDP and other types of sockets).
|-
| [[vmguarpages]] || Memory allocation guarantee.
|-
| colspan="2" align="center" | [[UBC secondary parameters|Secondary parameters]]
|-
| [[kmemsize]] || Size of unswappable kernel memory, allocated allocated for processes in this [[container]].
|-
| [[tcpsndbuf]] || Total size of buffers used to send data over TCP network connections.
|-
| [[tcprcvbuf]] || Total size of buffers used to temporary store the data coming from TCP network connections.
|-
| [[othersockbuf]] || Total size of UNIX-domain socket buffers, UDP and other datagram protocol send buffers.
|-
| [[dgramrcvbuf]] || Receive buffers of UDP and other datagram protocols.
|-
| [[oomguarpages]] || The guaranteed amount of memory for the case the memory is “over-booked” (out-of-memory kill guarantee).
|-
| [[privvmpages]] || Memory allocation limit, in pages.
|-
| colspan="2" align="center" | [[UBC auxiliary parameters|Auxiliary parameters]]
|-
| [[lockedpages]] || Process pages not allowed to be swapped out (pages locked by <code>mlock(2)</code>).
|-
| [[shmpages]] || Total size of shared memory (IPC, shared anonymous mappings and <code>tmpfs</code> objects), in pages.
|-
| [[physpages]] || Total number of RAM pages used by processes.
|-
| [[numfile]] || Number of open files.
|-
| [[numflock]] || Number of file locks.
|-
| [[numpty]] || Number of pseudo-terminals.
|-
| [[numsiginfo]] || Number of <code>siginfo</code> structures.
|-
| [[dcachesize]] || Total size of <code>dentry</code> and <code>inode</code> structures locked in memory.
|-
| [[numiptent]] || Number of NETFILTER (IP packet filtering) entries.
|-
| [[swappages]] || Amount of swap space to show in container.
|}

UBC parameters table

2011-08-01T23:04:05Z

Nathanhaigh: Link primary, secondary and auxiliary to their own pages

{{UBC toc}}

{| class="wikitable"
! Name !! Description
|-
| colspan="2" align="center" | [[UBC primary parameters|Primary parameters]]
|-
| [[numproc]] || Number of processes and threads.
|-
| [[numtcpsock]] || Number of TCP sockets.
|-
| [[numothersock]] || Number of sockets other than TCP.
|-
| [[vmguarpages]] || Memory allocation guarantee, in pages.
|-
| colspan="2" align="center" | [[UBC secondary parameters|Secondary parameters]]
|-
| [[kmemsize]] || Size of unswappable kernel memory, allocated for processes in this [[container]].
|-
| [[tcpsndbuf]] || Total size of TCP send buffers.
|-
| [[tcprcvbuf]] || Total size of TCP receive buffers.
|-
| [[othersockbuf]] || Total size of UNIX-domain socket buffers, UDP and other datagram protocol send buffers.
|-
| [[dgramrcvbuf]] || Receive buffers of UDP and other datagram protocols.
|-
| [[oomguarpages]] || The guaranteed amount of memory for the case the memory is “over-booked” (out-of-memory kill guarantee), in pages.
|-
| [[privvmpages]] || Memory allocation limit, in pages.
|-
| colspan="2" align="center" | [[UBC auxiliary parameters|Auxiliary parameters]]
|-
| [[lockedpages]] || Process pages not allowed to be swapped out (pages locked by <code>mlock(2)</code>).
|-
| [[shmpages]] || Total size of shared memory (IPC, shared anonymous mappings and <code>tmpfs</code> objects), in pages.
|-
| [[physpages]] || Total number of RAM pages used by processes.
|-
| [[numfile]] || Number of open files.
|-
| [[numflock]] || Number of file locks.
|-
| [[numpty]] || Number of pseudo-terminals.
|-
| [[numsiginfo]] || Number of <code>siginfo</code> structures.
|-
| [[dcachesize]] || Total size of <code>dentry</code> and <code>inode</code> structures locked in memory.
|-
| [[numiptent]] || Number of NETFILTER (IP packet filtering) entries.
|-
| [[swappages]] || Amount of swap space to show in container.
|}

UBC parameters table

2011-08-01T23:02:21Z

Nathanhaigh: Link each parameter to the details page of each parameter

{{UBC toc}}

{| class="wikitable"
! Name !! Description
|-
| colspan="2" align="center" | Primary parameters
|-
| [[numproc]] || Number of processes and threads.
|-
| [[numtcpsock]] || Number of TCP sockets.
|-
| [[numothersock]] || Number of sockets other than TCP.
|-
| [[vmguarpages]] || Memory allocation guarantee, in pages.
|-
| colspan="2" align="center" | Secondary parameters
|-
| [[kmemsize]] || Size of unswappable kernel memory, allocated for processes in this [[container]].
|-
| [[tcpsndbuf]] || Total size of TCP send buffers.
|-
| [[tcprcvbuf]] || Total size of TCP receive buffers.
|-
| [[othersockbuf]] || Total size of UNIX-domain socket buffers, UDP and other datagram protocol send buffers.
|-
| [[dgramrcvbuf]] || Receive buffers of UDP and other datagram protocols.
|-
| [[oomguarpages]] || The guaranteed amount of memory for the case the memory is “over-booked” (out-of-memory kill guarantee), in pages.
|-
| [[privvmpages]] || Memory allocation limit, in pages.
|-
| colspan="2" align="center" | Auxiliary parameters
|-
| [[lockedpages]] || Process pages not allowed to be swapped out (pages locked by <code>mlock(2)</code>).
|-
| [[shmpages]] || Total size of shared memory (IPC, shared anonymous mappings and <code>tmpfs</code> objects), in pages.
|-
| [[physpages]] || Total number of RAM pages used by processes.
|-
| [[numfile]] || Number of open files.
|-
| [[numflock]] || Number of file locks.
|-
| [[numpty]] || Number of pseudo-terminals.
|-
| [[numsiginfo]] || Number of <code>siginfo</code> structures.
|-
| [[dcachesize]] || Total size of <code>dentry</code> and <code>inode</code> structures locked in memory.
|-
| [[numiptent]] || Number of NETFILTER (IP packet filtering) entries.
|-
| [[swappages]] || Amount of swap space to show in container.
|}

UBC parameter units

2011-08-01T23:00:42Z

Nathanhaigh: Reformatted for easier reading

[[User Beancounters]] have default units which are used when viewing and setting the parameter values. When setting the values for some parameters, you are able to specify [[#Overriding Default Units|different units]] thereby avoiding the need to do manual conversion of units and making life a little easier!

{{UBC toc}}

== Pages ==

Parameters which have the suffix "page" are measured in numbers of [[memory page|pages]]. These include:

* [[UBC primary parameters|Primary Parameters]]
** <code>[[vmguarpages]]</code>

* [[UBC secondary parameters|Secondary Parameters]]
** <code>[[privvmpages]]</code>
** <code>[[oomguarpages]]</code>

* [[UBC auxiliary parameters|Auxiliary Parameters]]
** <code>[[lockedpages]]</code>
** <code>[[shmpages]]</code>
** <code>[[physpages]]</code>
** <code>[[swappages]]</code>

When using [[vzctl]] to set these beancounter parameters, you can override the default units of "pages" by using a [[#Overriding Default Units|valid suffix]].

== Numbers of Items ==

Parameters with the prefix "num" are measured in numbers of items:

* [[UBC primary parameters|Primary Parameters]]
** <code>[[numproc]]</code>
** <code>[[numtcpsock]]</code>
** <code>[[numothersock]]</code>

* [[UBC secondary parameters|Secondary Parameters]]
** None

* [[UBC auxiliary parameters|Auxiliary Parameters]]
** <code>[[numfile]]</code>
** <code>[[numflock]]</code>
** <code>[[numpty]]</code>
** <code>[[numsiginfo]]</code>
** <code>[[numiptent]]</code>

== Bytes ==

Other parameters are measured in bytes:

* [[UBC primary parameters|Primary Parameters]]
** None

* [[UBC secondary parameters|Secondary Parameters]]
** <code>[[kmemsize]]</code>
** <code>[[tcprcvbuf]]</code>
** <code>[[tcpsndbuf]]</code>
** <code>[[othersockbuf]]</code>
** <code>[[dgramrcvbuf]]</code>

* [[UBC auxiliary parameters|Auxiliary Parameters]]
** <code>[[dcachesize]]</code>

When using [[vzctl]] to set these beancounter parameters, you can override the default units of "bytes" by using a [[#Overriding Default Units|valid suffix]].

== Overriding Default Units ==

When using [[vzctl]] to set beancounter parameters which use "Pages" or "Bytes" as the default units, alternative units may be specified using one of the following suffixes:

{| class="wikitable"
!Suffix
! Alternative Suffix
! Units
|-
| g || G || gigabytes
|-
| m || M || megabytes
|-
| k || K || kilobytes
|-
| p || P || [[memory page|pages]]
|}

=== Examples ===

The following are some examples to demonstrate the use of different units when specifying the value of a parameter. Where <code>$CTID</code> is the container ID.

* Set <code>kmemsize</code> limit to 512 Kb
<pre>
# vzctl set $CTID --kmemsize 512k
</pre>

* Set <code>privvmpages</code> limit to 256 Mb
<pre>
# vzctl set $CTID --privvmpages 256m
</pre>

* Set <code>tcprcvbuf</code> limit to 1000 pages (totals to almost 4 Mb on x86)
<pre>
# vzctl set $CTID --tcprcvbuf 1000p
</pre>

[[Category: HOWTO]]

UBC secondary parameters

2011-08-01T04:57:12Z

Nathanhaigh: /* privvmpages */

{{UBC toc}}

'''Secondary (dependant) UBC parameters''' are directly connected
to the [[UBC primary parameters|primary ones]] and can't be configured arbitrarily.

== kmemsize ==
Size of unswappable memory in bytes, allocated by the operating system kernel.

It includes all the kernel internal data structures associated with the
container's processes, except the network buffers discussed below.
These data structures reside in the first gigabyte of the computer's RAM,
so called [[UBC systemwide configuration#“Low memory”|“low memory”]].

This parameter is related to the number of processes ([[numproc]]).
Each process consumes certain amount of kernel memory —
24 kilobytes at minimum, 30–60 KB typically.
Very large processes may consume much more than that.

It is important to have a certain safety gap between the <code>barrier</code> and
the <code>limit</code> of the <code>kmemsize</code> parameter
(for example, 10%, as in [[UBC configuration examples]]). Equal <code>barrier</code> and <code>limit</code> of
the <code>kmemsize</code> parameter may lead to the situation where the kernel will
need to kill container's applications to keep the <code>kmemsize</code>
usage under the limit.

<code>Kmemsize</code> limits can't be set arbitrarily high.
The total amount of <code>kmemsize</code> consumable by all containers
in the system plus the socket buffer space (see below) is limited by the
hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== tcpsndbuf ==
The total size of buffers used to send data over TCP network connections.
These socket buffers reside in [[UBC systemwide configuration#“Low memory”|“low memory”]].

<code>Tcpsndbuf</code> parameter depends on number of TCP
sockets ([[numtcpsock]]) and should allow for some minimal amount of
socket buffer memory for each socket, as discussed in [[UBC consistency check]]:

<math>tcpsndbuf_{lim} - tcpsndbuf_{bar} \ge 2.5KB \cdot numtcpsock \rm.</math>

If this restriction is not satisfied, some network connections
may silently stall, being unable to transmit data.

Setting high values for <code>tcpsndbuf</code> parameter
may, but doesn't necessarily, increase performance of network communications.
Note that, unlike most other parameters, hitting <code>tcpsndbuf</code>
limits and failed socket buffer allocations
do not have strong negative effect on the applications, but just reduce
performance of network communications.

If you use rtorrent in a container, a low value for <code>tcpsndbuf</code> may cause rtorrent to take unusual amount of cpu. In this case, you must put a higher value. Also watch the number of failcnt in /proc/user_beancounters.

<code>Tcpsndbuf</code> limits can't be set arbitrarily high.
The total amount of <code>tcpsndbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers is limited
by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== tcprcvbuf ==
The total size of buffers used to temporary store the data coming from TCP network connections.
These socket buffers also reside in [[UBC systemwide configuration#“Low memory”|“low memory”]].

<code>Tcprcvbuf</code> parameter depends on number of TCP
sockets ([[numtcpsock]]) and should allow for some minimal amount of
socket buffer memory for each socket, as discussed in [[UBC consistency check]]:

<math>tcprcvbuf_{lim} - tcprcvbuf_{bar} \ge 2.5KB \cdot numtcpsock \rm.</math>

If this restriction is not satisfied, some network connections
may stall, being unable to receive data, and will be terminated
after a couple of minutes.

Similarly to <code>tcpsndbuf</code>, setting high values for <code>tcprcvbuf</code>
parameter may, but doesn't necessarily, increase performance of network
communications.
Hitting <code>tcprcvbuf</code> limits and failed socket buffer allocations
do not have strong negative effect on the applications, but just reduce
performance of network communications.
However, staying above the <code>barrier</code> of <code>tcprcvbuf</code> parameter
for a long time is less harmless than for <code>tcpsndbuf</code>.
Long periods of exceeding the <code>barrier</code> may cause termination
of some connections.

<code>Tcprcvbuf</code> limits can't be set arbitrarily high.
The total amount of <code>tcprcvbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers is limited
by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== othersockbuf ==
The total size of buffers used by local (UNIX-domain) connections between
processes inside the system (such as connections to a local database server)
and send buffers of UDP and other datagram protocols.

<code>Othersockbuf</code> parameter depends on number of non-TCP sockets (<code>[[numothersock]]</code>).

<code>Othersockbuf</code> configuration should satisfy

<math>othersockbuf_{lim} - othersockbuf_{bar} \ge 2.5KB \cdot numothersock.</math>

Increased limit for <code>othersockbuf</code> is necessary for high performance of
communications through local (UNIX-domain) sockets.
However, similarly to <code>tcpsndbuf</code>, hitting <code>othersockbuf</code> affects
the communication performance only and does not affect the functionality.

<code>Othersockbuf</code> limits can't be set arbitrarily high.
The total amount of <code>othersockbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers
is limited by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== dgramrcvbuf ==
The total size of buffers used to temporary store the incoming packets of UDP and
other datagram protocols.

<code>Dgramrcvbuf</code> parameters depend on number of
non-TCP sockets (<code>[[numothersock]]</code>).

<code>Dgramrcvbuf</code> limits usually don't need to be high.
Only if the containers needs to send and receive very large
datagrams, the <code>barrier</code>s for both <code>othersockbuf</code> and
<code>dgramrcvbuf</code> parameters should be raised.

Hitting <code>dgramrcvbuf</code> means that some datagrams are dropped, which may
or may not be important for application functionality.
UDP is a protocol with not guaranteed delivery, so even if the buffers
permit, the datagrams may be as well dropped later on any stage of the
processing, and applications should be prepared for it.

Unlike other socket buffer parameters, for <code>dgramrcvbuf</code>
the <code>barrier</code> should be set to the <code>limit</code>.

<code>Dgramrcvbuf</code> limits can't be set arbitrarily high.
The total amount of <code>dgramrcvbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers
is limited by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== oomguarpages ==
The guaranteed amount of memory for the case the memory is “over-booked”
(out-of-memory kill guarantee).

<code>Oomguarpages</code> parameter is related to <code>[[vmguarpages]]</code>.
If applications start to consume more memory than the computer has,
the system faces an out-of-memory condition.
In this case the operating system will start to kill container's
processes to free some memory and prevent the total death
of the system. Although it happens very rarely in typical system loads,
killing processes in out-of-memory situations is a normal reaction of the
system, and it is built into every Linux kernel<ref>The possible reasons of out-of-memory situations are the excess of total <code>[[vmguarpages]]</code> guarantees the available physical resources or high memory consumption by system processes. Also, the kernel might allow some containers to allocate memory above their <code>[[vmguarpages]]</code> guarantees when the system had a lot of free memory, and later, when other containers claim their guarantees, the system will experience the memory shortage.</ref>.

<code>[[Oomguarpages]]</code> parameter accounts the total amount of
memory and swap space used by the processes of a particular
container.
The <code>barrier</code> of the <code>oomguarpages</code> parameter is the out-of-memory
guarantee.

If the current usage of memory and swap space
(the value of <code>oomguarpages</code>) plus the amount of used kernel memory
(<code>[[kmemsize]]</code>) and socket buffers is below the <code>barrier</code>,
processes in this container are guaranteed not to be killed in
out-of-memory situations.
If the system is in out-of-memory situation and there are several
containers with <code>oomguarpages</code> excess, applications in the
container with the biggest excess will be killed first.
The <code>failcnt</code> counter of <code>oomguarpages</code> parameter
increases when a process in this container is killed because
of out-of-memory situation.

If the administrator needs to make sure that some application won't be
forcedly killed regardless of the application's behavior,
setting the <code>[[privvmpages]]</code> limit to a value not greater than the
<code>oomguarpages</code> guarantee significantly reduce the likelihood of
the application being killed,
and setting it to a half of the <code>oomguarpages</code> guarantee completely
prevents it.
Such configurations are not popular because they significantly reduce
the utilization of the hardware.

The meaning of the <code>limit</code> for the <code>oomguarpages</code> parameter is
unspecified in the current version.

The total out-of-memory guarantees given to the containers should
not exceed the physical capacity of the computer, as discussed in [[UBC systemwide configuration#Memory and swap space]].
If guarantees are given for more than the system has, in out-of-memory
situations applications in containers with guaranteed level of
service and system daemons may be killed.

== privvmpages ==
Memory allocation limit in [[Memory_page|pages]] (which are typically 4096 bytes in size).

<code>Privvmpages</code> parameter
allows controlling the amount of memory allocated by applications.

The <code>barrier</code> and the <code>limit</code> of <code>privvmpages</code> parameter
control the upper boundary of the total size of allocated memory.
Note that this upper boundary doesn't guarantee that the container
will be able to allocate that much memory, neither does it guarantee that
other containers will be able to allocate their fair share of
memory.
The primary mechanism to control memory allocation is the <code>[[vmguarpages]]</code>
guarantee.

<code>Privvmpages</code> parameter accounts allocated (but, possibly,
not used yet) memory.
The accounted value is an estimation how much memory will be really consumed
when the container's applications start to use the allocated
memory.
Consumed memory is accounted into <code>[[oomguarpages]]</code> parameter.

Since the memory accounted into <code>privvmpages</code> may not be actually used,
the sum of current <code>privvmpages</code> values for all containers
may exceed the RAM and swap size of the computer.

There should be a safety gap between the <code>barrier</code> and the <code>limit</code>
for <code>privvmpages</code> parameter to reduce the number of memory allocation
failures that the application is unable to handle.
This gap will be used for “high-priority” memory allocations, such
as process stack expansion.
Normal priority allocations will fail when the <code>barrier</code> of
<code>privvmpages</code> is reached.

Total <code>privvmpages</code> should correlate with the physical resources of the
computer.
Also, it is important not to allow any container to allocate a
significant portion of all system RAM to avoid serious service level
degradation for other containers.
Both these configuration requirements are discussed in [[UBC systemwide configuration#Allocated memory]].

There's also an article describing how [[user pages accounting]] works.

== Units ==
<code>Oomguarpages</code> and <code>privvmpages</code>
values are measured in [[memory page]]s.
For other secondary parameters, the values are in bytes.

== System-wide limits ==
All secondary parameters are related to memory.
Total limits on memory-related parameters must not exceed the physical
resources of the computer.
The restrictions on the configuration of memory-related parameters are listed
in [[UBC systemwide configuration]].
Those restrictions are very important, because their violation may
allow any container cause the whole system to hang.

== Notes ==
<references/>

UBC secondary parameters

2011-08-01T04:56:09Z

Nathanhaigh: /* privvmpages */ specified number of bytes per page for ease of accounting on beancounter stats

{{UBC toc}}

'''Secondary (dependant) UBC parameters''' are directly connected
to the [[UBC primary parameters|primary ones]] and can't be configured arbitrarily.

== kmemsize ==
Size of unswappable memory in bytes, allocated by the operating system kernel.

It includes all the kernel internal data structures associated with the
container's processes, except the network buffers discussed below.
These data structures reside in the first gigabyte of the computer's RAM,
so called [[UBC systemwide configuration#“Low memory”|“low memory”]].

This parameter is related to the number of processes ([[numproc]]).
Each process consumes certain amount of kernel memory —
24 kilobytes at minimum, 30–60 KB typically.
Very large processes may consume much more than that.

It is important to have a certain safety gap between the <code>barrier</code> and
the <code>limit</code> of the <code>kmemsize</code> parameter
(for example, 10%, as in [[UBC configuration examples]]). Equal <code>barrier</code> and <code>limit</code> of
the <code>kmemsize</code> parameter may lead to the situation where the kernel will
need to kill container's applications to keep the <code>kmemsize</code>
usage under the limit.

<code>Kmemsize</code> limits can't be set arbitrarily high.
The total amount of <code>kmemsize</code> consumable by all containers
in the system plus the socket buffer space (see below) is limited by the
hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== tcpsndbuf ==
The total size of buffers used to send data over TCP network connections.
These socket buffers reside in [[UBC systemwide configuration#“Low memory”|“low memory”]].

<code>Tcpsndbuf</code> parameter depends on number of TCP
sockets ([[numtcpsock]]) and should allow for some minimal amount of
socket buffer memory for each socket, as discussed in [[UBC consistency check]]:

<math>tcpsndbuf_{lim} - tcpsndbuf_{bar} \ge 2.5KB \cdot numtcpsock \rm.</math>

If this restriction is not satisfied, some network connections
may silently stall, being unable to transmit data.

Setting high values for <code>tcpsndbuf</code> parameter
may, but doesn't necessarily, increase performance of network communications.
Note that, unlike most other parameters, hitting <code>tcpsndbuf</code>
limits and failed socket buffer allocations
do not have strong negative effect on the applications, but just reduce
performance of network communications.

If you use rtorrent in a container, a low value for <code>tcpsndbuf</code> may cause rtorrent to take unusual amount of cpu. In this case, you must put a higher value. Also watch the number of failcnt in /proc/user_beancounters.

<code>Tcpsndbuf</code> limits can't be set arbitrarily high.
The total amount of <code>tcpsndbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers is limited
by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== tcprcvbuf ==
The total size of buffers used to temporary store the data coming from TCP network connections.
These socket buffers also reside in [[UBC systemwide configuration#“Low memory”|“low memory”]].

<code>Tcprcvbuf</code> parameter depends on number of TCP
sockets ([[numtcpsock]]) and should allow for some minimal amount of
socket buffer memory for each socket, as discussed in [[UBC consistency check]]:

<math>tcprcvbuf_{lim} - tcprcvbuf_{bar} \ge 2.5KB \cdot numtcpsock \rm.</math>

If this restriction is not satisfied, some network connections
may stall, being unable to receive data, and will be terminated
after a couple of minutes.

Similarly to <code>tcpsndbuf</code>, setting high values for <code>tcprcvbuf</code>
parameter may, but doesn't necessarily, increase performance of network
communications.
Hitting <code>tcprcvbuf</code> limits and failed socket buffer allocations
do not have strong negative effect on the applications, but just reduce
performance of network communications.
However, staying above the <code>barrier</code> of <code>tcprcvbuf</code> parameter
for a long time is less harmless than for <code>tcpsndbuf</code>.
Long periods of exceeding the <code>barrier</code> may cause termination
of some connections.

<code>Tcprcvbuf</code> limits can't be set arbitrarily high.
The total amount of <code>tcprcvbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers is limited
by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== othersockbuf ==
The total size of buffers used by local (UNIX-domain) connections between
processes inside the system (such as connections to a local database server)
and send buffers of UDP and other datagram protocols.

<code>Othersockbuf</code> parameter depends on number of non-TCP sockets (<code>[[numothersock]]</code>).

<code>Othersockbuf</code> configuration should satisfy

<math>othersockbuf_{lim} - othersockbuf_{bar} \ge 2.5KB \cdot numothersock.</math>

Increased limit for <code>othersockbuf</code> is necessary for high performance of
communications through local (UNIX-domain) sockets.
However, similarly to <code>tcpsndbuf</code>, hitting <code>othersockbuf</code> affects
the communication performance only and does not affect the functionality.

<code>Othersockbuf</code> limits can't be set arbitrarily high.
The total amount of <code>othersockbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers
is limited by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== dgramrcvbuf ==
The total size of buffers used to temporary store the incoming packets of UDP and
other datagram protocols.

<code>Dgramrcvbuf</code> parameters depend on number of
non-TCP sockets (<code>[[numothersock]]</code>).

<code>Dgramrcvbuf</code> limits usually don't need to be high.
Only if the containers needs to send and receive very large
datagrams, the <code>barrier</code>s for both <code>othersockbuf</code> and
<code>dgramrcvbuf</code> parameters should be raised.

Hitting <code>dgramrcvbuf</code> means that some datagrams are dropped, which may
or may not be important for application functionality.
UDP is a protocol with not guaranteed delivery, so even if the buffers
permit, the datagrams may be as well dropped later on any stage of the
processing, and applications should be prepared for it.

Unlike other socket buffer parameters, for <code>dgramrcvbuf</code>
the <code>barrier</code> should be set to the <code>limit</code>.

<code>Dgramrcvbuf</code> limits can't be set arbitrarily high.
The total amount of <code>dgramrcvbuf</code> consumable by all containers
in the system plus the <code>kmemsize</code> and other socket buffers
is limited by the hardware resources of the system.
This total limit is discussed in [[UBC systemwide configuration#“Low memory”|“low memory”]].

== oomguarpages ==
The guaranteed amount of memory for the case the memory is “over-booked”
(out-of-memory kill guarantee).

<code>Oomguarpages</code> parameter is related to <code>[[vmguarpages]]</code>.
If applications start to consume more memory than the computer has,
the system faces an out-of-memory condition.
In this case the operating system will start to kill container's
processes to free some memory and prevent the total death
of the system. Although it happens very rarely in typical system loads,
killing processes in out-of-memory situations is a normal reaction of the
system, and it is built into every Linux kernel<ref>The possible reasons of out-of-memory situations are the excess of total <code>[[vmguarpages]]</code> guarantees the available physical resources or high memory consumption by system processes. Also, the kernel might allow some containers to allocate memory above their <code>[[vmguarpages]]</code> guarantees when the system had a lot of free memory, and later, when other containers claim their guarantees, the system will experience the memory shortage.</ref>.

<code>[[Oomguarpages]]</code> parameter accounts the total amount of
memory and swap space used by the processes of a particular
container.
The <code>barrier</code> of the <code>oomguarpages</code> parameter is the out-of-memory
guarantee.

If the current usage of memory and swap space
(the value of <code>oomguarpages</code>) plus the amount of used kernel memory
(<code>[[kmemsize]]</code>) and socket buffers is below the <code>barrier</code>,
processes in this container are guaranteed not to be killed in
out-of-memory situations.
If the system is in out-of-memory situation and there are several
containers with <code>oomguarpages</code> excess, applications in the
container with the biggest excess will be killed first.
The <code>failcnt</code> counter of <code>oomguarpages</code> parameter
increases when a process in this container is killed because
of out-of-memory situation.

If the administrator needs to make sure that some application won't be
forcedly killed regardless of the application's behavior,
setting the <code>[[privvmpages]]</code> limit to a value not greater than the
<code>oomguarpages</code> guarantee significantly reduce the likelihood of
the application being killed,
and setting it to a half of the <code>oomguarpages</code> guarantee completely
prevents it.
Such configurations are not popular because they significantly reduce
the utilization of the hardware.

The meaning of the <code>limit</code> for the <code>oomguarpages</code> parameter is
unspecified in the current version.

The total out-of-memory guarantees given to the containers should
not exceed the physical capacity of the computer, as discussed in [[UBC systemwide configuration#Memory and swap space]].
If guarantees are given for more than the system has, in out-of-memory
situations applications in containers with guaranteed level of
service and system daemons may be killed.

== privvmpages ==
Memory allocation limit in pages (which are typically 4096 bytes in size).

<code>Privvmpages</code> parameter
allows controlling the amount of memory allocated by applications.

The <code>barrier</code> and the <code>limit</code> of <code>privvmpages</code> parameter
control the upper boundary of the total size of allocated memory.
Note that this upper boundary doesn't guarantee that the container
will be able to allocate that much memory, neither does it guarantee that
other containers will be able to allocate their fair share of
memory.
The primary mechanism to control memory allocation is the <code>[[vmguarpages]]</code>
guarantee.

<code>Privvmpages</code> parameter accounts allocated (but, possibly,
not used yet) memory.
The accounted value is an estimation how much memory will be really consumed
when the container's applications start to use the allocated
memory.
Consumed memory is accounted into <code>[[oomguarpages]]</code> parameter.

Since the memory accounted into <code>privvmpages</code> may not be actually used,
the sum of current <code>privvmpages</code> values for all containers
may exceed the RAM and swap size of the computer.

There should be a safety gap between the <code>barrier</code> and the <code>limit</code>
for <code>privvmpages</code> parameter to reduce the number of memory allocation
failures that the application is unable to handle.
This gap will be used for “high-priority” memory allocations, such
as process stack expansion.
Normal priority allocations will fail when the <code>barrier</code> of
<code>privvmpages</code> is reached.

Total <code>privvmpages</code> should correlate with the physical resources of the
computer.
Also, it is important not to allow any container to allocate a
significant portion of all system RAM to avoid serious service level
degradation for other containers.
Both these configuration requirements are discussed in [[UBC systemwide configuration#Allocated memory]].

There's also an article describing how [[user pages accounting]] works.

== Units ==
<code>Oomguarpages</code> and <code>privvmpages</code>
values are measured in [[memory page]]s.
For other secondary parameters, the values are in bytes.

== System-wide limits ==
All secondary parameters are related to memory.
Total limits on memory-related parameters must not exceed the physical
resources of the computer.
The restrictions on the configuration of memory-related parameters are listed
in [[UBC systemwide configuration]].
Those restrictions are very important, because their violation may
allow any container cause the whole system to hang.

== Notes ==
<references/>