Difference between revisions of "VSwap"

From OpenVZ Virtuozzo Containers Wiki
Jump to: navigation, search
(vswap in vz7 details added)
 
(19 intermediate revisions by 2 users not shown)
Line 1: Line 1:
New [[Download/kernel/rhel6|RHEL6-based OpenVZ]] kernel has a new memory management model, which supersedes [[UBC|User beancounters]]. It is called '''VSwap'''.
+
'''New [[Download/kernel/rhel6|RHEL6-based OpenVZ]] kernel''' has a new memory management model, which supersedes [[UBC|User beancounters]]. It is called '''VSwap'''.
  
Now you can set two primary parameters: <code>physpages</code> and <code>swappages</code>, while all the other beancounters become secondary.
+
== Primary parameters ==
 +
 
 +
With VSwap, there are two required parameters: <code>ram</code> and <code>swap</code> (a.k.a. <code>physpages</code> and <code>swappages</code>). All the other beancounters become optional.
  
 
* '''physpages'''
 
* '''physpages'''
: This parameter limits the physical memory (RAM) available to processes inside a container.
+
: This parameter sets the amount of fast physical memory (RAM) available to processes inside a container, in memory pages. Currently (as of 042stab042 kernel) the user memory, the kernel memory and the page cache are accounted into <code>physpages</code>.
: The <code>barrier</code> is ignored, and the <code>limit</code> sets the limit.
+
 
: Currently the user memory and the page cache are accounted into <code>physpages</code>.
+
: The <code>barrier</code> is ignored and should be set to 0, and the <code>limit</code> sets the limit.
 +
 
 +
* '''ram'''
 +
: is an easy shortcut for physpages.limit, and is measured in bytes
  
 
* '''swappages'''
 
* '''swappages'''
: This parameter limits the amount of swap space which can be used for processes inside a container.
+
: This parameter sets the amount of "slower memory" (vswap) available to processes inside a container, in memory pages.
: The <code>barrier</code> is ignored, and the <code>limit</code> sets the limit.
+
 
 +
: The <code>barrier</code> is ignored and should be set to 0, and the <code>limit</code> sets the limit.
 +
 
 +
* '''swap'''
 +
: is an easy shortcut for swappages.limit, and is measured in bytes
  
 
The sum of <code>physpages.limit</code> and <code>swappages.limit</code> limits the maximum amount
 
The sum of <code>physpages.limit</code> and <code>swappages.limit</code> limits the maximum amount
of allocated memory which can be used by a container. When physpages limit
+
of memory which can be used by a container. When physpages limit
 
is reached, memory pages belonging to the container are pushed out to
 
is reached, memory pages belonging to the container are pushed out to
 
so called virtual swap (''vswap''). The difference between normal swap
 
so called virtual swap (''vswap''). The difference between normal swap
Line 21: Line 30:
 
on the system.
 
on the system.
  
=== Secondary parameters ===
+
{{Note|swap used by a container can exceed <code>swappages.limit</code>, but is always within sum of <code>physpages.limit</code> and <code>swappages.limit</code>.}}
'''Even though these settings are classed as secondary, they are still important settings!'''
+
 
 +
=== Implicit UBC parameters ===
 +
 
 +
Since vzctl 4.6, if some optional beancounters are not set, vzctl sets them implicitly,
 +
based on '''ram''' and '''swap''' settings.
 +
 
 +
The following formulae are used:
 +
 
 +
<math>lockedpages_{bar} = oomguarpages_{bar} = ram</math>
 +
 
 +
<math>lockedpages_{lim} = oomguarpages_{lim} = \infty</math>
 +
 
 +
<math>vmguarpages_{bar} = vmguarpages_{lim} = ram + swap</math>
 +
 
 +
==== VM overcommit and privvmpages ====
 +
 
 +
vzctl 4.6 adds a new parameter, <code>--vm_overcommit</code>.
 +
Its only purpose is to be used in privvmpages calculation,
 +
in case VSwap is used and there is no explicit setting
 +
for privvmpages.
 +
 
 +
If <math>vm\_overcommit</math> is set:
 +
 
 +
: <math>privvmpages_{bar} = privvmpages_{lim} = (ram + swap) \times vm\_overcommit</math>
 +
 
 +
If it is not set:
 +
 
 +
: <math>privvmpages_{bar} = privvmpages_{lim} = \infty</math>
 +
 
 +
== Setting ==
 +
 
 +
{{Note|for VSwap, you need vswap-enabled kernel, ie [[Download/kernel/rhel6|RHEL6-based OpenVZ]] kernel.}}
 +
 
 +
Since vzctl 3.0.30, you can use <code>--ram</code> and <code>--swap</code> parameters, like this:
 +
 +
vzctl set 777 --ram 512M --swap 1G --save
 +
 
 +
== Convert non-VSwap CT to VSwap ==
 +
 
 +
If you have an existing container with usual UBC parameters set, and you want to convert this one into VSwap enabled config, here's what you need to do.
 +
 
 +
# Decide on how much RAM and swap you want this CT to have. Generally, sum of your new RAM+swap should be more or less equal to sum of old PRIVVMPAGES and KMEMSIZE.
 +
# Manually remove all UBC parameters from config. '''This is optional''', you can still have UBC limits applied if you want.
 +
# Add PHYSPAGES and SWAPPAGES parameters to config. Easiest way is to use <code>vzctl set $CTID --ram N --swap M --save</code>
 +
 
 +
Now your config is vswap enabled, and when you (re)start it (or use <code>--reset_ub</code>), vswap mechanism will be used by the kernel for this CT.
 +
 
 +
Here is an example of the above steps:
 +
 
 +
CTID=123
 +
RAM=1G
 +
SWAP=2G
 +
CFG=/etc/vz/conf/${CTID}.conf
 +
cp $CFG $CFG.pre-vswap
 +
grep -Ev '^(KMEMSIZE|LOCKEDPAGES|PRIVVMPAGES|SHMPAGES|NUMPROC|PHYSPAGES|VMGUARPAGES|OOMGUARPAGES|NUMTCPSOCK|NUMFLOCK|NUMPTY|NUMSIGINFO|TCPSNDBUF|TCPRCVBUF|OTHERSOCKBUF|DGRAMRCVBUF|NUMOTHERSOCK|DCACHESIZE|NUMFILE|AVNUMPROC|NUMIPTENT|ORIGIN_SAMPLE|SWAPPAGES)=' > $CFG <  $CFG.pre-vswap
 +
vzctl set $CTID --ram $RAM --swap $SWAP --save
 +
vzctl set $CTID --reset_ub
 +
 
 +
== How to distinguish between vswap and non-vswap configs? ==
 +
 
 +
Both <code>vzctl</code> and the kernel treats a configuration file as vswap one if PHYSPAGES limit is '''not''' set to <code>unlimited</code> (a.k.a. [[LONG_MAX]]). You can also use the following command:
 +
 
 +
# vzlist -o vswap $CTID
  
In addition to <code>physpages</code> and <code>swappages</code> being set, it is ''currently'' recommended to:
+
In addition, vzctl checks if kernel support vswap, and refuses to start a vswap-enabled container on a non vswap capable kernel. The check is presence of <code>/proc/vz/vswap</code> file.
  
* Set <code>kmemsize</code> to no more than 1/2 of <code>physpages</code>
+
== Details about vSwap implementation in Virtuozzo 7 ==
* Set <code>dcachesize</code> to no more than 1/2 of <code>kmemsize</code>
 
* Set <code>lockedpages</code> to no more than 1/2 of <code>physpages</code>
 
  
VSwap CT config samples (like [http://git.openvz.org/?p=vzctl;a=blob;f=etc/conf/ve-vswap-256m.conf-sample ve-vswap-256m.conf-sample] that come with vzctl are currently implementing these recommendations.
+
The Container swapping process is similar to that on a standalone computer.<br>
 +
This means in particular that some pages may get into the swap even if there is some free memory reported in the Container.<br>
 +
This may validly happen in case kernel memory management system detects some anonymous memory which is not touched for a long time by Container processes and decides that it's more
 +
efficient to put these anonymous pages into the swap and use more caches in a Container instead.
 +
 +
The Container swap space resides in physical node swap file.<br>
 +
When the swap-out for a Container starts, appropriate number of pages are allocated in physical swap on the host. Next
 +
# if there is no free memory on the host, the real swap-out of Container's memory to physical swap happens
 +
# if there is free memory on the host, the Container's memory is saved in a special swap cache in host's RAM and no real write to host's physical swap occurs
 +
 +
{{Note|The physical swap space is allocated in both cases anyway, this guarantees all amount of host's swap cache for Containers memory can be written into physical swap on host in case the host gets short of RAM.}}
 +
 +
* '''Consequence 1''': without any configured node swap file the Container's `SWAPPAGES` parameter will be ignored.
 +
 +
* '''Consequence 2''': if node's swap size is less than sum of all Containers' swap sizes on the node, Containers won't be able to use 100% of their swap simultaneously - similar to RAM settings.
  
{{Note|VSwap feature is still being worked on, so these recommendation might change in the future.}}
+
== See also ==
 +
* [http://openvz.livejournal.com/39644.html On vSwap and 042stab04x kernel improvements]
 +
* [http://openvz.livejournal.com/39765.html Recent improvements in vzctl]
  
 
[[Category: UBC]]
 
[[Category: UBC]]

Latest revision as of 15:57, 17 July 2020

New RHEL6-based OpenVZ kernel has a new memory management model, which supersedes User beancounters. It is called VSwap.

Primary parameters[edit]

With VSwap, there are two required parameters: ram and swap (a.k.a. physpages and swappages). All the other beancounters become optional.

  • physpages
This parameter sets the amount of fast physical memory (RAM) available to processes inside a container, in memory pages. Currently (as of 042stab042 kernel) the user memory, the kernel memory and the page cache are accounted into physpages.
The barrier is ignored and should be set to 0, and the limit sets the limit.
  • ram
is an easy shortcut for physpages.limit, and is measured in bytes
  • swappages
This parameter sets the amount of "slower memory" (vswap) available to processes inside a container, in memory pages.
The barrier is ignored and should be set to 0, and the limit sets the limit.
  • swap
is an easy shortcut for swappages.limit, and is measured in bytes

The sum of physpages.limit and swappages.limit limits the maximum amount of memory which can be used by a container. When physpages limit is reached, memory pages belonging to the container are pushed out to so called virtual swap (vswap). The difference between normal swap and vswap is that with vswap no actual disk I/O usually occurs. Instead, a container is artificially slowed down, to emulate the effect of the real swapping. Actual swap out occurs only if there is a global memory shortage on the system.

Yellowpin.svg Note: swap used by a container can exceed swappages.limit, but is always within sum of physpages.limit and swappages.limit.

Implicit UBC parameters[edit]

Since vzctl 4.6, if some optional beancounters are not set, vzctl sets them implicitly, based on ram and swap settings.

The following formulae are used:

VM overcommit and privvmpages[edit]

vzctl 4.6 adds a new parameter, --vm_overcommit. Its only purpose is to be used in privvmpages calculation, in case VSwap is used and there is no explicit setting for privvmpages.

If is set:

If it is not set:

Setting[edit]

Yellowpin.svg Note: for VSwap, you need vswap-enabled kernel, ie RHEL6-based OpenVZ kernel.

Since vzctl 3.0.30, you can use --ram and --swap parameters, like this:

vzctl set 777 --ram 512M --swap 1G --save

Convert non-VSwap CT to VSwap[edit]

If you have an existing container with usual UBC parameters set, and you want to convert this one into VSwap enabled config, here's what you need to do.

  1. Decide on how much RAM and swap you want this CT to have. Generally, sum of your new RAM+swap should be more or less equal to sum of old PRIVVMPAGES and KMEMSIZE.
  2. Manually remove all UBC parameters from config. This is optional, you can still have UBC limits applied if you want.
  3. Add PHYSPAGES and SWAPPAGES parameters to config. Easiest way is to use vzctl set $CTID --ram N --swap M --save

Now your config is vswap enabled, and when you (re)start it (or use --reset_ub), vswap mechanism will be used by the kernel for this CT.

Here is an example of the above steps:

CTID=123
RAM=1G
SWAP=2G
CFG=/etc/vz/conf/${CTID}.conf
cp $CFG $CFG.pre-vswap
grep -Ev '^(KMEMSIZE|LOCKEDPAGES|PRIVVMPAGES|SHMPAGES|NUMPROC|PHYSPAGES|VMGUARPAGES|OOMGUARPAGES|NUMTCPSOCK|NUMFLOCK|NUMPTY|NUMSIGINFO|TCPSNDBUF|TCPRCVBUF|OTHERSOCKBUF|DGRAMRCVBUF|NUMOTHERSOCK|DCACHESIZE|NUMFILE|AVNUMPROC|NUMIPTENT|ORIGIN_SAMPLE|SWAPPAGES)=' > $CFG <  $CFG.pre-vswap
vzctl set $CTID --ram $RAM --swap $SWAP --save
vzctl set $CTID --reset_ub

How to distinguish between vswap and non-vswap configs?[edit]

Both vzctl and the kernel treats a configuration file as vswap one if PHYSPAGES limit is not set to unlimited (a.k.a. LONG_MAX). You can also use the following command:

# vzlist -o vswap $CTID

In addition, vzctl checks if kernel support vswap, and refuses to start a vswap-enabled container on a non vswap capable kernel. The check is presence of /proc/vz/vswap file.

Details about vSwap implementation in Virtuozzo 7[edit]

The Container swapping process is similar to that on a standalone computer.
This means in particular that some pages may get into the swap even if there is some free memory reported in the Container.
This may validly happen in case kernel memory management system detects some anonymous memory which is not touched for a long time by Container processes and decides that it's more efficient to put these anonymous pages into the swap and use more caches in a Container instead.

The Container swap space resides in physical node swap file.
When the swap-out for a Container starts, appropriate number of pages are allocated in physical swap on the host. Next

  1. if there is no free memory on the host, the real swap-out of Container's memory to physical swap happens
  2. if there is free memory on the host, the Container's memory is saved in a special swap cache in host's RAM and no real write to host's physical swap occurs
Yellowpin.svg Note: The physical swap space is allocated in both cases anyway, this guarantees all amount of host's swap cache for Containers memory can be written into physical swap on host in case the host gets short of RAM.
  • Consequence 1: without any configured node swap file the Container's `SWAPPAGES` parameter will be ignored.
  • Consequence 2: if node's swap size is less than sum of all Containers' swap sizes on the node, Containers won't be able to use 100% of their swap simultaneously - similar to RAM settings.

See also[edit]