6,534
edits
Changes
→Other reasons of strange numbers: +ext4
The aim of this article is to understand where the numbers that are shown by <code>stat</code>/<code>df</code> utils in [[VEcontainer]] come from.
Consider typical OpenVZ setup, where <code>ext2</code> separate file system is mounted on <code>/vz</code>. <code>ext2</code> is called ''underlying file system'' in such situation. Linux VFS design allows every filesystem file system to export to userspace user space the following information concerning disk space (here and further I use <math>{subscript }</math> is used to specify paricular filesystem particular file system type):
* <math>total_{ext2}</math> - total amount of disk space that potentially can be acquired (e.g. HDD capacity)
* <math>avail_{ext2}</math> - amount of disk space that is still available for non-root users
Note, that not all free blocks can be used by non-root users: some amount of disk space is reserved for root. For example on <code>ext2</code> filesystem file system only root can use last free 5 percent (by default) of disk space. This is the difference between <math>avail_{ext2}</math> and <math>free_{ext2}</math>. Also marknote, that the following inequality is always true:
: <math>avail_{ext2} \le free_{ext2}</math> (1)
Inside a [[VEcontainer]] , a special filesystem file system type is used: , called <code>simfs</code>. This filesystem file system allows to isolate a particular [[VECT]] from other [[VE]]sCTs. Hence, when <code>df</code> or <code>stat</code> utils utilities are invoked , they get information from <code>simfs</code>, which exports to them the following values (by analogy with <code>ext2</code>):
* <math>total_{simfs}</math>
* <math>avail_{simfs}</math>
This article is in fact devoted to how simfs filesystem file system calculates the values above.
To produce any calculations , input data are required. What are input data for <code>simfs</code>? Except Aside from already discussed mentioned information from underlying filesystem file system (<math>total_{ext2}</math>, <math>total_{ext2}</math>, <math>total_{ext2}</math>) , one more element comes into force in OpenVZ environment. It is OpenVZ ''per-VE container disk quotas''. The values that provide this element are:
* <math>quota_{used}</math> - the number of blocks currently used by a [[VECT]]* <math>quota_{barrier}</math> - the number of blocks this VE CT can potentially can obtain.
OpenVZ disk quota counts the number of blocks currently used by VE a CT and prevents this number to be greater than the limit/barrier set.
<!-- TODO: Uncoment Uncomment after adding examples.
First let's use
<pre>
command to get information we need, and later discuss df output.
-->
:: <math>total_{simfs}</math> = <math>total_{ext2}</math>
:: <math>free_{simfs}</math> = <math>free_{ext2}</math>
:: <math>avail_{simfs} = avail_{ext2}</math>
:: <math>total_{simfs} = quota_{barrier}</math>
: Amount The amount of free space in this case should logically be the following:
:: <math>free_{simfs} = quota_{barrier} - quota_{used}</math>
: However here is a pitfall. Suppose that the amount of free disk space actually on the underlying filesystem is less than it is estimated from quota using the formule formula above, i.e.::: <math>free_{ext2} < quota_{barrier} - quota_{used} > free_{ext2}</math>: Then, definitely, the amount of free disk space reported by <code>simfs </code> should be other! different. This situation will be considered in the next point and in this point examined later; here we assume that there is enough space on partition, i.e:: <math>free_{ext2} \ge quota_{barrier} - quota_{used} \le free_{ext2}</math> (2): As concerns for amount of disk space available for non-root users, if there is enough disk space::: <math>avail_{ext2} \ge quota_{barrier} - quota_{used} \le avail_{ext2}</math>: and this is right in current point due to assumption (2) and the inequality (1), then amount of disk space available for non-root users in [[VE]] a CT equals the free space estimated from quota:
:: <math>free_{simfs} = quota_{barrier} - quota_{used}</math>
{{Clear}}
=== Quota is on for CT and there is NOT enough space on partition ===
{|
[[Image:Vzquota2.png|thumb|left]]
[[Image:Vzquota3.png|thumb|left]]
[[Image:Vzquota4.png|thumb|left]]
[[Image:Vzquota5.png|thumb|left]]
||
This is the most interesting and difficult to explain case. Nevertheless I tried to do it. So, our assumption is that:
: <math>quota_{barrier} - quota_{used} > free_{ext2}</math>
What should be reported as free space in such case? Of course, <math>free_{ext2}</math>! This is the actual amount of space that can be used by a CT. Hence:
: <math>free_{simfs} = free_{ext2}</math>
Now consider the following situation. There are two containers. First CT writes nothing to disk. Second CT writes something to the disk. An administrator of CT #1 looks at <code>df</code> output, noting the "Usage" column. What does she see?
:: <math>df_{usage} = total_{simfs} - free_{simfs} = total_{simfs} - free_{ext2}</math> (3)
<math>free_{ext2}</math> decreases because CT #2 writes to disk, consequently <math>df_{usage}</math> increases! “What the hell is going on?!”, — thinks the administrator: “Nobody writes to the disk [in my container], but the usage increases”! To avoid such a situation, the following approach is used in OpenVZ: decrease <math>total_{simfs}</math>, so that <math>df_{usage}</math> remains the same, i.e.:
: <math>total_{simfs} = quota_{usage} + free_{ext2}</math> (4)
By substituting (4) to (3), we get:
: <math>df_{usage} = total_{simfs} - free_{simfs} = quota_{usage} + free_{ext2} - free_{ext2} = quota_{usage} = const</math>
In this case, administrator of CT #1 sees that total amount of space decreases, but usage however is constant.
The same reason as with <math>free_{simfs}</math> fits for calculating <math>avail_simfs</math>. Two cases are possible. If
: <math>avail_{ext2} \ge quota_{barrier} - quota_{used}</math>
then
: <math>avail_{simfs} = free_{ext2}</math>
and if
: <math>avail_{ext2} < quota_{barrier} - quota_{used}</math>
then
: <math>avail_{simfs} = quota_{barrier} - quota_{used}</math>
The table below summarizes all possible cases.
|}
{{Clear}}
== Cases Conclusion ==
So there are three basic variants. Variant number one is not good, because a container's administrator can not get information about CT disk usage and the [[host system]] administrator can't limit CT disk usage. Variant three is not good because we have some weird (but logical) values in <code>df</code>/<code>stat</code> output in CT, e.g. total disk space can decrease. Variant two is perfect. How can we make sure that this variant always take place? Here is the simple rule:
{{Warning|Do not set random disk quota barrier/limit!}}
Even if you want a container to be unlimited, consider reasonable values. Use the following formula:
:: <math>\sum_{i=1}^Nq_i \le S - s</math> (5)
Here <math>q_i</math> is quota barrier for CT<math>i</math>,<br/>
<math>S</math> — total amount of space on underlying file system<br/>
<math>s</math> — amount of space used by everything else than CT private area: templates, locks, etc.
Note that if you install a template, you decrease <math>s</math>. This is bad because, ideally, after each template
installation you have to check inequality (5). To avoid this I suggest to mount separate partition on <code>/vz/private</code>, rather than on <code>/vz</code>. In such case <math>s</math> always equals <math>0</math>.
== Cases Summarizing Table ==
{| class="wikitable" align="center"
| colspan="2" | Quota off
| <math>total_{simfs} = total_{ext2}</math>
<math>Sfree_{simfs} = quota_{limit} - quota_{used}</math> - total amount of space on underlying filesystem
<math>savail_{simfs} = avail_{ext2}</math> |- amount of space | <math>free_{ext2} \le quota_{limit} - quota_{used by not VE private data: templates, locks, etc.}</math>'''<math>avail_{ext2} \le quota_{limit} - quota_{used}</math>| <math>total_{simfs} = quota_{usage} + free_{ext2}</math>
== TODO ==
TODO: Add examples with stat/df
[[Category: Troubleshooting]]
[[Category: Resource management]]
[[Category: Disk quota]]