Editing Disk quota, df and stat weird behaviour

The aim of this article is to understand where the numbers that are shown by <code>stat</code>/<code>df</code> utils in [[VE]] come from. 

== Conventions and Notations ==
Consider typical OpenVZ setup, where <code>ext2</code> separate filesystem is mounted on <code>/vz</code>. <code>ext2</code> is called ''underlying filesystem'' in such situation.

Linux VFS design allows every filesystem to export to userspace the following information concerning disk space (here and further I use subscript to specify paricular filesystem type):

* <math>total_{ext2}</math> - total amount of disk space that potentially can be acquired (e.g. HDD capacity)
* <math>free_{ext2}</math>  - amount of disk space that is still free
* <math>avail_{ext2}</math> - amount of disk space that is still available for non-root users

Note, that not all free blocks can be used by non-root users: some amount of disk space is reserved for root. For example on <code>ext2</code> filesystem only root can use last free 5 percent (by default)  of disk space. This is the difference between <math>avail_{ext2}</math> and <math>free_{ext2}</math>. Also mark, that the following inequality is always true:
: <math>avail_{ext2} \le free_{ext2}</math> (1)

Inside [[VE]] special filesystem type is used: <code>simfs</code>. This filesystem allows to isolate particular [[VE]] from other [[VE]]s. Hence, when <code>df</code> or <code>stat</code> utils are invoked they get information from <code>simfs</code>, which exports to them the following values (by analogy with <code>ext2</code>):

* <math>total_{simfs}</math>
* <math>free_{simfs}</math>
* <math>avail_{simfs}</math>

This article is in fact devoted to how simfs filesystem calculates the values above. 

To produce any calculations input data are required. What are input data for <code>simfs</code>? Except already discussed information from underlying filesystem (<math>total_{ext2}</math>, <math>total_{ext2}</math>, <math>total_{ext2}</math>) one more element comes into force in OpenVZ environment. It is OpenVZ ''per-VE disk quotas''. The values that provide this element are:

* <math>quota_{used}</math> - the number of blocks currently used by [[VE]]
* <math>quota_{barrier}</math> - the number of blocks this [[VE]] potentially can obtain

OpenVZ disk quota counts the number of blocks currently used by VE and prevents this number to be greater than the limit/barrier set.

<!-- TODO: Uncoment after adding examples.
First let's use
<pre>
stat -f -c "total:%b free:%f avail:%a bs:%s" /path/to/fs
</pre>
command to get information we need, and later discuss df output.
-->

== Cases ==
Consider three basic scenarios, that are possible.
=== '''Quota is off for VE''' ===
: If quota is off for [[VE]] (DISK_QUOTA=no), the total amount of space, that [[VE]] potentially can acquire, equals amount of total space on partition. Certainly some space can be used by other [[VE]]s, but potentially [[VE]] can have all the space on device. Number of free blocks for [[VE]] equals number of free blocks on partition. Note, that it implies that [[VE]] root user, can fill all the space, including the space, that is reserved for root user of [[HN]]. This is why, you shouldn't reside [[VE]]s private areas on root filesystem of your [[HN]]. Amount of available disk space for [[VE]] equals the number of available blocks for underlying filsystem. Thus, we have the following relationships:
:: <math>total_{simfs}</math> = <math>total_{ext2}</math>
:: <math>free_{simfs}</math> = <math>free_{ext2}</math>
:: <math>avail_{simfs} = avail_{ext2}</math>
: Rather valuable disadvantage of swithching OpenVZ quota off (besides having unlimited [[VE]]s!) is that you will not be able to get information about how much disk space is used by [[VE]] (without doing possibly long term <code>du</code> command) using <code>df</code>/<code>stat</code>. I mean, that 
:: <math>df_{usage} = total_{simfs} - free_{simfs} = total_{ext2} - free_{ext2}</math>
: thus in [[VE]] you obtain information about disk usage of partition, but not disk usage of [[VE]].

=== '''Quota is on for VE and there is enough space on partition''' ( [[:Image:Vzquota1.png|illustration 1]]) ===
: If quota is on, amount of disk space that [[VE]] potentially can acquire should be equal quota barrier: 
:: <math>total_{simfs} = quota_{barrier}</math>
: Amount of free space in this case should logically be the following:
:: <math>free_{simfs} = quota_{barrier} - quota_{used}</math>
: However here is a pitfall. Suppose that the amount of free disk space actually on underlying filesystem is less than it is estimated from quota using the formule above, i.e.:
:: <math> free_{ext2} < quota_{barrier} - quota_{used} </math>
:  Then, definitely, amount of free disk space reported by <code>simfs</code> should be other!  This situation will be considered in the next point and in this point we assume that there is enough space on partition, i.e
:: <math> free_{ext2} \ge quota_{barrier} - quota_{used} </math> (2)
: As concerns amount of disk space available for non-root users, if there is enough disk space:
:: <math>avail_{ext2} \ge quota_{barrier} - quota_{used}</math>
: then amount of disk space available for non-root users in [[VE]] equals free space estimated from quota:
:: <math>free_{simfs} = quota_{barrier} - quota_{used}</math>

=== '''Quota is on for VE and there is NOT enough space on partition''' ===
: This is the most interesting and difficult to explain case. Nevertheless I tried to do it. So, our assumption is that:
:: <math>quota_{barrier} - quota_{used} > free_{ext2}</math>
: What should be reported as free space in such case? Of course, <math>free_{ext2}</math>! This is the actual amount of space that can be used by [[VE]]. Hence:
:: <math>free_{simfs} = free_{ext2}</math>
: And now consider the following situation. There is two [[VE]]s. One of [[VE]]s writes nothing to disk. Second [[VE]] writes to disc some information. Administrator of [[VE]] #1 looks at <code>df</code> output. He observes the "Usage" column. What does she see?
:: <math>df_{usage} = total_{simfs} - free_{simfs} = total_{simfs} - free_{ext2}</math> (3)
: <math>free_{ext2}</math> decreases because [[VE]] #2 writes to disc, consequently <math>df_{usage}</math> increases! "What the hell is going on?!" - thinks the administrator - "Nobody writes on the disk in my [[VE]], but the usage increases!" To avoid such situation the following approach is used in OpenVZ: decrease <math>total_{simfs}</math> so, that <math>df_{usage}</math> remains the same, i.e.:
:: <math>total_{simfs} = total_{ext2} - (quota_{barrier} - quota_{usage})</math> (4)
: Substituting (4) in (3) obtain:
:: <math>df_{usage} =  (total_{ext2} - free_{ext2}) - (quota_{barrier} - quota_{usage}) = const</math>
:  In this case, administrator of [[VE]] #1 sees that total amount of space decreases, but usage however is constant.
: The same reasoning as with <math>free_{simfs}</math> suits for calculating <math>avail_simfs</math>. Two cases are possible. If
:: <math>avail_{ext2} \ge quota_{barrier} - quota_{used}</math>
: then 
:: <math>avail_{simfs} = free_{ext2}</math>
: and if
:: <math>avail_{ext2} < quota_{barrier} - quota_{used}</math>
: then
:: <math>avail_{simfs} = quota_{barrier} - quota_{used}</math>

The table below summarizes all possible cases.

==  Cases Conclusion ==
So we have three basic variants. Variant number one is not good, because [[VE]] administrator can't get information about [[VE]] disk usage and [[HN]] administrator can't limit [[VE]] disk usage. Variant three is not good 'cause we have some weird (but logical) values in <code>df</code>/<code>stat</code> output in [[VE]], e.g. total disk space can decrease.  Variant two is perfect. How can we provide this varaint always take place? Here is the simple rule:

{{Out|Do not set random disk quota barrier/limit!}}

Even if you want [[VE]] to be unlimited, consider reasonable values. Use the following formula:
:: <math>\sum_{i=1}^Nq_i \le S - s</math> (5)
<math>q_i</math> - quota barrier for [[VE]] <math>i</math>

<math>S</math> - total amount of space on underlying filesystem

<math>s</math> - amount of space used by not [[VE]]s private area: templates, locks, etc.

Note, that if you install template - you decrease <math>s</math>. This is bad, because, ideally, after each template
installation you have to check inequality (5). To avoid this I suggest to mount separate partion on /vz/private, rather than
on /vz/. In such case <math>s</math> always equals <math>0</math>.

== Cases Summarizing Table ==
{|  border="1" cellpadding="5" cellspacing="0" align="center"
| colspan="2" | Quota off
| <math>total_{simfs} = total_{ext2}</math>

<math>free_{simfs} = free_{ext2}</math>

<math>avail_{simfs} = avail_{ext2}</math>
|- 
| rowspan = 3 | Quota on
| <math>free_{ext2} \ge quota_{limit} - quota_{used}</math>
<math>avail_{ext2} \ge quota_{limit} - quota_{used}</math>
| <math>total_{simfs} = quota_{barrier}</math>

<math>free_{simfs} = quota_{limit} - quota_{used}</math>

<math>avail_{simfs} = quota_{limit} - quota_{used}</math>
|-
| <math>free_{ext2} \ge quota_{limit} - quota_{used}</math>
<math>avail_{ext2} \le quota_{limit} - quota_{used}</math>
| <math>total_{simfs} = quota_{barrier}</math>

<math>free_{simfs} = quota_{limit} - quota_{used}</math>

<math>avail_{simfs} = avail_{ext2}</math>
|-
| <math>free_{ext2} \le quota_{limit} - quota_{used}</math>
<math>avail_{ext2} \le quota_{limit} - quota_{used}</math>
| <math>total_{simfs} = total_{ext2} - ( quota_{barrier} - quota_{usage} )</math>

<math>free_{simfs} = free_{ext2}</math>

<math>avail_{simfs} = avail_{ext2}</math>
|}

== Other reasons of strange numbers ==
At the moment I see only two more reasons, why numbers in <code>df</code>/<code>stat</code> output can confuse you.
* The quota is inconsistent. This can happen if you turned quota off for some time, if you wrote directly to private area (<code>/vz/private</code>), but not through <code>simfs</code>, etc. When you have doubts whether your quota is consistent or not, just drop quota (<code>vzquota drop <veid></code>, where <code><veid></code> is the id of stopped [[VE]]). While starting [[VE]] <code>vzctl</code> will automatically initalize quota.
* Unsupported underlying filesystem. Currently OpenVZ quota only supports <code>ext2</code> and <code>ext3</code>. With other filesystem types you can have unpredictable results. Praemonitus praemunitus!

== TODO ==

TODO: Add Roma's images

TODO: Add examples with stat/df