The aim of this article is to understand where the numbers, that are shown by <code>stat</code>/<code>df</code> utils in VE, [[container]] come from.

Linux VFS design allows simfs (root Consider typical OpenVZ setup, where <code>ext2</code> separate file system type for VE) to get the following information concerning disk space from is mounted on <code>/vz</code>. <code>ext2</code> is called ''underlying filesystem:file system'' in such situation.

* '''ext2_total''' - total amount of disk space that potentially can be acquired* '''ext2_free''' - amount of disk Linux VFS design allows every file system to export to user space that is still free* '''ext2_avail''' - amount of the following information concerning disk space that (here and further <math>{subscript}</math> is still available for non-root usersused to specify particular file system type):

Note, * <math>total_{ext2}</math> - total amount of disk space that not all free blocks potentially can be used by non-root usersacquired (e.g. By default HDD capacity)* <math>free_{ext2 sets 5% barrier: 5 percent }</math> - amount of total disk space that is still free* <math>avail_{ext2}</math> - amount of disk space that is reserved still available for super user. This is the difference between '''ext2_avail''' and '''ext2_free'''. Note, that the following inequality is always true:: '''ext2_avail ≤ ext2_free (1)'''non-root users

Absolutely the same set Note that not all free blocks can be used by non-root users: some amount of information about disk space simfs should export to user-space if it is asked about it (e.greserved for root. For example on <code>statext2</code> or file system only root can use last free 5 percent (by default) of disk space. This is the difference between <math>avail_{ext2}<code/math> and <math>free_{ext2}</math>. Also note, that the following inequality is always true:: <math>dfavail_{ext2} \le free_{ext2}</codemath> command were invoked(1):

In OpenVZ environment one more element is pertinent in the situation we're considering: OpenVZ disk quota. First level of OpenVZ disk quota counts the number of blocks currently used by VE ('''q_used''') and prevents this number to be greater than the limit* <math>total_{simfs}</barrier set ('''q_barrier''').math>* <math>free_{simfs}</math>* <math>avail_{simfs}</math>

This article is in fact devoted to how simfs file system calculates the values above.  To produce any calculations, input data are required. What are input data for <code>simfs</code>? Aside from already mentioned information from underlying file system (<math>total_{ext2}</math>, <math>total_{ext2}</math>, <math>total_{ext2}</math>), one more element comes into force in OpenVZ environment. It is OpenVZ ''per-container disk quotas''. The values that provide this element are: * <math>quota_{used}</math> - the number of blocks currently used by a [[CT]]* <math>quota_{barrier}</math> - the number of blocks this CT can potentially obtain OpenVZ disk quota counts the number of blocks currently used by a CT and prevents this number to be greater than the limit/barrier set. <!-- TODO: Uncoment Uncomment after adding examples.

* '''== Cases == Consider three basic possible scenarios. === Quota is off for VE'''CT ===: If quota is off for VE a CT (<code>DISK_QUOTA=no</code>), the total amount of space, that VE this CT potentially can acquire, equals the amount of total space on a partition. Certainly some space can be used by other VEsCTs, but potentially VE a CT can have all the space on device. Number So, the number of free blocks for VE the CT equals the number of free blocks on partition. Note, that it implies that VE a CT root user, can fill in all the space, including the space, that is reserved for root user of HNthe [[host system]]. This is why, you shouldn't one should not reside VEs CTs private areas on root filesystem file system of your HNhost system. Amount The amount of available disk space for VE also CT equals the number of available blocks for the underlying filsystemfile system. Thus, we have the following relationships: :: '''simfs_total <math>total_{simfs}</math> = ext2_total''' <math>total_{ext2}</math>:: '''simfs_free <math>free_{simfs}</math> = ext2_free''' <math>free_{ext2}</math>:: '''simfs_avail <math>avail_{simfs} = ext2_avail'''avail_{ext2}</math> : Rather valueable valuable disadvantage of swithching switching off OpenVZ quota off (besides having unlimited CTs) is that you will not be able to get information about how much disk space is used by VE a CT (without doing possibly long term <code>du</code> command)using <code>df</code>/<code>stat</code>. I mean, that :: '''df_usage <math>df_{usage} = simfs_total total_{simfs} - simfs_free free_{simfs} = ext2_total total_{ext2} - ext2_free'''free_{ext2},</math>: thus in VE the CT you obtain information about disk usage of partition, but not disk usage of VEthe CT. === Quota is on for CT, and there is enough space on partition ===: [[Image:Vzquota1. By png|thumb|left]]: If disk quota is on, the amount of disk space that a CT can potentially acquire should be equal to the way, just quota barrier: :: <math>total_{simfs} = quota_{barrier}</math>: The amount of free space in this number case should logically be::: <math>free_{simfs} = quota_{barrier} - quota_{used}</math>: However here is a pitfall. Suppose that the amount of free disk space on the underlying filesystem is less than it is displayed estimated from quota using the formula above, i.e.::: <math> free_{ext2} < quota_{barrier} - quota_{used} </math>: Then, definitely, the amount of free disk space reported by <code>simfs</code> should be different. This situation will be examined later; here we assume that there is enough space on partition, i.e:: <math> free_{ext2} \ge quota_{barrier} - quota_{used} </math> (2): As for amount of disk space available for non-root users, if there is enough disk space::: <math>avail_{ext2} \ge quota_{barrier} - quota_{used}</math>: then amount of disk space available for non-root users in a CT equals the free space estimated from quota::: <math>free_{simfs} = quota_{barrier} - quota_{used}</math>{{Clear}} === Quota is on for CT and there is NOT enough space on partition ==={|[[Image:Vzquota2.png|thumb|left]][[Image:Vzquota3.png|thumb|left]][[Image:Vzquota4.png|thumb|left]][[Image:Vzquota5.png|thumb|left]]||This is the most interesting and difficult to explain case. Nevertheless I tried to do it. So, our assumption is that:: <math>quota_{barrier} - quota_{used} > free_{ext2}</math>What should be reported as free space in such case? Of course, <math>free_{ext2}</math>! This is the actual amount of space that can be used by a CT. Hence:: <math>free_{simfs} = free_{ext2}</math>Now consider the following situation. There are two containers. First CT writes nothing to disk. Second CT writes something to the disk. An administrator of CT #1 looks at <code>df</code> tool in output, noting the "Usage" column.What does she see?:: <math>df_{usage} = total_{simfs} - free_{simfs} = total_{simfs} - free_{ext2}</math> (3)<math>free_{ext2}</math> decreases because CT #2 writes to disk, consequently <math>df_{usage}</math> increases! “What the hell is going on?!”, — thinks the administrator: “Nobody writes to the disk [in my container], but the usage increases”! To avoid such a situation, the following approach is used in OpenVZ: decrease <math>total_{simfs}</math>, so that <math>df_{usage}</math> remains the same, i.e.:: <math>total_{simfs} = quota_{usage} + free_{ext2}</math> (4)By substituting (4) to (3), we get:: <math>df_{usage} = total_{simfs} - free_{simfs} = quota_{usage} + free_{ext2} - free_{ext2} = quota_{usage} = const</math>In this case, administrator of CT #1 sees that total amount of space decreases, but usage however is constant. The same reason as with <math>free_{simfs}</math> fits for calculating <math>avail_simfs</math>. Two cases are possible. If: <math>avail_{ext2} \ge quota_{barrier} - quota_{used}</math>then : <math>avail_{simfs} = free_{ext2}</math>and if: <math>avail_{ext2} < quota_{barrier} - quota_{used}</math>then: <math>avail_{simfs} = quota_{barrier} - quota_{used}</math> The table below summarizes all possible cases.|}{{Clear}} == Cases Conclusion ==So there are three basic variants. Variant number one is not good, because a container's administrator can not get information about CT disk usage and the [[host system]] administrator can't limit CT disk usage. Variant three is not good because we have some weird (but logical) values in <code>df</code>/<code>stat</code> output in CT, e.g. total disk space can decrease. Variant two is perfect. How can we make sure that this variant always take place? Here is the simple rule: {{Warning|Do not set random disk quota barrier/limit!}} Even if you want a container to be unlimited, consider reasonable values. Use the following formula: :: <math>\sum_{i=1}^Nq_i \le S - s</math> (5) Here <math>q_i</math> is quota barrier for CT<math>i</math>,<br/><math>S</math> — total amount of space on underlying file system<br/><math>s</math> — amount of space used by everything else than CT private area: templates, locks, etc. Note that if you install a template, you decrease <math>s</math>. This is bad because, ideally, after each templateinstallation you have to check inequality (5). To avoid this I suggest to mount separate partition on <code>/vz/private</code>, rather than on <code>/vz</code>. In such case <math>s</math> always equals <math>0</math>. == Cases Summarizing Table =={| class="wikitable" align="center"| colspan="2" | Quota off| <math>total_{simfs} = total_{ext2}</math>

* '''Quota is on for VE and there is enough space on partition''': If quota is on, amount of disk space that VE potentially can acquire should be equal quota barrier: :: '''simfs_total = q_barrier''': Amount of free space in this case should logically be the following::: '''simfs_free = q_barrier - q_used''': However here is a pitfall. Suppose that amount of free disk space actually on underlying filesystem is less than it is estimated from quota using the formule above, i.e.::: '''q_barrier - q_used <math> ext2_free''': Then, definitely, amount of free disk space reported by free_{simfs should be other! This situation will be considered in the next point and in this point we assume that there is enough space on partition, i.e:: '''q_barrier - q_used ≤ ext2_free (2)''': As concerns amount of disk space available for non-root users, if there is enough disk space::: '''q_barrier - q_used ≤ ext2_avail''': and this is right in current point due to assumption (2) and the inequality (1), then amount of disk space available for non-root users in VE equals free space estimated from quota::: '''simfs_free } = q_barrier - q_used'''free_{ext2}</math>

* '''Quota is on for VE and there is NOT enough space on partition''': This is the most interesting and difficult to explain case. Nevertheless I tried to do it. So, our assumption is that::: '''q_barrier - q_used <math> ext2_free''': What should be reported as free space in such case? Of course, '''ext2_free'''! This is the actual amount of space that can be used by VE. Hence::: '''simfs_free avail_{simfs} = ext2_free''': And now consider the following situation. There is two VEs. One of VEs writes nothing to disk. Second VE writes to disc some information. Administrator of VE #1 looks at <code>dfavail_{ext2}</codemath> output. He observes the "Usage" column. What does he see?:: '''df_usage = simfs_total |- simfs_free | rowspan = simfs_total - ext2_free (3)'''| Quota on: '''ext2_free''' decreases because VE #2 writes to disc, consequently '''df_usage''' increases! "What the hell is going on?!" - thinks the administrator? | <math>free_{ext2} \ge quota_{limit} - "Nobody writes on the disk in my VE, but the usage increases!" To avoid such situation the following approach is quota_{used in OpenVZ disk quota: decrease '''simfs_total''' so, that '''df_usage''' remains the same, i.e.:}</math>:: '''simfs_total = ext2_total <math>avail_{ext2} \ge quota_{limit} - (q_barrier - q_usage) (4)''' quota_{used}</math>: Substituting (4) in (3) obtain::: '''df_usage | <math>total_{simfs} = (ext2_total - ext2_free) - (q_barrier - q_usage) = const''': In this case, administrator of VE #1 sees that total amount of space decreases, but usage however is constant and it is good.: The same reasoning as with '''simfs_free''' suits for calculating '''simfs_avail'''. Two cases are possible. If:: '''q_barrier - q_used ≤ ext2_avail''': then :: '''simfs_avail = ext2_free''': and if:: '''q_barrier - q_used quota_{barrier}</math> ext2_avail''': then:: '''simfs_avail = q_barrier - q_used'''

We have three variants. Variant number one is not good, because VE administrator can't get information about VE disk usage. Variant three is not good 'cause we have some weird (but logical) values in <codemath>dfavail_{simfs} = quota_{limit} - quota_{used}</codemath>/|-| <codemath>statfree_{ext2} \ge quota_{limit} - quota_{used}</codemath> output in VE, e.g. total disk space can decrease. Variant two is perfect. How can we provide this varaint always take place? Here is the rule:* '''Do not set random disk quota barrier/limit! Even if you want VE to be unlimited, consider reasonable values. Use the following fromule:: <math>avail_{ext2} \sum_le quota_{i=1limit}^Nq_i = S - squota_{used}</math>| <math>q_itotal_{simfs} = quota_{barrier}</math> - quota barrier for VE i

<math>Sfree_{simfs} = quota_{limit} - quota_{used}</math> - total amount of space on underlying filesystem

<math>savail_{simfs} = avail_{ext2}</math> |- amount of space | <math>free_{ext2} \le quota_{limit} - quota_{used by not VE private data: templates, locks, etc.}</math>'''<math>avail_{ext2} \le quota_{limit} - quota_{used}</math>| <math>total_{simfs} = quota_{usage} + free_{ext2}</math>

TODO[[Category: Change to mathTroubleshooting]][[Category: <math> avail_{ext2} <= free_{ext2} (1) </math> an alikeResource management]][[Category: Disk quota]]