Editing Disk quota, df and stat weird behaviour
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.
The edit can be undone.
Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 1: | Line 1: | ||
− | The aim of this article is to understand where the numbers that are shown by <code>stat</code>/<code>df</code> utils in [[ | + | The aim of this article is to understand where the numbers that are shown by <code>stat</code>/<code>df</code> utils in [[VE]] come from. |
− | + | Consider typical OpenVZ setup, where <code>ext2</code> separate filesystem is mounted on <code>/vz</code>. <code>ext2</code> is called ''underlying filesystem'' in such situation. | |
− | + | Linux VFS design allows every filesystem to export to userspace the following information concerning disk space (here and further I use subscript to specify paricular filesystem type): | |
− | |||
− | Linux VFS design allows every | ||
* <math>total_{ext2}</math> - total amount of disk space that potentially can be acquired (e.g. HDD capacity) | * <math>total_{ext2}</math> - total amount of disk space that potentially can be acquired (e.g. HDD capacity) | ||
Line 11: | Line 9: | ||
* <math>avail_{ext2}</math> - amount of disk space that is still available for non-root users | * <math>avail_{ext2}</math> - amount of disk space that is still available for non-root users | ||
− | Note that not all free blocks can be used by non-root users: some amount of disk space is reserved for root. For example on <code>ext2</code> | + | Note, that not all free blocks can be used by non-root users: some amount of disk space is reserved for root. For example on <code>ext2</code> filesystem only root can use last free 5 percent (by default) of disk space. This is the difference between <math>avail_{ext2}</math> and <math>free_{ext2}</math>. Also mark, that the following inequality is always true: |
: <math>avail_{ext2} \le free_{ext2}</math> (1) | : <math>avail_{ext2} \le free_{ext2}</math> (1) | ||
− | Inside | + | Inside [[VE]] special filesystem type is used: <code>simfs</code>. This filesystem allows to isolate particular [[VE]] from other [[VE]]s. Hence, when <code>df</code> or <code>stat</code> utils are invoked they get information from <code>simfs</code>, which exports to them the following values (by analogy with <code>ext2</code>): |
* <math>total_{simfs}</math> | * <math>total_{simfs}</math> | ||
Line 20: | Line 18: | ||
* <math>avail_{simfs}</math> | * <math>avail_{simfs}</math> | ||
− | This article is in fact devoted to how simfs | + | This article is in fact devoted to how simfs filesystem calculates the values above. |
− | To produce any calculations | + | To produce any calculations input data are required. What are input data for <code>simfs</code>? Except already discussed information from underlying filesystem (<math>total_{ext2}</math>, <math>total_{ext2}</math>, <math>total_{ext2}</math>) one more element comes into force in OpenVZ environment. It is OpenVZ ''per-VE disk quotas''. The values that provide this element are: |
− | * <math>quota_{used}</math> - the number of blocks currently used by | + | * <math>quota_{used}</math> - the number of blocks currently used by [[VE]] |
− | * <math>quota_{barrier}</math> - the number of blocks this | + | * <math>quota_{barrier}</math> - the number of blocks this VE potentially can obtain. |
− | OpenVZ disk quota counts the number of blocks currently used by | + | OpenVZ disk quota counts the number of blocks currently used by VE and prevents this number to be greater than the limit/barrier set. |
− | <!-- TODO: | + | <!-- TODO: Uncoment after adding examples. |
First let's use | First let's use | ||
<pre> | <pre> | ||
Line 36: | Line 34: | ||
command to get information we need, and later discuss df output. | command to get information we need, and later discuss df output. | ||
--> | --> | ||
+ | Consider three basic scenarios, that are possible: | ||
− | + | * '''Quota is off for VE''' | |
− | + | : If quota is off for VE (DISK_QUOTA=no), the total amount of space, that VE potentially can acquire, equals amount of total space on partition. Certainly some space can be used by other VEs, but potentially VE can have all the space on device. Number of free blocks for VE equals number of free blocks on partition. Note, that it implies that VE root user, can fill all the space, including the space, that is reserved for root user of HN. This is why, you shouldn't reside VEs private areas on root filesystem of your HN. Amount of available disk space for VE also equals the number of available blocks for underlying filsystem. Thus, we have the following relationships: | |
− | |||
− | |||
− | |||
− | : If quota is off for | ||
− | |||
:: <math>total_{simfs}</math> = <math>total_{ext2}</math> | :: <math>total_{simfs}</math> = <math>total_{ext2}</math> | ||
:: <math>free_{simfs}</math> = <math>free_{ext2}</math> | :: <math>free_{simfs}</math> = <math>free_{ext2}</math> | ||
:: <math>avail_{simfs} = avail_{ext2}</math> | :: <math>avail_{simfs} = avail_{ext2}</math> | ||
+ | : Rather valuable disadvantage of swithching OpenVZ quota off is that you will not be able to get information about how much disk space is used by VE (without doing possibly long term <code>du</code> command). I mean, that | ||
+ | :: <math>df_{usage} = total_{simfs} - free_{simfs} = total_{ext2} - free_{ext2}</math> | ||
+ | : thus in VE you obtain information about disk usage of partition, but not disk usage of VE. By the way, just this number is displayed by <code>df</code> tool in "Usage" column. | ||
− | + | * '''Quota is on for VE and there is enough space on partition''' | |
− | + | : If quota is on, amount of disk space that VE potentially can acquire should be equal quota barrier: | |
− | |||
− | |||
− | |||
− | |||
− | : If | ||
:: <math>total_{simfs} = quota_{barrier}</math> | :: <math>total_{simfs} = quota_{barrier}</math> | ||
− | : | + | : Amount of free space in this case should logically be the following: |
:: <math>free_{simfs} = quota_{barrier} - quota_{used}</math> | :: <math>free_{simfs} = quota_{barrier} - quota_{used}</math> | ||
− | : However here is a pitfall. Suppose that | + | : However here is a pitfall. Suppose that amount of free disk space actually on underlying filesystem is less than it is estimated from quota using the formule above, i.e.: |
− | :: <math> | + | :: <math>quota_{barrier} - quota_{used} > free_{ext2}</math> |
− | : Then, definitely, | + | : Then, definitely, amount of free disk space reported by simfs should be other! This situation will be considered in the next point and in this point we assume that there is enough space on partition, i.e |
− | :: <math> | + | :: <math>quota_{barrier} - quota_{used} \le free_{ext2}</math> (2) |
− | : As | + | : As concerns amount of disk space available for non-root users, if there is enough disk space: |
− | :: <math> | + | :: <math>quota_{barrier} - quota_{used} \le avail_{ext2}</math> |
− | : then amount of disk space available for non-root users in | + | : and this is right in current point due to assumption (2) and the inequality (1), then amount of disk space available for non-root users in [[VE]] equals free space estimated from quota: |
:: <math>free_{simfs} = quota_{barrier} - quota_{used}</math> | :: <math>free_{simfs} = quota_{barrier} - quota_{used}</math> | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | <math>free_{simfs} = free_{ext2}</math> | + | * '''Quota is on for VE and there is NOT enough space on partition''' |
+ | : This is the most interesting and difficult to explain case. Nevertheless I tried to do it. So, our assumption is that: | ||
+ | :: <math>quota_{barrier} - quota_{used} > free_{ext2}</math> | ||
+ | : What should be reported as free space in such case? Of course, <math>free_{ext2}</math>! This is the actual amount of space that can be used by VE. Hence: | ||
+ | :: <math>free_{simfs} = free_{ext2}</math> | ||
+ | : And now consider the following situation. There is two VEs. One of VEs writes nothing to disk. Second VE writes to disc some information. Administrator of VE #1 looks at <code>df</code> output. He observes the "Usage" column. What does he see? | ||
+ | :: <math>df_{usage} = total_{simfs} - free_{simfs} = total_simfs - free_{ext2}</math> (3) | ||
+ | : <math>free_{ext2}</math> decreases because [[VE]] #2 writes to disc, consequently <math>df_usage</math> increases! "What the hell is going on?!" - thinks the administrator? - "Nobody writes on the disk in my [[VE]], but the usage increases!" To avoid such situation the following approach is used in OpenVZ disk quota: decrease <math>simfs_total</math> so, that <math>df_usage</math> remains the same, i.e.: | ||
+ | :: <math>total_{simfs} = total_{ext2} - (quota_{barrier} - quota_{usage})</math> (4) | ||
+ | : Substituting (4) in (3) obtain: | ||
+ | :: <math>df_{usage} = (total_{ext2} - free_{ext2}) - (quota_{barrier} - quota_{usage}) = const</math> | ||
+ | : In this case, administrator of VE #1 sees that total amount of space decreases, but usage however is constant and it is good. | ||
+ | : The same reasoning as with <math>free_simfs</math> suits for calculating <math>avail_simfs</math>. Two cases are possible. If | ||
+ | :: <math>quota_{barrier} - quota_{used} \le avail_{ext2}</math> | ||
+ | : then | ||
+ | :: <math>avail_{simfs} = free_{ext2}</math> | ||
+ | : and if | ||
+ | :: <math>quota_{barrier} - quota_{used} > avail_{ext2}</math> | ||
+ | : then | ||
+ | :: <math>avail_{simfs} = quota_{barrier} - quota_{used}</math> | ||
− | + | The table below summarize all possible cases. | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | < | + | We have three variants. Variant number one is not good, because VE administrator can't get information about VE disk usage. Variant three is not good 'cause we have some weird (but logical) values in <code>df</code>/<code>stat</code> output in VE, e.g. total disk space can decrease. Variant two is perfect. How can we provide this varaint always take place? Here is the rule: |
+ | {{Out|Do not set random disk quota barrier/limit!}} | ||
− | <math> | + | Even if you want VE to be unlimited, consider reasonable values. Use the following formula: |
− | + | :: <math>\sum_{i=1}^Nq_i = S - s</math> | |
− | + | <math>q_i</math> - quota barrier for VE i | |
− | |||
− | |||
− | <math> | + | <math>S</math> - total amount of space on underlying filesystem |
− | <math> | + | <math>s</math> - amount of space used by not VE private data: templates, locks, etc. |
− | + | ''' | |
− | |||
− | |||
− | |||
− | + | ----- | |
+ | TODO: add info about inconsistent quota problems | ||
− | + | TODO: Add Roma's images | |
− | |||
− | + | TODO: Add table | |
− | |||
− | |||
− | |||
− | |||
TODO: Add examples with stat/df | TODO: Add examples with stat/df | ||
− | |||
− | |||
− | |||
− |