Difference between revisions of "Disk quota, df and stat weird behaviour"

From OpenVZ Virtuozzo Containers Wiki
Jump to: navigation, search
(Summary table is added)
(Table of contents, incondistent quota, other)
Line 1: Line 1:
 
The aim of this article is to understand where the numbers that are shown by <code>stat</code>/<code>df</code> utils in [[VE]] come from.  
 
The aim of this article is to understand where the numbers that are shown by <code>stat</code>/<code>df</code> utils in [[VE]] come from.  
  
 +
== Conventions and Notations ==
 
Consider typical OpenVZ setup, where <code>ext2</code> separate filesystem is mounted on <code>/vz</code>. <code>ext2</code> is called ''underlying filesystem'' in such situation.
 
Consider typical OpenVZ setup, where <code>ext2</code> separate filesystem is mounted on <code>/vz</code>. <code>ext2</code> is called ''underlying filesystem'' in such situation.
  
Line 34: Line 35:
 
command to get information we need, and later discuss df output.
 
command to get information we need, and later discuss df output.
 
-->
 
-->
Consider three basic scenarios, that are possible:
 
  
* '''Quota is off for VE'''
+
== Cases ==
 +
Consider three basic scenarios, that are possible.
 +
=== '''Quota is off for VE''' ===
 
: If quota is off for [[VE]] (DISK_QUOTA=no), the total amount of space, that [[VE]] potentially can acquire, equals amount of total space on partition. Certainly some space can be used by other [[VE]]s, but potentially [[VE]] can have all the space on device. Number of free blocks for [[VE]] equals number of free blocks on partition. Note, that it implies that [[VE]] root user, can fill all the space, including the space, that is reserved for root user of [[HN]]. This is why, you shouldn't reside [[VE]]s private areas on root filesystem of your [[HN]]. Amount of available disk space for [[VE]] equals the number of available blocks for underlying filsystem. Thus, we have the following relationships:
 
: If quota is off for [[VE]] (DISK_QUOTA=no), the total amount of space, that [[VE]] potentially can acquire, equals amount of total space on partition. Certainly some space can be used by other [[VE]]s, but potentially [[VE]] can have all the space on device. Number of free blocks for [[VE]] equals number of free blocks on partition. Note, that it implies that [[VE]] root user, can fill all the space, including the space, that is reserved for root user of [[HN]]. This is why, you shouldn't reside [[VE]]s private areas on root filesystem of your [[HN]]. Amount of available disk space for [[VE]] equals the number of available blocks for underlying filsystem. Thus, we have the following relationships:
 
:: <math>total_{simfs}</math> = <math>total_{ext2}</math>
 
:: <math>total_{simfs}</math> = <math>total_{ext2}</math>
Line 45: Line 47:
 
: thus in [[VE]] you obtain information about disk usage of partition, but not disk usage of [[VE]].
 
: thus in [[VE]] you obtain information about disk usage of partition, but not disk usage of [[VE]].
  
* '''Quota is on for VE and there is enough space on partition'''
+
=== '''Quota is on for VE and there is enough space on partition''' ===
 
: If quota is on, amount of disk space that [[VE]] potentially can acquire should be equal quota barrier:  
 
: If quota is on, amount of disk space that [[VE]] potentially can acquire should be equal quota barrier:  
 
:: <math>total_{simfs} = quota_{barrier}</math>
 
:: <math>total_{simfs} = quota_{barrier}</math>
Line 59: Line 61:
 
:: <math>free_{simfs} = quota_{barrier} - quota_{used}</math>
 
:: <math>free_{simfs} = quota_{barrier} - quota_{used}</math>
  
* '''Quota is on for VE and there is NOT enough space on partition'''
+
=== '''Quota is on for VE and there is NOT enough space on partition''' ===
 
: This is the most interesting and difficult to explain case. Nevertheless I tried to do it. So, our assumption is that:
 
: This is the most interesting and difficult to explain case. Nevertheless I tried to do it. So, our assumption is that:
 
:: <math>quota_{barrier} - quota_{used} > free_{ext2}</math>
 
:: <math>quota_{barrier} - quota_{used} > free_{ext2}</math>
Line 80: Line 82:
 
:: <math>avail_{simfs} = quota_{barrier} - quota_{used}</math>
 
:: <math>avail_{simfs} = quota_{barrier} - quota_{used}</math>
  
The table below summarize all possible cases.
+
The table below summarizes all possible cases.
  
 +
==  Cases Conclusion ==
 
So we have three basic variants. Variant number one is not good, because [[VE]] administrator can't get information about [[VE]] disk usage and [[HN]] administrator can't limit [[VE]] disk usage. Variant three is not good 'cause we have some weird (but logical) values in <code>df</code>/<code>stat</code> output in [[VE]], e.g. total disk space can decrease.  Variant two is perfect. How can we provide this varaint always take place? Here is the simple rule:
 
So we have three basic variants. Variant number one is not good, because [[VE]] administrator can't get information about [[VE]] disk usage and [[HN]] administrator can't limit [[VE]] disk usage. Variant three is not good 'cause we have some weird (but logical) values in <code>df</code>/<code>stat</code> output in [[VE]], e.g. total disk space can decrease.  Variant two is perfect. How can we provide this varaint always take place? Here is the simple rule:
  
Line 87: Line 90:
  
 
Even if you want [[VE]] to be unlimited, consider reasonable values. Use the following formula:
 
Even if you want [[VE]] to be unlimited, consider reasonable values. Use the following formula:
:: <math>\sum_{i=1}^Nq_i = S - s</math>
+
:: <math>\sum_{i=1}^Nq_i \le S - s</math> (5)
<math>q_i</math> - quota barrier for VE i
+
<math>q_i</math> - quota barrier for [[VE]] <math>i</math>
  
 
<math>S</math> - total amount of space on underlying filesystem
 
<math>S</math> - total amount of space on underlying filesystem
  
 
<math>s</math> - amount of space used by not [[VE]]s private area: templates, locks, etc.
 
<math>s</math> - amount of space used by not [[VE]]s private area: templates, locks, etc.
'''
+
 
 +
Note, that if you install template - you decrease <math>s</math>. This is bad, because, ideally, after each template
 +
installation you have to check inequality (5). To avoid this I suggest to mount separate partion on /vz/private, rather than
 +
on /vz/. In such case <math>s</math> always equals <math>0</math>.
 +
 
 +
== Cases Summarizing Table ==
 
{|  border="1" cellpadding="5" cellspacing="0" align="center"
 
{|  border="1" cellpadding="5" cellspacing="0" align="center"
|+'''Summarizing table'''
 
 
| colspan="2" | Quota off
 
| colspan="2" | Quota off
 
| <math>total_{simfs} = total_{ext2}</math>
 
| <math>total_{simfs} = total_{ext2}</math>
Line 128: Line 135:
 
<math>avail_{simfs} = avail_{ext2}</math>
 
<math>avail_{simfs} = avail_{ext2}</math>
 
|}
 
|}
-----
+
 
TODO: add info about inconsistent quota problems
+
== Other reasons of strange numbers ==
 +
At the moment I see only two more reasons, why numbers in <code>df</code>/<code>stat</code> output can confuse you.
 +
* The quota is inconsistent. This can happen if you turned quota off for some time, if you wrote directly to private area (<code> /vz/private</code>), but not through <code>simfs</code>, etc. When you have doubts wether your quota consistent or not just drop quota (<code>vzquota drop <veid></code>, where <code><veid></code> is the id of stoped [[VE]]). While starting [[VE]] <code>vzctl</code> will automatically initalize quota.
 +
* Unsupported underlying filesystem. Currently OpenVZ quota supports only <code>ext2</code>, <code>ext3</code>. With other filesystem types you can have unpredictable results. Praemonitus praemunitus!
 +
 
 +
== TODO ==
  
 
TODO: Add Roma's images
 
TODO: Add Roma's images
 
TODO: Add table
 
  
 
TODO: Add examples with stat/df
 
TODO: Add examples with stat/df

Revision as of 10:37, 29 November 2006

The aim of this article is to understand where the numbers that are shown by stat/df utils in VE come from.

Conventions and Notations

Consider typical OpenVZ setup, where ext2 separate filesystem is mounted on /vz. ext2 is called underlying filesystem in such situation.

Linux VFS design allows every filesystem to export to userspace the following information concerning disk space (here and further I use subscript to specify paricular filesystem type):

  • - total amount of disk space that potentially can be acquired (e.g. HDD capacity)
  • - amount of disk space that is still free
  • - amount of disk space that is still available for non-root users

Note, that not all free blocks can be used by non-root users: some amount of disk space is reserved for root. For example on ext2 filesystem only root can use last free 5 percent (by default) of disk space. This is the difference between and . Also mark, that the following inequality is always true:

(1)

Inside VE special filesystem type is used: simfs. This filesystem allows to isolate particular VE from other VEs. Hence, when df or stat utils are invoked they get information from simfs, which exports to them the following values (by analogy with ext2):

This article is in fact devoted to how simfs filesystem calculates the values above.

To produce any calculations input data are required. What are input data for simfs? Except already discussed information from underlying filesystem (, , ) one more element comes into force in OpenVZ environment. It is OpenVZ per-VE disk quotas. The values that provide this element are:

  • - the number of blocks currently used by VE
  • - the number of blocks this VE potentially can obtain

OpenVZ disk quota counts the number of blocks currently used by VE and prevents this number to be greater than the limit/barrier set.


Cases

Consider three basic scenarios, that are possible.

Quota is off for VE

If quota is off for VE (DISK_QUOTA=no), the total amount of space, that VE potentially can acquire, equals amount of total space on partition. Certainly some space can be used by other VEs, but potentially VE can have all the space on device. Number of free blocks for VE equals number of free blocks on partition. Note, that it implies that VE root user, can fill all the space, including the space, that is reserved for root user of HN. This is why, you shouldn't reside VEs private areas on root filesystem of your HN. Amount of available disk space for VE equals the number of available blocks for underlying filsystem. Thus, we have the following relationships:
=
=
Rather valuable disadvantage of swithching OpenVZ quota off (besides having unlimited VEs!) is that you will not be able to get information about how much disk space is used by VE (without doing possibly long term du command) using df/stat. I mean, that
thus in VE you obtain information about disk usage of partition, but not disk usage of VE.

Quota is on for VE and there is enough space on partition

If quota is on, amount of disk space that VE potentially can acquire should be equal quota barrier:
Amount of free space in this case should logically be the following:
However here is a pitfall. Suppose that the amount of free disk space actually on underlying filesystem is less than it is estimated from quota using the formule above, i.e.:
Then, definitely, amount of free disk space reported by simfs should be other! This situation will be considered in the next point and in this point we assume that there is enough space on partition, i.e
(2)
As concerns amount of disk space available for non-root users, if there is enough disk space:
then amount of disk space available for non-root users in VE equals free space estimated from quota:

Quota is on for VE and there is NOT enough space on partition

This is the most interesting and difficult to explain case. Nevertheless I tried to do it. So, our assumption is that:
What should be reported as free space in such case? Of course, ! This is the actual amount of space that can be used by VE. Hence:
And now consider the following situation. There is two VEs. One of VEs writes nothing to disk. Second VE writes to disc some information. Administrator of VE #1 looks at df output. He observes the "Usage" column. What does she see?
(3)
decreases because VE #2 writes to disc, consequently increases! "What the hell is going on?!" - thinks the administrator - "Nobody writes on the disk in my VE, but the usage increases!" To avoid such situation the following approach is used in OpenVZ: decrease so, that remains the same, i.e.:
(4)
Substituting (4) in (3) obtain:
In this case, administrator of VE #1 sees that total amount of space decreases, but usage however is constant.
The same reasoning as with suits for calculating . Two cases are possible. If
then
and if
then

The table below summarizes all possible cases.

Cases Conclusion

So we have three basic variants. Variant number one is not good, because VE administrator can't get information about VE disk usage and HN administrator can't limit VE disk usage. Variant three is not good 'cause we have some weird (but logical) values in df/stat output in VE, e.g. total disk space can decrease. Variant two is perfect. How can we provide this varaint always take place? Here is the simple rule:

Do not set random disk quota barrier/limit!

Even if you want VE to be unlimited, consider reasonable values. Use the following formula:

(5)

- quota barrier for VE

- total amount of space on underlying filesystem

- amount of space used by not VEs private area: templates, locks, etc.

Note, that if you install template - you decrease . This is bad, because, ideally, after each template installation you have to check inequality (5). To avoid this I suggest to mount separate partion on /vz/private, rather than on /vz/. In such case always equals .

Cases Summarizing Table

Quota off

Quota on

Other reasons of strange numbers

At the moment I see only two more reasons, why numbers in df/stat output can confuse you.

  • The quota is inconsistent. This can happen if you turned quota off for some time, if you wrote directly to private area ( /vz/private), but not through simfs, etc. When you have doubts wether your quota consistent or not just drop quota (vzquota drop <veid>, where <veid> is the id of stoped VE). While starting VE vzctl will automatically initalize quota.
  • Unsupported underlying filesystem. Currently OpenVZ quota supports only ext2, ext3. With other filesystem types you can have unpredictable results. Praemonitus praemunitus!

TODO

TODO: Add Roma's images

TODO: Add examples with stat/df