99
edits
Changes
no edit summary
{{UBC toc}}
This article describes internals of user-space pages accounting.
== Introduction ==
User pages is the second importaint resource (after kmemsize) which must be accounted.
Despite kernel memory, which is either used or not, the set of user pages may have different classifications. Pages may be backed by file, locked, unused, i.e. requesed with mmap/brk but not yet touched and so on. Thus user pages accounting is trickier that kernel pages' one.
== Ways of accounting ==
There are tree ways of user pages' accouting.
; Account all the mappings on mmap/brk and reject as soon as the sum of VMA-s lenghts reaches the barrier.
: This type is very bad as applications ''always'' map more than they really use and very oftem MUCH more.
; Account only the really used memory and reject as soon as RSS <ref>Resident Set Size</ref> reaches the limit.
: This type is not good either as the only place where pages appear in user space is page fault handler and the only way to reject is killing the task. Comparing to previous scenarion this is much worse as application won't even be able to close correctly.
; Account a part of memory on mmap/brk and reject there, and account the rest of the memory in page fault handlers without any rejects.
: This type of accounting it used in UBC.
== UBC user pages accounting ==
=== Terms ===
The following terms are used by UBC
# ''shmem mapping'' - this is the mapping of file, that belong to tmpfs. UBC accounts these pages separately and the description below doesn't take them into account;
# ''private mapping'' - this includes the following types of mappings:
## writable anonymous mappings
## writable private file mappings
#: Both types are not backed by disk file and thus may not be just freed. These mappings are charged with possible reject right when they are made - in sys_mmap()/sys_brk().
# ''unused pages'' - this is the number of pages which belong to private mapping, but are not yet touched.
=== Math model ===
The following notations are used:
<ol>
<li> <math>UB_{privvm}</math> is the total number of privvmpages accounted on UB. I.e. the value seen in <tt>/proc/user_beancounters</tt> in <tt>privvmpages.held</tt>;</li>
<li> <math>UB_{unused}</math> is the total number of unused pages. This parameter is shown in <tt>/proc/user_beancounters_debug</tt> file;</li>
<li> <math>Frac(page, UB)</math> is some value that represents the part of the page charged to beancounter in case, when page is mapped;</li>
<li> <math>RSS</math> is the amount of physical memory used by processes. </li>
</ol>
Page fraction (<math>Frac(page, UB)</math>) normally should be <math>\frac{1}{N}</math>, where <math>N</math> is the number of UBs the pages is shared between, but this is bad as adding a new UB to page shared set would require recalculating of the whole current set. In UB <math>Frac(page, UB) = \frac{1}{2^{UB_{shift}}}</math>, where <math>UB_{shift}</math> is some parameter which is calculated so that
<center>
<math>\sum _{UB : page \in UB} Frac(page, UB) = 1, \forall page</math>.
</center>
The notation <math>page \in UB</math> means "<em>there exists an mm_struct, where the page <math>page</math> is mapped to and this mm_struct belongs to <math>UB</math> beancounter</em>". This type of calculation allowed to make an O(1) algorithm of fractions accounting.
<tt>Privvmpages</tt> accounts the sum of unused pages and the "normalized" number of rss pages:
<center>
<math>
UB_{privvm} = UB_{unused} + \sum _{page \in UB} Frac(page, UB)
</math>
</center>
If you summ the <tt>privvmpages</tt> of all beancounters in the system you'll get an upper estimation of current physical memory usage by processes.
<center>
<math>
\sum _{UB} UB_{privvm} =
</math>
<math>
\sum _{UB} UB_{unused} + \sum _{UB} {\sum _{page \in UB} Frac(page, UB)} =
</math>
<math>
\sum _{UB} UB_{unused} + \sum _{page} \sum _{UB: page \in UB} Frac(page, UB) =
</math>
<math>
\sum _{page: page\ is\ mapped} 1 + \sum _{UB} UB_{unused} =
</math>
<math>
RSS + \sum _{UB} UB_{unused}
</math>
</center>
To get ''real'' physical memory usage <tt>physpages</tt> resource is used <ref>Actually <tt>physpages</tt> also include tmpfs resident pages</ref>.
<center>
<math>
UB_{physpages} = \sum _{page \in UB} Frac(page, UB)
</math>
</center>
Thus
<center>
<math>
\sum _{UB} UB_{physpages} = RSS
</math>
</center>
----
<references/>
This article describes internals of user-space pages accounting.
== Introduction ==
User pages is the second importaint resource (after kmemsize) which must be accounted.
Despite kernel memory, which is either used or not, the set of user pages may have different classifications. Pages may be backed by file, locked, unused, i.e. requesed with mmap/brk but not yet touched and so on. Thus user pages accounting is trickier that kernel pages' one.
== Ways of accounting ==
There are tree ways of user pages' accouting.
; Account all the mappings on mmap/brk and reject as soon as the sum of VMA-s lenghts reaches the barrier.
: This type is very bad as applications ''always'' map more than they really use and very oftem MUCH more.
; Account only the really used memory and reject as soon as RSS <ref>Resident Set Size</ref> reaches the limit.
: This type is not good either as the only place where pages appear in user space is page fault handler and the only way to reject is killing the task. Comparing to previous scenarion this is much worse as application won't even be able to close correctly.
; Account a part of memory on mmap/brk and reject there, and account the rest of the memory in page fault handlers without any rejects.
: This type of accounting it used in UBC.
== UBC user pages accounting ==
=== Terms ===
The following terms are used by UBC
# ''shmem mapping'' - this is the mapping of file, that belong to tmpfs. UBC accounts these pages separately and the description below doesn't take them into account;
# ''private mapping'' - this includes the following types of mappings:
## writable anonymous mappings
## writable private file mappings
#: Both types are not backed by disk file and thus may not be just freed. These mappings are charged with possible reject right when they are made - in sys_mmap()/sys_brk().
# ''unused pages'' - this is the number of pages which belong to private mapping, but are not yet touched.
=== Math model ===
The following notations are used:
<ol>
<li> <math>UB_{privvm}</math> is the total number of privvmpages accounted on UB. I.e. the value seen in <tt>/proc/user_beancounters</tt> in <tt>privvmpages.held</tt>;</li>
<li> <math>UB_{unused}</math> is the total number of unused pages. This parameter is shown in <tt>/proc/user_beancounters_debug</tt> file;</li>
<li> <math>Frac(page, UB)</math> is some value that represents the part of the page charged to beancounter in case, when page is mapped;</li>
<li> <math>RSS</math> is the amount of physical memory used by processes. </li>
</ol>
Page fraction (<math>Frac(page, UB)</math>) normally should be <math>\frac{1}{N}</math>, where <math>N</math> is the number of UBs the pages is shared between, but this is bad as adding a new UB to page shared set would require recalculating of the whole current set. In UB <math>Frac(page, UB) = \frac{1}{2^{UB_{shift}}}</math>, where <math>UB_{shift}</math> is some parameter which is calculated so that
<center>
<math>\sum _{UB : page \in UB} Frac(page, UB) = 1, \forall page</math>.
</center>
The notation <math>page \in UB</math> means "<em>there exists an mm_struct, where the page <math>page</math> is mapped to and this mm_struct belongs to <math>UB</math> beancounter</em>". This type of calculation allowed to make an O(1) algorithm of fractions accounting.
<tt>Privvmpages</tt> accounts the sum of unused pages and the "normalized" number of rss pages:
<center>
<math>
UB_{privvm} = UB_{unused} + \sum _{page \in UB} Frac(page, UB)
</math>
</center>
If you summ the <tt>privvmpages</tt> of all beancounters in the system you'll get an upper estimation of current physical memory usage by processes.
<center>
<math>
\sum _{UB} UB_{privvm} =
</math>
<math>
\sum _{UB} UB_{unused} + \sum _{UB} {\sum _{page \in UB} Frac(page, UB)} =
</math>
<math>
\sum _{UB} UB_{unused} + \sum _{page} \sum _{UB: page \in UB} Frac(page, UB) =
</math>
<math>
\sum _{page: page\ is\ mapped} 1 + \sum _{UB} UB_{unused} =
</math>
<math>
RSS + \sum _{UB} UB_{unused}
</math>
</center>
To get ''real'' physical memory usage <tt>physpages</tt> resource is used <ref>Actually <tt>physpages</tt> also include tmpfs resident pages</ref>.
<center>
<math>
UB_{physpages} = \sum _{page \in UB} Frac(page, UB)
</math>
</center>
Thus
<center>
<math>
\sum _{UB} UB_{physpages} = RSS
</math>
</center>
----
<references/>