OpenVZ Virtuozzo Containers Wiki - User contributions [en]

Containers/Mini-summit 2008

2008-07-22T10:46:31Z

Balbir:

There will be a containers mini-summit at the [http://www.linuxsymposium.org/2008/ OLS'08]. This page is for organizing this mini-summit. Feel free to edit.

'''When''': 22nd of July 2008, 8:30-16:30<br/>
'''Where''': Ottawa, ON, Canada, Novotel Hotel (Albion A).

== Proposal ==

The mini-summit proposal sent to OLS organizers. '''See [[/Proposal|proposal]]'''.

== Topics to discuss ==

* Device accessibility cgroup (maybe with remap ability)
* TTYs
* Syslog
* Checkpoint/restart
* Memory controllers
* more?..

== List of attendees ==
Please fill in your name here if you are going to attend, or email kir at openvz dot org if you are too lazy. Surely the list is not final, so put your name even if you are not sure you can make it.

This list is in no particular order.


<div style="-moz-column-count:3; -webkit-column-count:3; column-count:3; text-align: left; background: #fefef0; border: 1px solid #ddddc0;">
# Pavel Emelyanov
# Denis Lunev
# Andrey Mirkin
# Serge Hallyn
# Dave Hansen
# Daniel Lezcano
# Srivatsa Vaddagiri
# Balbir Singh
# Sukadev Bhattiprolu
# Paul Menage
# Eric W. Biederman
# Oren Laadan
# Yamamoto Takashi
# Kamezawa Hiroyuki
# Benjamin Thery
# Herbert Pötzl
# Oleg Nesterov
# Dhaval Giani
# Bart Trojanowski
# Joseph Ruscio
# Constant Chan
# Linda Knippers
# Satoshi Uchida
# Masahiko Takahashi
# Martine Silbermann
# Benoit des Ligneris
# Patrick Naubert
# Daisuke Nishimura
# Sudhir Kumar
# Munehiro Ikeda
# Kamalesh Babulal
# John Schulz
# Poornima Nayak
# Gyuil Cha
# YoungHo Kim
# Rob Woolley
# Daniel Robbins
# Jason Baron
# Subrata Modak
# Veerendra C
# Joe MacDonald
# Andrew Theurer
# Myron Stowe
# Peter Teoh
</div>

== Agenda ==

* Namespaces/Containers (8:30am-11am)
** sysfs issues (and any /proc issues)
*** uevents/hotplug
** Network namespaces issues
*** multiple namespaces in one process
** Device namespace design?
** User namespace
** Additional needed namespaces
*** Small namespaces ''What to do with small subsystem that might need virtualization. E.g. in openvz we have FUSE, binfmt_misc and some other small stuff virtualized. But how to merge it in mainline? Create a separate namespace for each? Mere them into one? How to call this then?''
** Handling filesystem/namespace synchronization (not sure what the issue is)
** Container design
*** How to enter a container
*** Nature of a 'container' — kernel object or userspace fiction

* Cgroups+Resource management (11:30-2pm)
** Cgroup implementation
*** Locking (don't let cgroup_lock() become the BKL)
*** Transactional attachment
*** "procs" file
*** User-space notification API
**** Resource counter hit soft/hard limit
**** Task entered/left cgroup
**** OOM occurred
*** Binary statistics API
** Existing cgroups
*** Memory (Balbir's NOTE: I would prefer to take some of this discussion to my memory controller BoF on Wednesday. Lets discuss this at the end)
**** Supporting over-commit and guarantees
**** Soft-limits
**** Hierarchical borrowing - in kernel or userspace?
**** Per-cgroup refault information?
*** Kernel memory
*** Device
*** Memrlimit
**** Some push-back over this - can we give real use cases?
*** CPU scheduler
** Additional cgroups and their design
*** Swap (separate subsystem or merge with memory?)
*** Disk I/O (several proposed designs)
*** Network traffic classification
*** Freezer
*** Signaller
*** OOM Handler
** libcg - userspace explotation of control groups/resource management
*** Overview so far
*** Is kernel-based reclassification needed?
*** Real use-cases
*** Future directions

* Checkpoint/Restart (2:30pm-5pm)
** Documentation : Look at "See Also" section below
** Goals and expectations of this summit
*** identify, discuss and (if possible) agree on the general design
*** identify, discuss and (if possible) agree on the technical points
*** decide on priorities for different components (eg. high, medium, low) such that the final outcome is a practical road-map that would keep us busy for (at least) until the next OLS (though the "O" may change ;)
** What are the problems that the linux community can solve with the checkpoint/restart ?
** Preparing the kernel internals
*** How we implement it without affecting long term maintainability ?
*** What are the kernel subsystems, process resources and framework for CR ?
*** Which pieces to target first ?

The following technical points can be discussed during the mini-summit if we have time or later at the OLS.

** Checkpointing / Restarting
*** Reaching a quescient point - network, processes, aio, avoiding side effects of quiesce/revive
*** Checkpoint - signal handler ? syscall ? crfs ? process hierarchy, resource dependencies, system and process resources
*** Restarting - New binary format handler ? converting between formats (from older kernel to newer)
*** Notification to processes which explicitly wish to be notified about quiesce, checkpoint and restart - container state ? new signals ?
** Determining the userspace API - Posix 1003.1m ?
** Passing the kernel internal state to/from userspace - coredump like file ? swap per container ? netlinks, CR filesystem ? army of different call for the CR (proc, existing syscalls, ...)
** Hopefully we can continue to discuss in the next days and get a bit of a hackfest going during OLS :)

== Moderators ==

* Namespaces/containers: Serge Hallyn, Dave Hansen
* Cgroups and resource management: Paul Menage, Balbir Singh, Dhaval Giani
* Checkpoint/restart: Daniel Lezcano, Oren Laadan

== See also ==
* http://www.linuxsymposium.org/2008/cfp.php — OLS call for papers
* https://lists.linux-foundation.org/pipermail/containers/2008-January/009688.html
* http://openvz.org/pipermail/devel/2008-July/012891.html
* Checkpoint/Restart
** Zap : http://www.ncl.cs.columbia.edu/publications/usenix2007_fordist.pdf
** Metacluster : http://lxc.sourceforge.net/doc/ols2006/lxc-ols2006.pdf
** OpenVZ : [[Checkpointing and live migration]]
** Checkpoint/Restart technology : http://en.wikipedia.org/wiki/Application_checkpointing
** Virtual Servers and Checkpoint/Restart in Mainstream Linux : Sigops document
** Remote fork: http://www.cse.nd.edu/~dthain/courses/classconf/wowsys2004/talks/rfork.pdf
** Vmadump : http://bproc.sourceforge.net/c268.html
** Posix CR : http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi/0650/bks/SGI_Admin/CPR_OG/sgi_html/ch03.html
** An OS services overview : http://sw-eng.falls-church.va.us/itsg/P08V31.htm

[[Category: Containers]]
[[Category: Events]]

Containers/UBC discussion

2006-09-18T17:49:37Z

Balbir: Add accounting information update thoughts

; unified interface for mem, cpu, disk I/O
: It is still not clear whether we need unified interface.
: Having one syscall for setting values for different resources seems OK if leaving alone the meaning of the "value" notion.

; memory reclamation
: Pending to be implemented on top of BC.

; moving tasks across beancounters
: Required changes:
# saving bc on vma instead of mm
# can two threads in a process be in different BC contexts?
# changing mm->bc in set_bc_id().

; what is implied by the term "guarantee"
# container will be able to touch that number of pages - I think this one (Hansendc)
# container will be able to map that number of pages
# container will not be killed unless it touches that number of pages
# anything else

; make it possible to charge full user page to its allocator and keep it charged till unmapped

; Consider creating resource groups via the use of aggregation (aggregated BC)

== Top Level Design - thoughts on how to accomplish the goal ==
# Create a BC per thread group
# Associate a group of BC's with an aggregated BC (we can call this a BC resource group)
# Enable migration of tasks by charging and un-charging aggregated BC when a BC moves across from one aggregated BC to another
# Change set_bcid() to create aggregated BC's instead of BC's

== Accounting information ==
# Is it possible to merge vm_acct_memory() with the accounting information in beancounters?

[[Category:UBC]]

Containers/Guarantees for resources

2006-09-18T17:46:38Z

Balbir: Changed the algorithm to compile and give correct results

[[Category:UBC]]

This page describes how guarantees for resources can be implemented.

= How to guarantee a guarantee =
It's not obvious at the first glance, but ''there are only two ways of how a guarantee can be provided'':
# reserve desired amount in advance
# limit consumers to keep some amount free

The first way has the followong disadvantages:
; Reservation is impossible for certain resources
: such as CPU time, disk or network bandwidth and similar can not be just reserved as their amount instantly increases;

; Reserved amount is essentially a limit, but much more strict
: cutting off X megabytes from RAM implies that all the rest groups are limited in their RAM consumption;

; Reservation reduces containers density
: if one wants to run some identical containers, each requiring 100Mb on 1Gb system, reservations can be done for only 10 containers, and starting the 11th is impossible.

On the other hand, ''limiting'' of containers can provide guarantees for them as well.

= Providing a guarantee through limiting =
The idea of getting a guarantee is simple:

{{out|if any group <math>g_i</math> requires a <math>G_i</math> units of resource from <math>R</math> units available then limiting all the rest groups with <math>R - G_i</math> units provides a desired guarantee}}

For <math>N</math> groups in the system this implies solving a linear equation set to get limits <math>L_i</math> like this:

<center><math>
\begin{cases}
L_2 + L_3 + \ldots + L_N = R - G_1 \\
L_1 + L_3 + \ldots + L_N = R - G_2 \\
\ldots \\
L_1 + L_2 + \ldots + L_{N-1} = R - G_N \\
\end{cases}
</math></center>

In a matrix form this looks like

<center><math>
A L = G\;
</math></center>

where

<center><math>
A = \begin{bmatrix}
0 & 1 & 1 & \cdots & 1 & 1 \\
1 & 0 & 1 & \cdots & 1 & 1 \\
& & \cdots \\
1 & 1 & 1 & \cdots & 1 & 0 \\
\end{bmatrix}
,
L = \begin{bmatrix} L_1 \\ L_2 \\ \vdots \\ L_N \end{bmatrix}
,
G = \begin{bmatrix} R - G_1 \\ R - G_2 \\ \vdots \\ R - G_N \end{bmatrix}
</math></center>

and thus the solution is

<center><math>
L = A^{-1}G\;
</math></center>

Skipping boring calculations, the reverse matrix <math>A^{-1}\;</math> is

<center><math>
A^{-1} = \frac {1} {N - 1} \left( A - (N - 2) E \right)
</math></center>

This solutions looks huge, but the <math>L</math> vector is calculated in <math>O(N)</math> time:

<pre>
void calculate_limits(int N, int *g, int *l)
{
int sum;
int i;

if (N == 1) {
l[0] = R;
return;
}

sum = 0;
for (i = 0; i < N; i++)
sum += R - g[i];
for (i = 0; i < N; i++)
l[i] = (sum - (R - g[i]) - (N - 2) * (R - g[i]))/(N - 1);
}
</pre>

== Disadvantages of this approach ==

This approach has only one disadvantage: O(n) time needed to start a new container.

Containers/Guarantees for resources

2006-09-18T08:26:52Z

Balbir:

[[Category:UBC]]

This page describes how guarantees for resources can be implemented.

= How to guarantee a guarantee =
It's not obvious at the first glance, but ''there are only two ways of how a guarantee can be provided'':
# reserve desired amount in advance
# limit consumers to keep some amount free

The first way has the followong disadvantages:
; Reservation is impossible for certain resources
: such as CPU time, disk or network bandwidth and similar can not be just reserved as their amount instantly increases;

; Reserved amount is essentially a limit, but much more strict
: cutting off X megabytes from RAM implies that all the rest groups are limited in their RAM consumption;

; Reservation reduces containers density
: if one wants to run some identical containers, each requiring 100Mb on 1Gb system, reservations can be done for only 10 containers, and starting the 11th is impossible.

On the other hand, ''limiting'' of containers can provide guarantees for them as well.

= Providing a guarantee through limiting =
The idea of getting a guarantee is simple:

{{out|if any group <math>g_i</math> requires a <math>G_i</math> units of resource from <math>R</math> units available then limiting all the rest groups with <math>R - G_i</math> units provides a desired guarantee}}

For <math>N</math> groups in the system this implies solving a linear equation set to get limits <math>L_i</math> like this:

<center><math>
\begin{cases}
L_2 + L_3 + \ldots + L_N = R - G_1 \\
L_1 + L_3 + \ldots + L_N = R - G_2 \\
\ldots \\
L_1 + L_2 + \ldots + L_{N-1} = R - G_N \\
\end{cases}
</math></center>

In a matrix form this looks like

<center><math>
A L = G\;
</math></center>

where

<center><math>
A = \begin{bmatrix}
0 & 1 & 1 & \cdots & 1 & 1 \\
1 & 0 & 1 & \cdots & 1 & 1 \\
& & \cdots \\
1 & 1 & 1 & \cdots & 1 & 0 \\
\end{bmatrix}
,
L = \begin{bmatrix} L_1 \\ L_2 \\ \vdots \\ L_N \end{bmatrix}
,
G = \begin{bmatrix} R - G_1 \\ R - G_2 \\ \vdots \\ R - G_N \end{bmatrix}
</math></center>

and thus the solution is

<center><math>
L = A^{-1}G\;
</math></center>

Skipping boring calculations, the reverse matrix <math>A^{-1}\;</math> is

<center><math>
A^{-1} = \frac {1} {N - 1} \left( A - (N - 2) E \right)
</math></center>

This solutions looks huge, but the <math>L</math> vector is calculated in <math>O(N)</math> time:

<pre>
void calculate_limits(int N, int *g, int *l)
{
int sum;

sum = 0;
for (i = 0; i < N; i++)
sum += g[i];
for (i = 0; i < N; i++)
l[i] = sum - (R - g[i]) - (N - 2) * (R - g[i]);
}
</pre>

= Disadvantages of this approach =

This approach has the following disadvantages

# Lets consider initialization - When we create <math>n</math> groups initially, we need to spend <math>O(n^2)</math> time to assign guarantees.
# Everytime a limit or a guarantee changes, we need to recalculate guarantees to ensure that the change will not break any guarantees
# The same thing as stated above, when a resource group is created or deleted

This can lead to some instability, a change in one group propagates to all other groups.

Containers/UBC discussion

2006-09-13T18:34:46Z

Balbir:

; unified interface for mem, cpu, disk I/O
: It is still not clear whether we need unified interface.
: Having one syscall for setting values for different resources seems OK if leaving alone the meaning of the "value" notion.

; memory reclamation
: Pending to be implemented on top of BC.

; moving tasks across beancounters
: Required changes:
# saving bc on vma instead of mm
# can two threads in a process be in different BC contexts?
# changing mm->bc in set_bc_id().

; what is implied by the term "guarantee"
# container will be able to touch that number of pages - I think this one (Hansendc)
# container will be able to map that number of pages
# container will not be killed unless it touches that number of pages
# anything else

; make it possible to charge full user page to its allocator and keep it charged till unmapped

; Consider creating resource groups via the use of aggregation (aggregated BC)
''Top Level Design - thoughts on how to accomplish the goal''
# Create a BC per thread group
# Associate a group of BC's with an aggregated BC (we can call this a BC resource group)
# Enable migration of tasks by charging and un-charging aggregated BC when a BC moves across from one aggregated BC to another
# Change set_bcid() to create aggregated BC's instead of BC's