6,534
edits
Changes
incorporated feedback from dev@
= OpenVZ Linux Containers technology whitepaper =
<s><big><big><b>{{PDFlink|[[Media:What are containers.pdf|Download in PDF]]}}</b></big></big></s>
OpenVZ is an open source virtualization technology for Linux that enables the partitioning of a single physical Linux machine into multiple smaller independent units called containers.
A namespace is an abstract environment created to hold a logical grouping of unique identifiers or symbols (i.e., names). An identifier defined in a namespace is associated with that namespace. The same identifier can be independently defined in multiple namespaces. An example of namespace is a directory on a file system. This allows two files with the same name to be stored on the same device as long as they are stored in different directories.
For OpenVZ, Linux kernel namespaces are used as containers building blocks. A simple case As an example of namespace usage, let's take a namespace is look at chroot.
[[Image:Chroot.png|right|400px]]
Traditional UNIX <code>chroot()</code> system call is used to change the root of the file system of a calling process to a particular directory. That way it limits the scope of file system for the process, so it can only see and access a limited sub tree of files and directories.
Chroot is still used for application isolation (although, unlike container, it does not provide full isolation— an application can escape from chroot under certain circumstances).
Chroot is also used by containers, so a container filesystem is just a directory on the host. Consequences are:
=== Other namespaces ===
OpenVZ builds on a chroot idea and expands it to everything else that applications have. In other words, every API that kernel provides to applications are is “namespaced”, making sure every container have its own isolated subset of a resource. Examples include:
* '''File system namespace''': this one is <code>chroot()</code> itself, making sure containers can not see each other's files.
* '''Process ID namespace''': this is so that every container processes has its own unique process IDs, and the first process inside a container has a PID of 1 (it is usually <code>/sbin/init</code> process which actually relies on its PID to be 1). For every process in container, its PID in container is different from the one at host. Containers can only see their own processes, and they can't see (or access in any way, like sending signals) processes in other containers.
* '''IPC namespace''': this is so that every container has its own System V IPC (Inter-Process Communication (IPC) shared memory segments, semaphores, and messages. For example, <code>ipcs</code> output is different in every container.
* '''Networking namespace''': this is so that every container has its own network devices, IP addresses, routing rules, firewall (iptables) rules, network caches and so on. See more details below at [[#Networking]].
* '''<code>/proc</code> and <code>/sys</code> namespaces''': this is so that every container to have its own representation of <code>/proc</code> and <code>/sys</code> — special filesystems used to export some kernel information to applications. In a nutshell, those are subsets of what a physical Linux host system have.
* '''UTS namespace''': this is so that every container can have its own hostname.
* and so on.
Note that memory and CPU need not be namespaced: existing virtual memory and multitask mechanisms address this.
To put it simply, a container is a sum of all its namespaces. Therefore, there is only one single OS kernel running, on top of it there are multiple isolated containers, sharing that single kernel.
Single kernel approach is much more light-weight than traditional VM-style virtualization(for more differences between CT and VM, see [[../Containers vs VMs/]]). The consequences of having only one kernel are:
The consequences of having only one kernel are: # A container can only run the same OS as the host, i.e. Linux in case of OpenVZ. Nevertheless, multiple different Linux distributions can be used in different containers. For example, RHEL4, RHEL5, RHEL6, Fedora 14 and Ubuntu 10.10 can run inside different containers on the same host system (running e.g. Gentoo).# Waiving the need to run multiple OS kernels leads to '''higher density''' of containers (compared to VMs). Practically that means that a few hundreds of typical containers can be started on a conventional notebook. # Software stack that lies in between the hardware and an end-user application is as thin as in usual non-virtualized system (see the image), this means higher native performance of containers and no virtualization overhead (compared to VMs). See more at [[# A container can only run the same OS as the host, i.e. Linux in case of OpenVZ. Nevertheless, multiple different Linux distributions can be used in different containersPerformance and density]] below.
== Resource management ==
It is important to understand that resources are not pre-allocated, they are just limited. That means:
* all the resources can be changed dynamically (run-time);
* if a resource is not used, it it is availablefor other containers, which makes resource overcommitting easy.
=== CPU ===
* '''CPU mask''': tells the kernel the exact CPUs that can be used to run this container on. This can also be used as a CPU limiting factor, and helps performance on a non-uniform memory (NUMA) systems.
* '''VCPU affinity''': tells the kernel a maximum number of CPUs a container can use. The difference from the previous option is you are not able to specify the exact CPUs, only the number of those, and then the kernel dynamically assigns / adjusts CPUs between containers based on current load.
=== Disk ===
* '''Disk space'''. In a default setup, all containers reside on the same hard drive partition (since a container is just a subdirectory). OpenVZ introduces a per-container disk space limit to control disk usage. So, to increase the disk space available to a container, one just needs to increase that limit -- dynamically, on the fly, without a need to resize a partition or a filesystem.
* '''Disk I/O priority'''. Containers compete for I/O operations, and can affect each other if they use the same disk drive. OpenVZ introduces a per-container I/O priority, which can be used to e.g. decrease the "bad guy" I/O rate in order to not trash the other containers.
* '''Disk I/O bandwidth'''. I/O bandwidth (in bytes per second) can be limited per-container (currently only available in commercial Parallels Virtuozzo Containers).
== Checkpointing and live migration ==
'''Checkpointing''' is an OpenVZ kernel feature that makes it able possible to freeze a running container (i.e. pause all its processes) and dump its complete in-kernel state into a file on disk. Such a dump file contains everything about processes inside a container: their memory, opened files, network connections, states etc. Then, a running container can be restored from the dump file and continue to run normally. The concept is somewhat similar to suspend-to-disk, only for a single container and much faster.
A container can be restored from a dump file on a different physical server, opening the door for live migration.
If container is residing on a shared storage (like NFS or SAN) there is no need to copy its files, one just checkpoints the container on one system and restores it on another.
Network connections are fully preserved and migrated. Upon finishing migration, destination server sends an ARP announce telling that this container IP address is now lives on this a new MAC. While the container is frozen, all the incoming packets for it are dropped. In case of TCP, such packets will be retransmitted by the sending side, while in case of UDP packets are supposed to be lost sometimes.
Unlike other containers functionality, which is architecture-agnostic (and therefore containers on ARM or MIPS are easy to have), checkpointing is architecture-dependent. It is currently supported for x86, x86_64 (previously it was also supported on , and IA64).
== Miscellaneous topics ==
=== Networking ===
Each container have has their own network stack. This includes network device(s), routing table, firewall rules (iptables), network caches, hash tables, etc. From the perspective of container owner it looks like a standalone Linux box.
Three major modes of operation are possible.
==== Route-based (venet) ====
This mode works in Layer 3 (network layer) of [[w:OSI model]]. That means that a container have a MAC-less network device (called <code>venet</code>), with one end in container and another end in the host system acting . Host system then acts as a router. Each IP packet is traversing both host and container's IP stack.
The major features of this setup are:
* '''Host system acts as a router'''* '''High security'''. It's the host system administrator who specifies container IP(s) and routing rule(s). No spoofing or harming is possible.
* '''High control'''. Host system administrator fully controls container networking, by means of routing, firewall, traffic shaper etc.
* '''NOARPNo MAC address'''. A container can not use broadcasts or multicasts (since these features are on Level 2 and require a MAC address).
==== Bridge-based (veth) ====
This mode works in OSI Layer 2(data link layer). For container, a Virtual Ethernet (<code>veth</code>) device is used. This device can be thought of as a pipe with two ends -- one end in the host system and another end in a CT, so if a packet goes to one end it will come out from the other end. The host system acts as a bridge, so veth is usually bridged together with eth0 or similar interface.
The major features of this setup are:
* '''Host system acts as a bridge'''
* '''High configurability''': container administrator can setup all the networking.
* '''Ability to use broadcasts/multicasts'''.* '''DHCP''' and dynamic IP addresses are possible* Broadcasting have negative performance impact (it is delivered separately to each CT)*
==== Real network device in a container ====
Host system administrator can move assign a network device (such as <code>eth1</code>) into a container. Container administrator can then manage it as usual.
Major features are:
* Low security
* Container is tied to hardware
=== Performance and density ===
=== Limitations ===
From the point of view of a container owner, it looks and feels like a real system. Nevertheless, it is important to understand what are container limitations:
* Container can't boot/use its own kernel (it uses host system kernel).
* Container can't load its own kernel modules (it uses host system kernel modules).
* Container By default, container can't set system time, unless . Such permission should be explicitly configured to do so (say to run <code>ntpd</code> in a CT)granted by host system administrator.
* Container By default, container does not have direct access to hardware such as hard drive, network card, or a PCI device. Such access can be granted by host system administrator if needed.
=== OpenVZ host system scope ===
