Density is a characteristic which tells how many containers (CTs) or Virtual Machines (VMs) virtualization technology can run successfully on a given hardware. Density which very much depends on:

* workload itself. Virtualization is not a miracle and can't handle more high loaded workloads then bare host can. e.g. do not expectany density except for 1VM/CT if you are running Oracle with huge request rates. On the other hand, if you are having multiple small web sites virtualization can easily consolidate them on a single host as long as host is capable to handle IO requests altogether and memory/CPU demands.

Speaking about hypervisors, some of them (like Xen) reserve the whole guest VM memory on its startup and do not allow memory overcommitment. This leads to inability to start more then, say, 15 VMs with 1Gb RAM assigned to each on a 16Gb RAM host. Theoretically this helps to deliver maximum performance as Xen can always use large pages and avoid swapping out or other overheads. However, as other hypervisors (like ESX) demonstrate it's possible to achieve both goals - high performance and density depending on the situation, though it obviously requires more complicated technology which is not yet fully available in any of open-source hypervisors. Typically users also care for quality of service of their software, i.e. how well their services are working on utilized node and how fast they respond to external requests. The metrics of quality of service are average/min/max response time, 99.9% requests response time and similar. When a hardware node is capable to handle the load, these metrics either do not grow much with bigger number of containers, or grow linearly. When a hardware node becomes over utilized, these metrics typically start to degrade exponentially (due to memory swap out). Below we summarize differences between containers (OpenVZ, Parallels Containers, Solaris Zones) and hypervisor (ESX, Xen, KVM, HyperV) which affect density and make containers to be more suitable for high density environments then virtual machines.

* Containers do not reserve memory assigned to them. They exhibit real-time elastic behavior: if a container do not use memory, then it is free for use by other containers. In other words, overcommitment is as natural and easy as on a standalone Linux box, when multiple applications compete for memory resources. As a result, a simple container running ssh, apache, init and cron takes only 10-20Mb of physical RAM. Sure, for efficient caching of apache pages more RAM is needed, but it depends purely on active working set of web site and most frequently accessed files. If site is not accessed frequently kernel may decide to free memory at all (or swap out) and still will be able to get pages back in seconds. * Containers memory management is system-wide wise. If one container needs more physical RAM (e.g. for apache pages caching) and hardware node have no more memory available, kernel will automatically reclaim last recently used caches of other containers. This process is dynamic by it's nature and do not have any latency like ballooning in case of VMs. * Global memory management allows one containers to cache more data then RAM assigned to it. In case other containers will need physical memory this cache will be quickly reclaimed automatically. Thus caching can be much more efficient and dynamically adjusts itself to the needs. * Containers CPU scheduler is interactive system-wide. If some process is woken up by an external event (like networking request) CPU scheduler can quickly preempt current CPU hog task and schedule interactive one. This is not true with VMs where hypervisor doesn't know the nature of tasks inside the VM and whether VM should preempt another VM on even or not. In high-density environments it is particularly important to make sure that interactive tasks get CPU fast enough and provide low latency response to users.

* Memory (Most) memory is basically reserved on the host on VM start. KVM and XEN reserve the whole memory by default and do not allow memory overcommitment. As a result you can't run more then 15 VMs with 1GB RAM on 16Gb box. ESX, being the most advanced hypervisor, uses page sharing and ballooning technologies to introduce memory overcommitment. However, on practice it allows to get only about 2x times overcommitment on the guests of the same type. Our experiments tell demonstrate that half of the improvement is due to page sharing, and half is due to ballooning. * Multiple Ballooning helps to reclaim fairly easily part of guest VM memory, however it have some issues. First, it have some noticeable latency since it takes time from deciding to reclaim memory till the moment when guest will do that for hypervisor (and BTW it can simply have problems and do not do that at all). Next, it may lead to swap out inside the guest which negatively affects the whole system I/O performance (IOPS is a scarce resource on practice). And what is worse it may lead to behavior different from standalone machine - e.g. OS may deny memory allocations though guest didn't used assigned resources or trigger Out-Of-Memory (OOM) killer on Linux. * Typical hypervisor overhead is 50-100Mb for 1Gb 1VCPU guest, i.e. almost 10% of RAM. Detailed overhead table from VMware ESX 4.0 product [p.30 http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf] also demonstrates how overhead significantly increases with a number of VCPUs assigned to guest. * VMs are running multiple kernels and their system data structures in memory. RAM pages sharing helps this problem, but on VM startup when memory scanner have not yet merged multiple copies this can be a problem and ballooning help a bit here, toolead to memory usage bursts. * what happens when CT is above itHypervisor CPU scheduler doesn's limit* what happens when node RAM is exhausted* plots t know details about tasks running inside and examplestheir interactivity. Kir had http_load plot in the past. We will have LAMP results as wellSee above explanation on containers.