WP/Containers density - Revision history

Sergey Bronnikov: Parallels Containers -> Virtuozzo

2015-05-08T11:38:34Z

Parallels Containers -> Virtuozzo

Dev at 14:26, 11 April 2011

2011-04-11T14:26:42Z

Dev at 14:12, 11 April 2011

2011-04-11T14:12:01Z

Dev at 13:24, 11 April 2011

2011-04-11T13:24:56Z

Dev at 12:33, 3 April 2011

2011-04-03T12:33:04Z

Kir: don't do TOC

2011-03-16T22:02:29Z

don't do TOC

Kir: multiple wording fixes

2011-03-15T14:30:09Z

multiple wording fixes

Kir: minor formatting fixes

2011-03-15T14:20:11Z

minor formatting fixes

Kir: moved WP/Containers Density to WP/Containers density

2011-03-15T14:18:11Z

moved WP/Containers Density to WP/Containers density

Dev at 18:36, 14 March 2011

2011-03-14T18:36:00Z

@@ Line 13: / Line 13: @@
 Typically users also care for quality of service of their software, i.e. how well their services are working on utilized node and how fast they respond to external requests. The metrics of quality of service are average/min/max response time, 99.9% requests response time and similar. When a hardware node is capable to handle the load, these metrics either do not grow much with bigger number of containers, or grow linearly. When a hardware node becomes over utilized, these metrics typically start to degrade exponentially (due to memory swap out).
-Below we summarize differences between containers (OpenVZ, Parallels Containers, Solaris Zones) and hypervisor (ESX, Xen, KVM, HyperV) which affect density and make containers to be more suitable for high density environments then virtual machines.
+Below we summarize differences between containers (OpenVZ, Virtuozzo, Solaris Zones) and hypervisor (ESX, Xen, KVM, HyperV) which affect density and make containers to be more suitable for high density environments then virtual machines.
 == What makes containers to be perfectly suitable for high density? ==
@@ Line 25: / Line 25: @@
 * Containers CPU scheduler is interactive system-wide. If some process is woken up by an external event (like networking request) CPU scheduler can quickly preempt current CPU hog task and schedule interactive one. This is not true with VMs where hypervisor doesn't know the nature of tasks inside the VM and whether VM should preempt another VM on even or not. In high-density environments it is particularly important to make sure that interactive tasks get CPU fast enough and provide low latency response to users.
-* Parallels Virtuozzo Containers product goes a step further by introducing container templates, used as a basis for all containers, and a special copy-on-write filesystem, which makes sure that original template is kept untouched and container gets its own private copy of template file when it tries to modify it. As a result all common files are shared across the containers and present on disk and in memory caches in a single instance. This saves memory, reduces I/O and makes L2/L3 caches work more effective.
+* Virtuozzo product goes a step further by introducing container templates, used as a basis for all containers, and a special copy-on-write filesystem, which makes sure that original template is kept untouched and container gets its own private copy of template file when it tries to modify it. As a result all common files are shared across the containers and present on disk and in memory caches in a single instance. This saves memory, reduces I/O and makes L2/L3 caches work more effective.
 == Why Virtual Machines are not that good? ==

← Older revision		Revision as of 14:26, 11 April 2011
Line 40:		Line 40:

	* Hypervisor CPU scheduler doesn't know details about tasks running inside and their interactivity. See above explanation on containers.		* Hypervisor CPU scheduler doesn't know details about tasks running inside and their interactivity. See above explanation on containers.
		+
		+	== Numbers ==
		+	XXX

@@ Line 4: / Line 4: @@
 == What is density? ==
-Density is a characteristic which tells how many containers (CTs) or Virtual Machines (VMs) virtualization technology can run successfully on a given hardware. Density very much depends on:
+Density is a characteristic which tells how many containers (CTs) or Virtual Machines (VMs) virtualization technology can run successfully on a given hardware which very much depends on:
 * virtualization technology used (containers or hypervisor);
 * features provided by the technology (page sharing, ballooning, memory overcommitment support etc.);
-* workload itself. Virtualization is not a miracle and can't handle more high loaded workloads then bare host can. e.g. do not expect
+* workload itself. Virtualization is not a miracle and can't handle more high loaded workloads then bare host can. e.g. do not expect any density except for 1VM/CT if you are running Oracle with huge request rates. On the other hand, if you are having multiple small web sites virtualization can easily consolidate them on a single host as long as host is capable to handle IO requests altogether and memory/CPU demands.
-Typically users also care for quality of service of their software, i.e. how well their services are working on utilized node and how fast they respond to external requests. The metrics of quality of service are average/min/max response time, 99.9% requests response time and similar. When a hardware node is capable to handle the load, these metrics either do not grow much with bigger number of containers, or grow linearly. When a hardware node becomes over utilized, these metrics typically start to degrade exponentially (due to memory swap out).
+Speaking about hypervisors, some of them (like Xen) reserve the whole guest VM memory on its startup and do not allow memory overcommitment. This leads to inability to start more then, say, 15 VMs with 1Gb RAM assigned to each on a 16Gb RAM host. Theoretically this helps to deliver maximum performance as Xen can always use large pages and avoid swapping out or other overheads. However, as other hypervisors (like ESX) demonstrate it's possible to achieve both goals - high performance and density depending on the situation, though it obviously requires more complicated technology which is not yet fully available in any of open-source hypervisors.
 == What makes containers to be perfectly suitable for high density? ==
-* Containers memory management is system-wide wise. If one container needs more physical RAM (e.g. for apache pages caching) and hardware node have no more memory available, kernel will automatically reclaim last recently used caches of other containers.
+* Containers do not reserve memory assigned to them. They exhibit real-time elastic behavior: if a container do not use memory, then it is free for use by other containers. In other words, overcommitment is as natural and easy as on a standalone Linux box, when multiple applications compete for memory resources. As a result, a simple container running ssh, apache, init and cron takes only 10-20Mb of physical RAM. Sure, for efficient caching of apache pages more RAM is needed, but it depends purely on active working set of web site and most frequently accessed files. If site is not accessed frequently kernel may decide to free memory at all (or swap out) and still will be able to get pages back in seconds.
 * Parallels Virtuozzo Containers product goes a step further by introducing container templates, used as a basis for all containers, and a special copy-on-write filesystem, which makes sure that original template is kept untouched and container gets its own private copy of template file when it tries to modify it. As a result all common files are shared across the containers and present on disk and in memory caches in a single instance. This saves memory, reduces I/O and makes L2/L3 caches work more effective.
@@ Line 23: / Line 31: @@
 Hypervisors (Xen, ESX, KVM) are not that good on high density scenarios. There are multiple reasons for that:
-* Memory is basically reserved on the host on VM start. KVM and XEN reserve the whole memory by default and do not allow memory overcommitment. As a result you can't run more then 15 VMs with 1GB RAM on 16Gb box. ESX, being the most advanced hypervisor, uses page sharing and ballooning technologies to introduce memory overcommitment. However, on practice it allows to get only about 2x times overcommitment on the guests of the same type. Our experiments tell that half of the improvement is due to page sharing, and half is due to ballooning.
+* (Most) memory is basically reserved on the host on VM start. KVM and XEN reserve the whole memory by default and do not allow memory overcommitment. As a result you can't run more then 15 VMs with 1GB RAM on 16Gb box. ESX, being the most advanced hypervisor, uses page sharing and ballooning technologies to introduce memory overcommitment. However, on practice it allows to get only about 2x times overcommitment on the guests of the same type. Our experiments demonstrate that half of the improvement is due to page sharing, and half is due to ballooning.
-* Multiple kernels and their system data structures in memory. RAM pages sharing and ballooning help a bit here, too.
-* what happens when CT is above it's limit
+* Ballooning helps to reclaim fairly easily part of guest VM memory, however it have some issues. First, it have some noticeable latency since it takes time from deciding to reclaim memory till the moment when guest will do that for hypervisor (and BTW it can simply have problems and do not do that at all). Next, it may lead to swap out inside the guest which negatively affects the whole system I/O performance (IOPS is a scarce resource on practice). And what is worse it may lead to behavior different from standalone machine - e.g. OS may deny memory allocations though guest didn't used assigned resources or trigger Out-Of-Memory (OOM) killer on Linux.
-* what happens when node RAM is exhausted
-* plots and examples. Kir had http_load plot in the past. We will have LAMP results as well.
+* Typical hypervisor overhead is 50-100Mb for 1Gb 1VCPU guest, i.e. almost 10% of RAM. Detailed overhead table from VMware ESX 4.0 product [p.30 http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf] also demonstrates how overhead significantly increases with a number of VCPUs assigned to guest.

@@ Line 3: / Line 3: @@
 == What is density? ==
-Density is a characteristic which tells how many containers (CTs) or Virtual Machines (VMs) virtualization technology can run successfully on a given hardware. Typically users also care for quality of service of their software, i.e. how well their services are working on utilized node and how fast they respond to external requests. Typical metrics of quality of service are average response time, 99.9% requests response time, and max response time. When a hardware node is capable to handle the load, these metrics either do not grow much with bigger number of containers, or grow linearly. When a hardware node becomes over utilized, these metrics typically start to degrade exponentially (due to memory swap out).
 == What makes containers to be perfectly suitable for high density? ==

@@ Line 17: / Line 17: @@
 * Memory is basically reserved on the host on VM start. KVM and XEN reserve the whole memory by default and do not allow memory overcommitment. As a result you can't run more then 15 VMs with 1GB RAM on 16Gb box. ESX, being the most advanced hypervisor, uses page sharing and ballooning technologies to introduce memory overcommitment. However, on practice it allows to get only about 2x times overcommitment on the guests of the same type. Our experiments tell that half of the improvement is due to page sharing, and half is due to ballooning.
 * Multiple kernels and their system data structures in memory. RAM pages sharing and ballooning help a bit here, too.
 * what happens when CT is above it's limit
 * what happens when node RAM is exhausted
 * plots and examples. Kir had http_load plot in the past. We will have LAMP results as well.

← Older revision		Revision as of 22:02, 16 March 2011
Line 1:		Line 1:
		+	__NOTOC__
	= Containers density. Why containers are so good at it? =		= Containers density. Why containers are so good at it? =

@@ Line 5: / Line 5: @@
 == What makes containers to be perfectly suitable for high density? ==
-# Containers do not reserve memory assigned to it. They exhibit real-time elastic behavior, if applications do not use memory, then it is free for other containers usage. i.e. overcommit is natural and easy like on standalone Linux box when multiple applications compete for memory resources.
+* Containers do not reserve memory assigned to it. They exhibit real-time elastic behavior, if applications do not use memory, then it is free for other containers usage. i.e. overcommit is natural and easy like on standalone Linux box when multiple applications compete for memory resources. As a result simple container running ssh, apache, init and cron takes only 10-20Mb of physical RAM. Sure, for efficient caching of apache pages more RAM is needed.
-# Containers memory management is system-wide wise. If one container needs more physical RAM (e.g. for apache pages caching) and hardware node have no more memory available, kernel will automatically reclaim last recently used caches of other containers.
+* Containers memory management is system-wide wise. If one container needs more physical RAM (e.g. for apache pages caching) and hardware node have no more memory available, kernel will automatically reclaim last recently used caches of other containers.
-# Parallels Virtuozzo containers product does a step further and introduces containers templates, which are used as a basis for all containers and special Copy-on-Write filesystem makes sure that original template is kept untouched and container gets its own private copy of template file when it tries to modify it. As a result all common files are shared across the containers and present on disk and in memory caches in a single instance. This saves memory, reduces I/O and makes L2/L3 caches work more effective.
+* Parallels Virtuozzo containers product does a step further and introduces containers templates, which are used as a basis for all containers and special Copy-on-Write filesystem makes sure that original template is kept untouched and container gets its own private copy of template file when it tries to modify it. As a result all common files are shared across the containers and present on disk and in memory caches in a single instance. This saves memory, reduces I/O and makes L2/L3 caches work more effective.
 == Why Virtual Machines are not that good? ==
 Hypervisors (Xen, ESX, KVM) are not that good on high density scenarios. There are multiple reasons for that:
-# Multiple kernels and their system data structures in memory.
+* Memory is basically reserved on the host on VM start. e.g. KVM and XEN by default reserve whole memory and do not allow memory overcommitment. As a result you can't run more then 15 VMs with 1GB RAM on 16Gb box. ESX as the most advanced hypervisor uses page sharing and ballooning technologies to introduce memory overcommitment. However, on practice it allows to get only about 2x times overcommitment on the guests of the same type. From our experiments half of this improvement is due to page sharing and half due to ballooning.
-# what happens when CT is above it's limit
+* Multiple kernels and their system data structures in memory. There are some technologies like RAM pages sharing and ballooning which help a bit.
-# what happens when node RAM is exhausted
-# plots and examples. Kir had http_load plot in the past. We will have LAMP results as well.
+* what happens when CT is above it's limit

← Older revision	Revision as of 14:18, 15 March 2011
(No difference)