Checkpointing and live migration

CPT is an extension to the OpenVZ kernel which can save the full state of a running VE and to restore it later on the same or on a different host in a way transparent to running applications and network connections. This technique has several applications, the most important being live (zero-downtime) migration of VEs and taking an instant snapshot of a running VE for later resume, i.e. CheckPoinTing.

Before CPT, it was only possible to migrate a VE through a shutdown and subsequent reboot. The procedure not only introduces quite a long downtime for network services, it is not transparent for clients using the VE, making migration impossible when clients run some tasks which are not tolerant to shutdowns.

Compared with this old scheme, CPT allows migration of a VE in a way which is essentially invisible both for users of this VE and for external clients using network services located inside the VE. It still introduces a short delay in service, required for actual checkpoint/restore of the processes, but this delay is indistinguishable from a short interruption of network connectivity.

Online migration

There is a special utility vzmigrate in the OpenVZ distribution intended to support VE migration. With its help one can perform live (a.k.a. online) migration, i.e. during migration the VE “freezes” for some time, and after migration it continues to work as though nothing had happened. Online migration can be performed with the

vzmigrate --online <host> VEID

command. During online migration all VE private data saved to an image file, which is transferred to the target host.

In order for vzmigrate to work without asking for a password, ssh public keys from the source host should be placed in the destination host's /root/.ssh/authorized_keys file. In other words, command ssh root@host should not ask you for a password. See ssh keys for more info.

Manual Checkpoint and Restore Functions

vzmigrate is not strictly required to perform online migration. The vzctl utility, accompanied with some file system backup tools, provides enough power to do all the tasks.

A VE can be checkpointed with command:

vzctl chkpnt VEID --dumpfile <path>

This command saves all the state of a running VE to the dump file and stops the VE. If the option --dumpfile is not set, vzctl uses a default path /vz/dump/Dump.VEID.

After this it is possible to restore the VE to the same state executing:

vzctl restore VEID --dumpfile <path>

If the dump file and file system is transferred to another HW node, the same command can restore the VE there with the same success.

It is a critical requirement that file system at the moment of restore must be identical to the file system at the moment of checkpointing. If this requirement is not met, depending on the severity of changes, the process of restoration can be aborted or the processes inside a VE can see this as an external corruption of open files. When a VE is restored on the same node where it was checkpointed, it is enough to not touch the file system accessible by the VE. When a VE is transferred to another node it is necessary to synchronize the VE file system before restore. vzctl does not provide this functionality and external tools (i.e. rsync) are required.

Step-by-step Checkpoint and Restore

The process of checkpointing can be performed in stages. It consists of three steps.

First step – suspend the VE. At this stage CPT moves all the processes to a special beforehand known state and stops VE network interfaces. This stage can be done with the

vzctl chkpnt VEID --suspend

command.

Second step – dumping VE. At this stage CPT saves the state of processes and global state of VE to an image file. All the process private data needs to be saved: address space, register set, opened files/pipes/sockets, System V IPC structures, current working directory, signal handlers, timers, terminal settings, user identities (uid, gid, etc), process identities (pid, pgrp, sid, etc), rlimit and other data. This stage can be done with the

vzctl chkpnt VEID --dump --dumpfile <path>

command.

Third step – killing or resuming processes. If the migration succeeds the VE can be stopped with the command:

vzctl chkpnt VEID --kill

If migration failed for some reason or if the goal was taking a snapshot of the VE state for later restore, CPT can resume the VE with:

vzctl chkpnt VEID --resume

The process of restoring consists of two steps.

The first step is to restore processes and to leave them in a special frozen state. After this step processes are ready to continue execution, however, in some cases CPT has to do some operations after a process is woken up, therefore CPT sets process return point to function in our module. This stage can be done with the

vzctl restore VEID --undump --dumpfile <path>

command.

Second step – waking up processes or killing them if the restore process failed. After CPT wakes up process, it performs necessary operations in our function and continues execution. This stage can be done with the

vzctl restore VEID --resume
vzctl restore VEID --kill

commands.