Building LKM against OpenVZ kernel from RPM

Sometimes it is necessary to build LKMs (LKM - Linux Kernel Module) against OpenVZ kernels. If you have used RPM package for installing OpenVZ kernel and your kernel is not native for your distribution, you probably will expirience troubles. As a native distribution I designate the distribution, where OpenVZ kernel team builds the kernel. We have here the following policy: rhel4-based kernels are build on rhel4/centos4 distributions, rhel5-based kernels are build on rhel5/centos5 distributions. So, for example, the situation when you install rhel5-based OpenVZ kernel on rhel5/centos5 distribution considered to be kernel on a native distribution. But if you install rhel5-base OpenVZ kernel ont rhel4/centos4, then you have kernel on not native distriubution.

Mainstream-based OpenVZ kernels are not guaranteed to be build on some definite distribution, so there is no permanent predefined native distribution for such kernels.

Here is an example of errors the user obtain trying to build LKM against OpenVZ kernel on not native distribution:

30855 Floating point exception| scripts/genksyms/genksyms -a x86_64 > /usr/local/src/zaptel-1.2.17.1/.tmp_zaptel-base.ver
/bin/sh: line 1:   527 Floating point exception scripts/basic/fixdep 
/bin/sh: line 1:  2232 Floating point exception scripts/mod/modpost -m -a -i

In this article I describe the reason of this problem and how to workaround it. Thank you to Mike Holloway for revealing the problem.

The Reason

In order to build LKM you need kernel headers and some other information from kernel sources. This information is located in files in -devel package for rhel5-based kernels and is incorporated in rhel4-based kernel packages. In addition to headers these packages content some executable binaries in scripts subdirectory of build tree:

$ rpm -qlp ovzkernel-2.6.9-023stab044.4.x86_64.rpm | grep scripts
/lib/modules/2.6.9-023stab044.4/build/scripts
/lib/modules/2.6.9-023stab044.4/build/scripts/.bin2c.cmd
/lib/modules/2.6.9-023stab044.4/build/scripts/.conmakehash.cmd
/lib/modules/2.6.9-023stab044.4/build/scripts/.kallsyms.cmd
...
/lib/modules/2.6.9-023stab044.4/build/scripts/basic/docproc                     # << binary executable
/lib/modules/2.6.9-023stab044.4/build/scripts/basic/docproc.c
/lib/modules/2.6.9-023stab044.4/build/scripts/basic/fixdep                      # << binary executable
/lib/modules/2.6.9-023stab044.4/build/scripts/basic/fixdep.c
/lib/modules/2.6.9-023stab044.4/build/scripts/basic/split-include               # << binary executable
...
...                                                                             # << other binary executable
...
$ rpm -qlp ovzkernel-devel-2.6.18-8.1.3.el5.028stab033.1.x86_64.rpm | grep scripts
...
/usr/src/kernels/2.6.18-8.1.3.el5.028stab033.1-x86_64/scripts
/usr/src/kernels/2.6.18-8.1.3.el5.028stab033.1-x86_64/scripts/.bin2c.cmd
/usr/src/kernels/2.6.18-8.1.3.el5.028stab033.1-x86_64/scripts/.conmakehash.cmd
/usr/src/kernels/2.6.18-8.1.3.el5.028stab033.1-x86_64/scripts/.gitignore
/usr/src/kernels/2.6.18-8.1.3.el5.028stab033.1-x86_64/scripts/.kallsyms.cmd
/usr/src/kernels/2.6.18-8.1.3.el5.028stab033.1-x86_64/scripts/basic/docproc     # << binary executable
/usr/src/kernels/2.6.18-8.1.3.el5.028stab033.1-x86_64/scripts/basic/docproc.c
/usr/src/kernels/2.6.18-8.1.3.el5.028stab033.1-x86_64/scripts/basic/fixdep      # << binary executable
/usr/src/kernels/2.6.18-8.1.3.el5.028stab033.1-x86_64/scripts/basic/fixdep.c
...
...                                                                             # << other binary executable
...

These executables are created while building kernel package on our build system. And they are linked to libc installed on the build system! Now imagine that you've installed OpenVZ kernel package based on rhel5 kernel, say, on rhel4 distribution. The binaries mentioned above are also installed, but they can't run on this node, because other libc is installed on this system!

Workaround

The clean way to solve the problem is to build the kernel from sources. Then all binaries will be linked with the libraries that are on your node and no problems will arise.

A dirty way to get past the problem is to replace all the scripts with ones from a distribution that matches your installed system. At a minimum, you will probably need to replace the files in these directories:

/usr/src/kernels/2.6.18-8.el5.028stab031.1-x86_64/scripts/genksyms
/usr/src/kernels/2.6.18-8.el5.028stab031.1-x86_64/scripts/basic
/usr/src/kernels/2.6.18-8.el5.028stab031.1-x86_64/scripts/mod

External links

• OpenVZ Bug #586