Remote console setup

In case you are experiencing a kernel crash (oops) and have already checked your hardware, you should report what kernel says to the console to Bugzilla. Sometimes kernel crashes so badly that syslogd is not working and what kernel says it never written to a file. If this is the case, you have to catch what kernel says. There are several ways possible.

Manual/Photo

If kernel backtrace is not long enough there are chances that it can fit into a single screen. In that case, you can just make a photo of the kernel crash screen and attach it to the bug report. If you do not have a camera, you still can carefully write down (using a piece of paper and a pen, that is) what you see on the screen, and later type it into the bug report.

Serial console

Here is a description of a common routine, that is necessary to set up a serial console.

First of all you should make sure that your node has a serial port. If there is no such port then unfortunately this way is not for you. Then you need to find second node with a serial port on it. This node will be used to collect logs from your primary mashine. Further you need to acquire so-called null modem cable and it must be long enough to connect two mashines in question.

Ok, suppose, that preliminary operations above are performed and you connect two machines. In your boot loader add the following kernel parameters:

console=ttyS0,11500 console=tty0

For example in GRUB boot loader it looks like such strings in /boot/grub/grub.conf:

title Fedora Core (2.6.16-026test014.1-smp)
        root (hd0,0)
        kernel /vmlinuz-2.6.16-026test014.1-smp ro root=LABEL=/ console=ttyS0,115200 console=tty debug silencelevel=8
        initrd /initrd-2.6.16-026test014.1-smp.img

Kernel loaded with such parameters will send all kernel messages to /dev/ttyS0 (serial port). Also be sure that your null modem cable is connected to appropriate port, if you have several ports.

On the second node you should turn on any software that can log from /dev/ttyS0. It can be usual

cat /dev/ttyS0 > /var/log/serial.log &

or something more sophisticated: syslogd, watchtty etc.

One more important thing. 115200 in the example above is the rate of emiting port. Also recieving port must work at equal rate. To tune ttyS0 rate use stty program:

stty 115200  < /dev/ttyS0

Netconsole

Kernel recompilation

If you use binary kernel from openvz.org, it already has netconsole module compiled in, so just skip to next section.

If you build the kernel yourself, you might need to check if netconsole is compiled. To that effect, change to your kernel source directory and grep your kernel .config for NETCONSOLE:

# cd /usr/src/openvz/linux-2.6.16
# grep NETCONSOLE .config

If you see nothing or "# CONFIG_NETCONSOLE is not set" you need to recompile the kernel.

Edit your kernel configuration file .config with a text editor (nano .config or vi .config). Set netconsole to Y or M (depending on whether you want it as a module or built into the kernel; I have compiled it as a module):

CONFIG_NETCONSOLE=m

Save the file, then recompile the kernel:

# make bzImage && make modules && make modules_install

Update your bootloader for the new updated kernel. In my case I use LILO so I just type lilo at the prompt.

Reboot into new kernel.

Setting up OpenVZ side

Next you want your netconsole to send the request to somewhere. Load netconsole module, specifying the remote server parameters:

# modprobe netconsole netconsole=4444@10.0.2.1/eth0,6666@10.0.2.2/00:05:5D:34:11:AF

This will load the module with your settings. Replace your local IP address with where 10.0.2.1 is, eth0 with your network interface card device, 6666 with the remote netconsole port (UDP), and 10.0.2.2 with your remote netconsole server IP. Also add in the mac address of your remote netconsole server, which in my case was 00:05:5D:34:11:AF. You can get the MAC address using arp utility:

# ping -c 1 10.0.2.2
# /sbin/arp -i 10.0.2.2
? (10.0.2.2) at 00:05:5D:34:11:AF [ether] on eth2

Netconsole documentation is available from Documentation/networking/netconsole.txt file under your kernel source directory.

Setting up remote side

Setup netcat on your console server to listen on port 6666 UDP:

netcat -u -l -p6666

When your kernel prints something on the console, the text will be also captured on this netconsole server.