6,534
edits
Changes
rm translate tags
In case you are experiencing a kernel crash ([[oops]]) and have already [[Troubleshooting:Hardwaretesting|checked tested your hardware]], you should report what kernel says to the console (i.e. an [[oops]] text) to [httphttps://bugzillabugs.openvz.org/ Bugzillabug tracker]. Sometimes kernel crashes so badly that <tt>syslogd </tt> is not working and what kernel says it is never written to a file. If this is the case, you have to catch what kernel says. There are several ways possible. == KDump == With RHEL6-based servers, kdump is pre-configured. See http://kb.odin.com/en/10044 to check the configuration. Dumps can be found under <code>/var/crash/</code> directory.
== Manual/Photo ==
== Serial console ==
== Netconsole ==
=== Kernel recompilation ===
If you use binary kernel from openvz.org, it already has netconsole module compiled in, so just skip to next section.
If you build the kernel yourself, you might need to check if netconsole is compiled. To that effect, change to your kernel source directory and grep your kernel <tt>.config</tt> for NETCONSOLE:
<pre>
# cd /usr/src/openvz/linux-2.6.16
# grep NETCONSOLE .config
</pre>
If you see nothing or "# CONFIG_NETCONSOLE is not set" you need to recompile the kernel.
Edit your kernel configuration file <tt>.config</tt> with a text editor (<tt>nano .config</tt> or <tt>vi .config</tt>). Set netconsole to Y or M (depending on whether you want it as a module or built into the kernel; I have compiled it as a module):
<pre>CONFIG_NETCONSOLE=m</pre>
Save the file, then recompile the kernel:
# make bzImage && make modules && make modules_install
Update your bootloader for the new updated kernel. In my case I use LILO so I just type lilo at the prompt.
Reboot into new kernel.
=== Setting up OpenVZ side ===
Next you want your netconsole to send the request to somewhere. Load netconsole module, specifying the remote server parameters:
# modprobe netconsole netconsole=4444@10.0.2.1/eth0,6666@10.0.2.2/00:05:5D:34:11:AF
This will load the module with your settings. Replace your local IP address with where <tt>10.0.2.1</tt> is, <tt>eth0</tt> with your network interface card device, <tt>6666</tt> with the remote netconsole port (UDP), and <tt>10.0.2.2</tt> with your remote netconsole server IP. Also add in the mac address of your remote netconsole server, which in my case was 00:05:5D:34:11:AF. You can get the MAC address using arp utility:
<pre>
# ping -c 1 10.0.2.2
# /sbin/arp -n 10.0.2.2
Address HWtype HWaddress Flags Mask Iface
10.0.2.2 ether 00:05:5D:34:11:AF C eth0
</pre>
If the remote netconsole server is outside of local network area, use mac address of default gateway or router on local network area and IP address of remote netconsole server (loging via WAN). Mac address of default gateway or router you can get the same way (ping to gateway/router and see mac address via arp command).
Netconsole documentation is available from <tt>Documentation/networking/netconsole.txt</tt> file under your kernel source directory.
=== Setting from initrd ===
To log the boot process before root filesystem is mounted, network device driver and netconsole modules must be loaded from initd.
RedHat 5/CentOS 5:
echo 'MODULES+="<network-driver-module> netconsole "' > /etc/sysconfig/mkinitrd/netconsole
chmod +x /etc/sysconfig/mkinitrd/netconsole
echo 'options netconsole netconsole=<sport>@<saddr>/<dev>,<dport>@<daddr>/<dmac>' >> /etc/modprobe.conf
Debian/Ubuntu:
echo '<network-driver-module>' >> /etc/initramfs-tools/modules
echo 'netconsole netconsole=<sport>@<saddr>/<dev>,<dport>@<daddr>/<dmac>' >> /etc/initramfs-tools/modules
and rebuild initrd.
=== Setting up rsyslogd ===
/etc/rsyslog.d/netconsole.conf
$template NetconsoleFile,"/var/log/netconsole/%FROMHOST%-%$NOW%.log"
$template NetconsoleFormat,"%rawmsg%"
$EscapeControlCharactersOnReceive off
$DropTrailingLFOnReception off
$RepeatedMsgReduction off
$RuleSet NetconsoleRuleset
*.* ?NetconsoleFile;NetconsoleFormat
$RuleSet RSYSLOG_DefaultRuleset
$ModLoad imudp
$InputUDPServerBindRuleset NetconsoleRuleset
$UDPServerRun 6666
=== Setting up remote side ===
Set up '''netcat''' ('''nc''' on some Linux distributions) on your console server to listen on port 6666 UDP:
netcat -u -l -p6666
or
nc -lu 6666
or
socat udp-listen:6666,reuseaddr -
When your kernel prints something on the console, the text will be also captured on this netconsole server.
==== Adding to inittab ====
For automatic care about capturing on console server you can use init respawn feature in this way:
echo "n1:23:respawn:/bin/netcat -u -l -p 6666 >> /var/log/netconsole" >> /etc/inittab
telinit q
==== Adding date/time to messages ====
If you want the log to contain date/time of each line, you can use '''awk''' like this:
netcat -u -l -p6666 | awk '{print strftime("%d %b %Y %H:%M:%S"), $0; fflush(stdout);}' >> /var/log/netconsole
See man strftime for info about how to tailor strftime() argument to your needs.
Note that if you want to add this to /etc/inittab, it should be done like this:
echo "netcat -u -l -p6666 | awk '{print \
strftime("%d %b %Y %H:%M:%S"), \$0; fflush(stdout);}' \
>> /var/log/netconsole" > /usr/local/sbin/netconsole
chmod a+x /usr/local/sbin/netconsole
echo "n1:23:respawn:/usr/local/sbin/netconsole" >> /etc/inittab
telinit q
==== Configuring logrotate ====
For long term capturing you would like to do log rotating some way. With logrotate you can do it by creating config file /etc/logrotate.d/netconsole:
<pre>
/var/log/netconsole {
weekly
rotate 8
missingok
compress
copytruncate
notifempty
# Need to restart logger after log file move
postrotate
# Below line assumes netcat will be restarted by init
killall -TERM netcat > /dev/null 2>&1 || true
}
</pre>
<!--T:53-->
For more details, see man logrotate.
