tips for baby steps in kernel debugging

Even if you have been a programming in C for a while, getting into kernel debugging can be intimidating. If you are used to c-tags or using an IDE for your development, the task is even more challenging. However, a few tools listed below help you get started very quickly:

Obviously the first step is getting the kernel, and depending on what kernel you are running and/or what Linux distribution you have, this can be tricky. For Fedora systems, there are simple ways to get the kernel source.

LXR - Linux Cross Reference
This is a very useful resource if you want to get a quick idea about the flow of code, structures. LXR is a toolset that has the entire kernel source indexed. It's an Ajax interface and makes it very easy to browse source code.

printk()
This is the printf of the kernel. The syntax is similar to printf. The useful argument is the loglevels that can attach a level of importance to your messages. The definition of the loglevels are in include/linux/kernel.h

dump_stack()
Sometimes, tracking the code flow is easier if you can show the program stack. Most architectures have dump_stack() implemented. This can be a very useful weapon in a newbie kernel debugger's arsenal.

There are some good tutorials that do some hand-holding for writing kernel modules, but if you have to track an issue in the core kernel or just want to get a better understanding, the tools described above are very useful.

Go on, get your feet wet in kernel debugging.

msft drivers for Linux

This has been all over tech news last week. Microsoft contributing code to the kernel (GPL for the first time). The drivers help Linux run better on a Windows host.

http://news.cnet.com/8301-13860_3-10290818-56.html

I like the note at the bottom of the article...
"For those that want to hear Microsoft's take on the news, here's a video of Hanrahan discussing the move with Sam Ramji, the company's senior director of platform strategy. (Note: Silverlight is required.) "

Irony anyone?

mcelog

At work, I have to decode kernel panics on 64-bit systems occasionally.

`mcelog´ it seems could be a useful tool.

mcelog decodes machine check events (hardware errors) on x86-64 machines running a 64-bit Linux kernel. It should be run regularly as a cron job on any x86-64 Linux system (if it is not in the default packages on your x86-64 distribution, please complain to your distributor). It can also decode machine check panic messages from console logs.

I don't have a good example on it´s usage, but on one of my systems, I noticed this in /var/log/mcelog (the cron script is setup to write to /var/log/mcelog in Fedora distributions).


MCE 0
HARDWARE ERROR. This is *NOT* a software problem!
Please contact your hardware vendor
CPU 2 BANK 3 TSC c82ff2586f6b0
ADDR 219540
STATUS 942000470001010a MCGSTATUS 0

et tu, Linux?

Well, how is that for a dramatic title? A little Shakespeare reference!

So, a few days ago, a _serious_ bug in the 2. 6 kernel (from 2.6.17 to 2.6.24.1) was discovered. Very well documented on what exactly it is and how to reproduce it locally (if you are one of those) in this Slashdot article. The issue is that a user can gain access as 'root' if the exploit is execute on your system, which means that now that user has complete access to your system.

On my personal laptop, it obviously doesn't matter. The issue is when you are running multiuser access servers, as in a University network. I am not a systems administrator, but if I was, I would be worried to say the list.

Well, it didn't take long to find a patch.

For us smart Ubuntu users, don't worry about things if you don't know what a patch means. Because we have the strength of Synaptic Update Manager with us. Just simply click on the update notification (which you should have received sometime today) and relax. If you are really curious, this is the issue that was patch'ed. It will update all the necessary headers, kernel image and source files.

If you are one of the unfortunate Fedora or RedHat users and running the affected 2.6 kernel, applying the patch to your kernel source and recompiling the kernel is do-able, but not without raising your heartbeat a few notches. This might work for you:

1. Get the patch from here. It also has a lot of information of how to apply it, etc.

2. cd to the kernel source (hopefully you have it installed). Generally should be /usr/src/linux-2.6.x.x. If not installed, try this:
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-x.y.z.tar.bz2

3. Apply the patch to the kernel source.
patch <

4. Compile and install. This can be little tricky if your kernel configuration (.config) is not created for your system. This would be the case if you just downloaded the source.

If you have the .config for your system, just follow these steps:
a) $ make

b) $ make modules

c) $ su -
# make modules_install

d) $ make install

This should have created the following in your /boot:
* System.map-2.6.x.x
* config-2.6.x.x
* vmlinuz-2.6.x.x

e) Create initrd image:
# cd /boot
# mkinitrd -o initrd.img-2.6.23 2.6.23

f) Update /etc/grub.conf (as in I am not a fan of LILO)

g) Say a prayer and reboot


in case of emergency

On one of the embedded devices that I write software for, I am running the 2.6.21 kernel patched with preempt-rt patch. I won't get into discussing what my opinion of real-time Linux is right now, so save that for a later post.

But, since the device is going to be deployed in a 24-7 environment, I figured it might help to reboot the system in case of a kernel panic rather than having it sit around since the privilege of console access is not affordable.

The solution, configure /etc/sysconfig.conf :

kernel.panic = N # to reboot immediately
kernel.panic = 5 # to reboot after a 5 second delay

You will notice the change in /proc/sys/kernel/panic, which means, you can change this at runtime by doing:

echo 5 > /proc/sys/kernel/panic