리눅스 커널 crash dump의 분석개요

[이누기]

출처: http://magazine.redhat.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/

리눅스 커널 crash dump의 분석개요

by Eugene Teo

Red Hat Crash Utility는 커널 특성의 디버거이다. 이것은 일반적으로 시스템이 패닉되었거나,LOCK UP 또는 반응이 없는 경우에 성능을 위한 시스템의 사후 검토를 위해 사용될 수 있다.

이 문서에서 Eugene Teo는 어떻게 crash를 설치하고, 디버깅과 root-crash이유를 분석하기 위해 , crash dump파일로부터 중요한 정보를 얻을 수 있는지를 알려줄 것이다.

필수 조건

crash utility는 아래의  세가지 필수적으로  따른다::

·         Kernel object file:  A vmlinux kernel object file.

·         Red Hat Enterprise Linux 3에서는 /boot에서 , Red Hat Enterprise Linux 4 and 5에 대해서는 /usr/lib/debug/lib/modules 에서 실행하는 커널이 vmlinux 찾게 된다.

·         Kernel crash dump: 아마 세가지 crash dump 기능(Diskdump, Netdump, or Kdump)을 통해서 만들어진 kernel crash dump파일로 만들어질 것이다. Vmcore 또는 vmcore.imcomplete(완전하기 만들어지지 않는 경우)라고 하는 이름일 것이다.그리고 이것은 기본적으로 /var/crash 디렉토리에 만들어진다. Diskdump는 다음 섹션에서 소개될 것이다.

·         Linux kernel versions:  crash utility는 아래로 최소 Red Hat Linux 6.0에서 Red Hat Enterprise Linux 5까지 지원할 것이다.

crash 설치

Red Hat Enterprise Linux 3 에서는 만약  Development Tools 패키지가 선택되었다면,자동적으로 시스템 설치시에 crash 유틸리티가 설치될 것이다.아래는 command상에서 설치하는 방법이다:

# rpm -ivh crash-4.0-2.30.i386.rpm

Preparing...                ########################################### [100%]

   1:crash                  ########################################### [100%]

실행 가능한 crash는 /usr/bin 디렉토리에 설치될 것이다.

또한 당신이 vmcore에서 crash를 호출하기 전에 ,당신은 kernel관련 debuginfo 패키지의 설치가 반드시 필요하다. vmlinux커널 디버그 정보는 따로 debuginfo파일내에 저장된다.

debuginfo패키지는 커널 버전과 맞아야 한다 (여러가지 smp또는 hugemem과 같이 또한 아키텍쳐도 맞아야 한다.)

당신은 ftp://ftp.redhat.com/pub에서 패키지를 다운로드 할 수 있고, 또한 up2date, yum을 통해서 rhn 사이트로부터 설치가 가능하다

# file ./vmcore <-- 당신의 커널 아키텍쳐를 보여줄 것이다.

./vmcore: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'vmlinux'

# strings vmcore | fgrep -m1 ‘Linux ‘ <-- will show you the kernel variant

Linux version 2.6.9-22.EL (bhcompile@porky.build.redhat.com) (gcc version 3.4.4

     20050721 (Red Hat 3.4.4-2)) #1 Mon Sep 19 18:20:28 EDT 2005

# rpm -qa kernel-debuginfo <-- will show you all the versions installed

# rpm -ivh kernel-debuginfo-2.6.9-22.EL.i686.rpm

Preparing...                ########################################### [100%]

   1:kernel-debuginfo       ########################################### [100%]

# ls /usr/lib/debug/lib/modules/ -l

total 24

drwxr-xr-x 3 root root 4096 May 2 10:41 2.6.9-22.EL

drwxr-xr-x 3 root root 4096 May 2 10:41 2.6.9-22.ELhugemem

drwxr-xr-x 3 root root 4096 May 2 10:41 2.6.9-22.ELsmp

# ls /usr/lib/debug/lib/modules/2.6.9-22.EL -l

total 32848

drwxr-xr-x 9 root root     4096 May 1 19:50 kernel

-rwxr-xr-x 1 root root 33583473 Sep 20 2005 vmlinux

Take note that:

·         커널 패키지를 설치할때에는 -Uvh보다는 –ivh를 사용하는 것이 좋다. 이것은 당신이 커널 설치에  문제가 발생했을 때를 대비하여 이전에 커널을 백업해두기 때문이다

·         위의 결과에서  file ./vmcore와   strings vmcore | fgrep -m1 ‘Linux ‘명령을 해보아라. 

·          Red Hat Enterprise Linux 5에서는 vmlinux kernel-debuginfo 패키지는 두개의 패키지로 나뉘어져 있다.: : kernel-debuginfo-version.arch.rpm 와 kernel-debuginfo-common-version.arch.rpm. Red Hat Enterprise Linux 5 kernels에서 crash dump분석을 위해서는 둘다 필요하다.

Run crash

Crash가 vmcore에서 실행할 때, 최소 두개의 인수가 항상 필요하다:

· 실행하고 있는 커널에 관련된 vmlinux파일은 /usr/lib/debug/lib/modules/ 디렉토리에서 찾는다..

· 커널 crash dump이름은 vmcore이다.

For example:

# crash /usr/lib/debug/lib/module/vmlinux
     /var/crash/127.0.0.1-2007-04-30-21\:38/vmcore
[...]
      KERNEL: /usr/lib/debug/lib/modules/2.6.9-22.EL/vmlinux
    DUMPFILE: /home/eteo/crash/127.0.0.1-2007-04-30-21:38/vmcore
        CPUS: 1
        DATE: Mon Apr 30 21:38:40 2007
      UPTIME: 00:04:04
LOAD AVERAGE: 0.36, 0.23, 0.08
       TASKS: 36
    NODENAME: localhost.localdomain
     RELEASE: 2.6.9-22.EL
     VERSION: #1 Mon Sep 19 18:20:28 EDT 2005
     MACHINE: i686 (1862 Mhz)
      MEMORY: 1 GB
       PANIC: "Oops: 0002 [#1]" (check log for details)
         PID: 2857
     COMMAND: "bash"
        TASK: f7b677f0 [THREAD_INFO: f7191000]
         CPU: 0
       STATE: TASK_RUNNING (SYSRQ)
crash>

일반적인 Crash 명령 사용법

crash에는 많은 명령들이 있다. 또한 새로운 명령들을 추가하여, 새로운 코드를 써서,

그리고 실행가능한 crash로 컴파일하여 또는 확장명령을 사용하여 불려질 수 있는 shared object library를 만들어서, crash를 확장시킬 수 있다.

아래는 일반적으로 사용되는 crash 명령들이다:

help – get help
crash는 help라고 치면 즉시 help정보를 볼수 있다.각 명령들은 각자 man페이지와 같이 가지고 있고, help command이름을 치면 그 사용법을 볼 수 있다..

crash> help

*      files   mod    runq   union
alias  foreach mount  search vm
ascii  fuser   net    set    vtop
bt     gdb     p      sig    waitq
btop   help    ps     struct whatis
dev    irq     pte    swap   wr
dis    kmem    ptob   sym    q
eval‎   list    ptov   sys
exit   log     rd     task
extend mach    repeat timer

crash version: 4.0-3.3   gdb version: 6.1
For help on any command above, enter "help ".
For help on input options, enter "help input".
For help on output options, enter "help output".

Tip: 모든 crash명령어들은 외부 명령에 pipe될 수있고, 또한 file에 redirect될 수 있다.

crash> log > log.txt

이것은 커널 내부 로그를 log.txt라고 불리는 로컬파일에 보낼 것이다..

crash> ps | fgrep bash | wc -l

이것은 실행한 bash 수를 카운트 할 것이다.

sys – system data

crash> sys

       KERNEL: /usr/lib/debug/lib/modules/2.6.9-22.EL/vmlinux

     DUMPFILE: /home/eteo/crash/127.0.0.1-2007-04-30-21:38/vmcore

         CPUS: 1

         DATE: Mon Apr 30 21:38:40 2007

       UPTIME: 00:04:04

 LOAD AVERAGE:  0.36, 0.23, 0.08

        TASKS: 36

     NODENAME: localhost.localdomain

      RELEASE: 2.6.9-22.EL

      VERSION: #1 Mon Sep 19 18:20:28 EDT 2005

      MACHINE: i686 (1862 Mhz)

       MEMORY: 1 GB

        PANIC: "Oops: 0002 [#1]" (check log for details)

sys 메시지는 시스템의 정보를 갖고 있다(. kernel release, kernel version, number of CPUs, amount of memory, etc) , vmcore를 받은 시간, 운영 기간, 그리고 panic (e.g. oops type, panic task/PID/command, etc).

bt – backtrace

crash> bt

PID: 2857   TASK: f7b677f0 CPU: 0    COMMAND: "bash"

 #0 [f7191e04] start_disk_dump at f89d7bb3

 #1 [f7191e18] die at c010682e

 #2 [f7191e48] do_page_fault at c011ab00

[...]

 #9 [f7191fc0] system_call at c030f918

    EAX: 00000004 EBX: 00000001 ECX: b7de7000 EDX: 00000002

    DS: 007b       ESI: 00000002 ES: 007b        EDI: b7de7000

    SS: 007b       ESP: bfe01650 EBP: bfe01670

    CS: 0073       EIP: 003297a2 ERR: 00000004 EFLAGS: 00000246

log – dump system message buffer

crash> log

[...]

SysRq : Crashing the kernel by request

Unable to handle kernel NULL pointer dereference at virtual address 00000000

 printing eip:

c0233fa7

*pde = 3e9f3067

Oops: 0002 [#1]

Modules linked in: md5 ipv6 autofs4 i2c_dev i2c_core sunrpc scsi_dump diskdump dm_mirror dm_mod button battery ac yenta_socket pcmcia_core uhci_hcd ehci_hcd shpchp snd_intel8x0 snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore ipw2200 ieee80211 ieee80211_crypt tg3 floppy ext3 jbd ata_piix libata sd_mod scsi_mod

CPU:    0

EIP:    0060:[]    Not tainted VLI

EFLAGS: 00010246    (2.6.9-22.EL)

EIP is at sysrq_handle_crash+0x0/0x8

eax: 00000063   ebx: c0370db4    ecx: 00000000  edx: 00000000

esi: 00000063   edi: 00000000    ebp: 00000000  esp: f7191f60

ds: 007b   es: 007b    ss: 0068

Process bash (pid: 2857, threadinfo=f7191000 task=f7b677f0)

Stack: c02342d8 c032dc4e c032f105 00000003 00000002 f7b6adc0 00000002

       f7191fac c01a8a13 c0362740 c0168205 f7191fac b7de7000 f7b6adc0

       fffffff7 b7de7000 f7191000 c01682cf f7191fac 00000000 00000000

       00000000 00000001 00000002

Call Trace:

 [] __handle_sysrq+0x58/0xc6

 [] write_sysrq_trigger+0x23/0x29

 [] vfs_write+0xb6/0xe2

 [] sys_write+0x3c/0x62

 [] syscall_call+0x7/0xb

Code: 4c 11 42 c0 05 00 00 00 c7 05 50 11 42 c0 2f cc 31 c0 c7 05 54 11 42 c0 00 00 00 00 c7 05 58 11 42 c0 00 00 00 00 e9 e5 0b f0 ff  05 00 00 00 00 00 c3 e9 e1 59 f3 ff e9 1e bc f3 ff 85 d2 89

Log 명령은 시간순서로 커널 로그 버퍼를 덤프한다.

이는 시스템에서 dmesg를 칠 때 보여주는 것과 비슷하다.

이것은 당신이 panic나 oops메시지를 보기를 원할 때 유용할 것이다. Oops는 몇가지 예외로 트리거된다. 이는 CPU 레지스터의 상태와 커널 스택을 즉시 덤프한다.

커널 패닉 메시지로부터 우리는 어떻게 패닉이 트리거되었는지, (e.g. the function or process or pid or command or address that triggered the panic)의 힌트를 찾을 수 있다, 레지스터의 정보, 커널 모듈리스트, 로드된 커널모듈이 다른 커널로 바뀌어져있는지(수정되었는지)등..

로그내에 각각의 세션 아래에 있는 주석을 보아라:

crash> log
[...]
SysRq : Crashing the kernel by request <-- this panic is intentional
Unable to handle kernel NULL pointer dereference at virtual address 00000000

이것은 시도되는 참조 주소이다.

 printing eip:
c0233fa7 <--이것은 실패가 발생한 주소이다.

*pde = 3e9f3067

Oops: 0002 [#1] <--종종 하나의 oops는 좀더 트리거할 것이다: 오직 첫번째 것만 신뢰할 수 있다

Modules linked in: md5 ipv6 autofs4 i2c_dev i2c_core sunrpc scsi_dump diskdump dm_mirror dm_mod button battery ac yenta_socket pcmcia_core uhci_hcd ehci_hcd shpchp snd_intel8x0 snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore ipw2200 ieee80211 ieee80211_crypt tg3 floppy ext3 jbd ata_piix libata sd_mod scsi_mod

CPU: 0

EIP: 0060:[]    Not tainted VLI <--이 첫번째 부분은 code segment 와 명령주소이다. 만약 문제가 있다면, 이것은 아래처럼 표현될 것이다.:

               G – All modules loaded have a GPL

               or compatible license

               P – Proprietary modules loaded

               F – Module forcibly loaded

               S – Oops on hardware that are not

               SMP capable

               R – Module forcibly unloaded

               M - Machine Check Exception (MCE)

               occurred

               etc (see Further readings section).

EFLAGS: 00010246    (2.6.9-22.EL) <--이것은 프로그램의 상태, 레지스터의 정보를 표현하는 것이다.

EIP is at sysrq_handle_crash+0x0/0x8
eax: 00000063 ebx: c0370db4 ecx: 00000000 edx: 00000000
esi: 00000063 edi: 00000000 ebp: 00000000 esp: f7191f60
ds: 007b es: 007b ss: 0068
Process bash (pid: 2857, threadinfo=f7191000 task=f7b677f0)
Stack: c02342d8 c032dc4e c032f105 00000003 00000002 f7b6adc0 00000002

<-- stack, operations, return address들이 있다.

       f7191fac f7191000 c01682cf f7191fac 00000000 00000000 00000000

       00000001 00000002

 Call Trace: <--이것은 함수호출의 backtrace 이다.

 [] __handle_sysrq+0x58/0xc6

 [] write_sysrq_trigger+0x23/0x29

 [] vfs_write+0xb6/0xe2

 [] sys_write+0x3c/0x62

 [] syscall_call+0x7/0xb

Code: 4c 11 42 c0 05 00 00 00 c7 05 50 11 42 c0 2f cc 31 c0 c7 05 54 11 42 c0 00 00 00 00 c7 05 58 11 42 c0 00 00 00 00 e9 e5 0b f0 ff  05 00 00 00 00 00 c3 e9 e1 59 f3 ff e9 1e bc f3 ff 85 d2 89

c0233fa7라인으로 부터 우리는 이 주소가 실패가 발생했다는 것을 알 수 있다.

따라오는 명령으로는 커널 안에서 트리거된 어떤 함수, 소스코드, 또는 assembly statement와 같은 좀더 많은 힌트들을 얻을 수 있다:

:

crash> dis -lr c0233fa7

/usr/src/build/614745-i686/BUILD/kernel-2.6.9/linux-

     2.6.9/drivers/char/sysrq.c: 115

0xc0233fa7 :        movb   $0x0,0x0

·  ps – display process status information

crash> ps

   PID    PPID CPU   TASK   ST %MEM  VSZ  RSS COMM

      0      0  0 c0358be0  RU  0.0    0    0 [swapper]

      1      0  0 f7e01770  IN  0.1 1680  684 init

[...]

   2380      1  0  f7ac2800 IN  0.0 1604  504 mingetty

   2769   2371  0  f7ac3970 IN  0.2 5740 1636 bash

   2852      1  0  f7b1a880 IN  0.2 4240 2012 sshd

  2855 2852 0 f7b66680 IN 0.3 8316 2756 sshd

> 2857 2855 0 f7b677f0 RU 0.2 6260 1628 bash

가끔 이것은 어떤 프로세스가 어떤 부모프로세스에 따르는지, 또는 그 반대인지를 알아내는데 유용하다. Ps는 –C 와 –p 옵션이 있어서 child와 parent 프로세스를 보여줄 수 있다.

·  crash> ps -p 2857

PID: 0       TASK: c0358be0 CPU: 0    COMMAND: "swapper"

 PID: 1       TASK: f7e01770 CPU: 0    COMMAND: "init"

  PID: 2852    TASK: f7b1a880 CPU: 0    COMMAND: "sshd"

   PID: 2855    TASK: f7b66680 CPU: 0    COMMAND: "sshd"

    PID: 2857    TASK: f7b677f0 CPU: 0    COMMAND: "bash"

·  files – open files

crash> files

PID: 2857    TASK: f7b677f0 CPU: 0   COMMAND: "bash"

ROOT: /     CWD: /root

 FD    FILE      DENTRY    INODE   TYPE  PATH

  0 f7a6e7c0 f7790198 f7b0fdcc     CHR   /dev/pts/0

  1 f7b6adc0 f7190130 f7b9ca4c     REG   /proc/sysrq-trigger

  2 f7a6e7c0 f7790198 f7b0fdcc     CHR   /dev/pts/0

 10 f7a6e7c0 f7790198 f7b0fdcc     CHR   /dev/pts/0

255 f7a6e7c0 f7790198 f7b0fdcc     CHR   /dev/pts/0

crash> files 2852

PID: 2852    TASK: f7b1a880 CPU: 0   COMMAND: "sshd"

ROOT: /     CWD: /

 FD    FILE      DENTRY    INODE   TYPE  PATH

  0 f7b336c0 f78001d8 f7cb1ba4     CHR   /dev/null

  1 f7b336c0 f78001d8 f7cb1ba4     CHR   /dev/null

  2 f7b336c0 f78001d8 f7cb1ba4     CHR   /dev/null

  3 f7b69600 f7bf5280 f7aadafc     SOCK  socket:/[6277]

·  dev – device data

crash> help dev

[...]

만약 인자가 없이 엔터를 치면, 명령은  chrdevs 와 blkdevs 배열을 덤프할 것이다..

crash> dev

CHRDEV    NAME         OPERATIONS

 1      mem              (none)

 4      /dev/vc/0        (none)

 4      tty              (none)

[...]

BLKDEV     NAME        OPERATIONS

 1      ramdisk        c0376d08

 2      fd            (unknown)

 8      sd             f880e070 

· 

Further reading

To learn more about this topic, check out the following reference links:

• White Paper: Red Hat Crash Utility by David Anderson, 2003
• More about Oops
• To find out more about Diskdump, read /usr/share/doc/diskdumputils-version/README
• Red Hat Kbase: How do I configure a Netdump Server and a Netdump Client?
• Red Hat Kbase: How do I configure kexec/kdump on Red Hat Enterprise Linux 5?

Acknowledgements

The author would like to thank Guy Streeter, Wade Mealing, and Masahiro Matsuya for reviewing and suggesting improvements for several drafts of this article.

About the author

Eugene Teo is Technical Account Manager of Red Hat’s Asia Pacific Global Support Services. Eugene received his bachelor’s degree in Computing from the National University of Singapore. In his spare time, Eugene enjoys learning new things, auditing the Linux kernel source code, and contributing kernel fixes.

This entry was posted by The editorial team on Wednesday, August 15th, 2007 at 10:25 am and is filed under Red Hat Enterprise Linux, technical. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Setup Diskdump

Diskdump is one of the two different crash dump facilities that we shipped with Red Hat Enterprise Linux 3 and 4. This article will not cover Netdump or Kdump.

Before you beginning setting up Diskdump on your machine, do read /usr/share/doc/diskdumputils-version/README to make sure that your machine has a supported storage adapter before proceeding.

Assign a disk device to dump memory. It may be:

• a full disk device (for Red Hat Enterprise Linux 3 only), e.g. /dev/sda
• a partition of a disk device, e.g. /dev/sda2
• a swap partition (for Red Hat Enterprise Linux 4 only), e.g. /dev/sda2

Define the disk device in /etc/sysconfig/diskdump. In this example, we will use /dev/sda2:

# vi /etc/sysconfig/diskdump

add the line "DEVICE=/dev/sda2"

Load the kernel module:

# tail -f /var/log/message &

# modprobe diskdump

Apr 30 21:29:20 kerndev kernel: disk_dump: Maximum block size: 16384

Apr 30 21:29:20 kerndev kernel: disk_dump: total blocks required: 261770

     (header 3 + bitmap 8 + memory 261759)

See /proc/diskdump after loading diskdump kernel module:

# cat /proc/diskdump

# sample_rate: 8

# block_order: 2

# fallback_on_err: 1

# allow_risky_dumps: 1

# dump_level: 0

# total_blocks: 261770

#

Format the diskdump device:

# service diskdump initialformat

/dev/sda2: [100.0%]

See /proc/diskdump after formatting:

# tail -n2 /proc/diskdump

sda2 102398310 10233405

Enable Diskdump service:

# chkconfig diskdump on

# service diskdump start

Starting diskdump:                            [ OK ]

# Apr 30 21:31:19 kerndev diskdump: activating succeeded

Test that Diskdump works. The following commands will crash your machine:

# echo 1 > /proc/sys/kernel/sysrq

# echo c > /proc/sysrq-trigger

Make sure that you run the above two commands in console (press Ctrl + Alt + F1), so that we can see what is happening when your system crashes. You have to perform this so that you can have a vmcore file to follow the rest of the paper. It will be located at /var/crash.

15 responses to “A quick overview of Linux kernel crash dump analysis”

1.        Basil Gohar says:
August 15th, 2007 at 12:27 pm

The link “ftp://ftp.redhat.com/pub” is formatted very strangely. The quotes for the tag’s href attribute appear to be “smart quotes” style or something other than a standard quote.

2.        Gabriel Donnell says:
August 24th, 2007 at 10:44 pm

The URLs for the Red Hat Knowledge Base are bad. The URL and text before and after the FAQ URL is the problem.

How do I configure a Netdump Server and a Netdump Client?
- Bad: http://www.redhatmagazine.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/%E2%80%9Dhttp://kbase.redhat.com/faq/FAQ_43_2467.shtm%E2%80%9D
- Good: http://kbase.redhat.com/faq/FAQ_43_2467.shtm

How do I configure kexec/kdump on Red Hat Enterprise Linux 5?
- Bad: http://www.redhatmagazine.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/%E2%80%9Dhttp://kbase.redhat.com/faq/FAQ_105_9036.shtm%E2%80%9D
- Good: http://kbase.redhat.com/faq/FAQ_105_9036.shtm

3.        Eugene says:
August 25th, 2007 at 4:51 am

Thanks guys. I have informed the editor about the formatting problems.

4.        Ashish Barot says:
September 4th, 2007 at 10:23 am

It is good article.

Can any one post few more details on crash dump on different hardware platform.

so we can come to know more error.

Thanks.
Ashish Barot.

5.        Safir says:
September 20th, 2007 at 8:40 am

Hi,

Not very usable because of the formatting.

It would be better to provide also a pdf version.

Thanks

6.        Eugene says:
September 27th, 2007 at 8:02 am

Ashish, thanks for the suggestion.

Safir, I will provide a pdf version, and post the link here soon. Stay tuned.

7.        Philip Pokorny says:
October 9th, 2007 at 6:30 pm

Where can we get the kernel-debuginfo RPM’s for the production Red Hat kernels? YUM, UP2DATE and RHN don’t seem to have them available.

Do I need to rebuild the kernel from the SRPM?

8.        Eugene says:
October 9th, 2007 at 6:34 pm

Hi Philip,

You can download the packages at ftp://ftp.redhat.com/pub. No you do not need to rebuild the kernel from the source RPM.

Thanks.

9.        Eugene says:
October 9th, 2007 at 6:57 pm

Hi Philip,

If you are using yum, take a look at the .repo files in /etc/yum.repos.d/. For example, in Fedora 7, you can set enabled=0 to 1 under [fedora-debuginfo] in /etc/yum.repos.d/fedora.repo file. Once you done that, type “yum clean all” on the command line, and then start yum -y install the kernel-debuginfo rpm you need.

Hope this info helps.

Eugene

10.     Mag says:
October 17th, 2008 at 2:28 pm

This is nice but do you have something a little bit more recent? It seems “printing eip:” has been depreciated in Redhat 5.2. Unless I am doing something wrong.

TIA

11.     Red Hat Magazine | Linux kernel crash dump analysis overview says:
November 16th, 2008 at 11:41 pm

[...] A quick overview of Linux kernel crash dump analysis [...]

12.     Eugene says:
November 18th, 2008 at 6:58 pm

Thanks Mag. I will try to update the article soon. Will keep you posted.

13.     Manish Sarori says:
November 25th, 2008 at 10:20 pm

In the calltrace, does anyone know what the numbers after the + mean (see below)? For example, after __handle_sysrq is 0×58/0xc6. Is this some sort of offset into the function where the call was made? I need to pinpoint an exception in a lengthy function. Thanks in advance.

Call Trace:
[] __handle_sysrq+0×58/0xc6
[] write_sysrq_trigger+0×23/0×29

14.     Kaiwan B says:
December 3rd, 2008 at 9:50 pm

Manish, reg:

+x/y

x represents the approx offset into the function .
y represents the approx total length of that function. Actually, y is the distance to the next global symbol. Therefore, these are approximations and not exact – but they do give you a “good enough” idea of where the faulting code lies. Typically, you could now try using objdump to disassemble and look at the offsets that show up, matching to the closest ‘x’ offset above.

15.     Manish Sarori says:
December 16th, 2008 at 6:31 pm

Thanks Kaiwan!

Linux-Kernel-Crash-Dump-Analysis.pdf

[이누기]

http://people.redhat.com/anderson/.crash_whitepaper/