NFS 관련 error log 입니다...

[Producer LEE~~]

장비: sun fire 280R

O/S: Soaris 8

NFS Server 로 사용중 갑자기 한번씩 다운됩니다..

고수님덜 많은 조언 부탁드립니다..... 

관련 log 입니다.=> messages.0

Jan 14 03:29:13 compsun2 /usr/lib/nfs/nfsd[870]: [ID 126371 daemon.error] unable to register with kernel rpc: Transport endpoint is not connected
Jan 14 03:35:13 compsun2 last message repeated 3 times
Jan 14 03:42:43 compsun2 /usr/lib/nfs/nfsd[870]: [ID 126371 daemon.error] unable to register with kernel rpc: Transport endpoint is not connected
Jan 14 03:47:53 compsun2 last message repeated 6 times
Jan 14 03:49:23 compsun2 /usr/lib/nfs/nfsd[870]: [ID 126371 daemon.error] unable to register with kernel rpc: Transport endpoint is not connected
Jan 14 03:52:53 compsun2 last message repeated 8 times
Jan 17 02:18:11 compsun2 /usr/lib/nfs/nfsd[870]: [ID 126371 daemon.error] unable to register with kernel rpc: Transport endpoint is not connected
Jan 17 02:22:11 compsun2 last message repeated 4 times
Jan 17 02:24:41 compsun2 /usr/lib/nfs/nfsd[870]: [ID 126371 daemon.error] unable to register with kernel rpc: Transport endpoint is not connected
Jan 17 02:25:11 compsun2 last message repeated 1 time
Jan 17 02:29:21 compsun2 /usr/lib/nfs/nfsd[870]: [ID 126371 daemon.error] unable to register with kernel rpc: Transport endpoint is not connected
Jan 17 02:34:31 compsun2 last message repeated 8 times
Jan 17 02:37:31 compsun2 /usr/lib/nfs/nfsd[870]: [ID 126371 daemon.error] unable to register with kernel rpc: Transport endpoint is not connected
Jan 17 02:41:21 compsun2 unix: [ID 836849 kern.notice]

Jan 17 02:41:21 compsun2 ^Mpanic[cpu0]/thread=30004462f60:
Jan 17 02:41:21 compsun2 unix: [ID 783603 kern.notice] Deadlock: cycle in blocking chain
Jan 17 02:41:21 compsun2 unix: [ID 100000 kern.notice]
Jan 17 02:41:21 compsun2 genunix: [ID 723222 kern.notice] 000002a10088f2b0 genunix:turnstile_block+360 (30004462f60, 0, 10410100, 30038745550, 0, 3006
a51f500)
Jan 17 02:41:21 compsun2 genunix: [ID 179002 kern.notice]   %l0-3: 0000000010047d6c 000003000321b828 0000030002dff100 0000000000000000
Jan 17 02:41:21 compsun2   %l4-7: 0000000010410100 0000000000000000 000002a100085d20 0000030002dff100
Jan 17 02:41:21 compsun2 genunix: [ID 723222 kern.notice] 000002a10088f380 unix:mutex_vector_enter+3b8 (3006a51f500, 10415fc0, 10424590, 3006b72f418,
0, 11)
Jan 17 02:41:21 compsun2 genunix: [ID 179002 kern.notice]   %l0-3: ffe3fe330dfe9af8 000003006a51f500 0000030002bce020 0000000000000640
Jan 17 02:41:21 compsun2   %l4-7: 0000030002dff100 0000030002bce020 000002a100085d21 000002a10088f3a0
Jan 17 02:41:21 compsun2 genunix: [ID 723222 kern.notice] 000002a10088f440 rpcmod:mir_svc_start+c (3003872c378, 3006a51f488, 22, 1, 30004462f64, 30004
462f5c)
Jan 17 02:41:21 compsun2 genunix: [ID 179002 kern.notice]   %l0-3: 000003006a51f500 0000000001010100 00000000ff000000 0000000000ff0000
Jan 17 02:41:21 compsun2   %l4-7: 000000000000ff00 0000000001010000 0000000000000000 0000000000000000
Jan 17 02:41:21 compsun2 genunix: [ID 723222 kern.notice] 000002a10088f4f0 rpcmod:svc_cots_kstart+4c (30001e9d3b0, 30001e9d348, 3006fc080c8, 30001e9d3
48, 81010100, ff00)
Jan 17 02:41:21 compsun2 genunix: [ID 179002 kern.notice]   %l0-3: 000000001018e6fc 0000000000008328 000003007073ac10 0000000000000800
Jan 17 02:41:21 compsun2   %l4-7: 000000001047ebd0 000002a10088f664 0000000000000020 0000000000000000
Jan 17 02:41:21 compsun2 genunix: [ID 723222 kern.notice] 000002a10088f5a0 rpcmod:svc_tli_kcreate+348 (1, 0, 3003872c378, 2a10088f788, 2a10088f780, 78
01a080)
Jan 17 02:41:21 compsun2 genunix: [ID 179002 kern.notice]   %l0-3: 00000000101990b4 000002a10088f7a0 0000000000000000 0000000000000000
Jan 17 02:41:21 compsun2   %l4-7: 0000000000000000 0000000000000000 000003007073ac10 0000000000000000
Jan 17 02:41:21 compsun2 genunix: [ID 723222 kern.notice] 000002a10088f6b0 nfssrv:nfs_svc+198 (3002a1890c8, 10, 8328, 3007073ac10, 100000, 2a10088f9e0
)
Jan 17 02:41:21 compsun2 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000000 0000000000100000 0000000000000000 0000000000000000
Jan 17 02:41:21 compsun2   %l4-7: 0000000000000000 0000000010411b58 0000000000000000 0000000000000000
Jan 17 02:41:21 compsun2 genunix: [ID 723222 kern.notice] 000002a10088f830 nfs:nfssys+d8 (b, ffbef674, ffffffff, 0, 0, 0)
Jan 17 02:41:21 compsun2 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000350 0000000000000000 0000000000000000 0000000000000000
Jan 17 02:41:21 compsun2   %l4-7: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Jan 17 02:41:21 compsun2 genunix: [ID 723222 kern.notice] 000002a10088fa30 genunix:loadable_syscall+7c (300000785f8, 10435438, 6a, 4482001a02, 0, 0)
Jan 17 02:41:21 compsun2 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000000 0000000000000000 000000000000000b 00000000ffbef674
Jan 17 02:41:21 compsun2   %l4-7: 00000000ffffffff 0000000000000000 0000000000000000 0000000000000000
Jan 17 02:41:21 compsun2 unix: [ID 100000 kern.notice]
Jan 17 02:41:21 compsun2 genunix: [ID 672855 kern.notice] syncing file systems...
Jan 17 02:41:22 compsun2 genunix: [ID 733762 kern.notice]  8
Jan 17 02:41:23 compsun2 genunix: [ID 733762 kern.notice]  7
Jan 17 02:41:25 compsun2 genunix: [ID 904073 kern.notice]  done
Jan 17 02:41:26 compsun2 genunix: [ID 353387 kern.notice] dumping to /dev/dsk/c1t0d0s1, offset 2097610752
Jan 17 02:44:08 compsun2 genunix: [ID 409368 kern.notice] ^M100% done: 274295 pages dumped, compression ratio 2.43,
Jan 17 02:44:08 compsun2 genunix: [ID 851671 kern.notice] dump succeeded
Jan 17 02:45:25 compsun2 genunix: [ID 540533 kern.notice] ^MSunOS Release 5.8 Version Generic_117350-39 64-bit
Jan 17 02:45:25 compsun2 genunix: [ID 913632 kern.notice] Copyright 1983-2003 Sun Microsystems, Inc.  All rights reserved.

UNIX.5 & VMCORE.5 분석결과

 

Solaris[TM] CAT 4.0 (build 509) for Solaris 8 64-bit SPARC(sun4u)

 

  Copyright ?2002 Sun Microsystems, Inc.  All rights reserved.
  Patents Pending.  Use is subject to license terms.
  Sun Microsystems proprietary - DO NOT DISTRIBUTE!

 

  Feedback regarding the tool should be sent to SolarisCAT_Feedback@Sun.COM
  Visit the Solaris CAT support forum at http://supportforum.sun.com/solariscat

 

opening vmcore.5 ...dumphdr...symtab...core...done
loading core data: modules...panic...memory...time...misc...done
loading stabs...patches...done

 

core file:      /var/crash/hst-ultra60/vmcore.5
user:           Super-User (root:0)
release:        5.8 (64-bit)
version:        Generic_117350-39
machine:        sun4u
node name:     

domain:           

hw_provider:    Sun_Microsystems
system type:    SUNW,Sun-Fire-280R
hostid:        

time of crash:  Mon Feb  5 11:31:31 KST 2007
age of system:  7 days 8 hours 47 minutes 16.05 seconds
panic cpu:      0 (ncpus: 2)
panic string:   Deadlock: cycle in blocking chain

 

running sanity checks.../etc/system...dump flags...misc...done

 

SolarisCAT(vmcore.5)> time

 

real    0m0.00s
user   0m0.00s
sys    0m0.00s

 

SolarisCAT(vmcore.5)> msgbuf => messages

 

todds12870 at ebus0: offset 1,300070
todds12870 is /pci@8,700000/ebus@5/rtc@1,300070
mc-us30 at root: SAFARI 0x0 0x400000 ...
mc-us30 is /memory-controller@0,400000
mc-us31 at root: SAFARI 0x1 0x400000 ...
mc-us31 is /memory-controller@1,400000
su0 at ebus0: offset 1,3062f8
su0 is /pci@8,700000/ebus@5/rsc-control@1,3062f8
su1 at ebus0: offset 1,3083f8
su1 is /pci@8,700000/ebus@5/rsc-console@1,3083f8
cpu0: UltraSPARC-III+ (portid 0 impl 0x15 ver 0x23 clock 900 MHz)
cpu1: UltraSPARC-III+ (portid 1 impl 0x15 ver 0x22 clock 900 MHz)
cpu 1 initialization complete - online
se0 at ebus0: offset 1,400000
se0 is /pci@8,700000/ebus@5/serial@1,400000
NOTICE: ce0: no fault external to device; service available
NOTICE: ce0: xcvr addr:0x00 - link up 1000 Mbps full duplex
dump on /dev/dsk/c1t0d0s1 size 10002 MB
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
pseudo-device: tod0
tod0 is /pseudo/tod@0
pseudo-device: pm0
pm0 is /pseudo/pm@0
pseudo-device: vol0
vol0 is /pseudo/vol@0
PCI-device: usb@5,3, ohci0
ohci0 is /pci@8,700000/usb@5,3
pseudo-device: fcode0
fcode0 is /pseudo/fcode@0
pseudo-device: devinfo0

devinfo0 is /pseudo/devinfo@0

WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f0 (ssd13):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f0 (ssd13):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,f1 (ssd5):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,f1 (ssd5):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f2 (ssd11):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f2 (ssd11):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f3 (ssd10):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f3 (ssd10):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,0 (ssd2):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 92160 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f3 (ssd10):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f3 (ssd10):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,f2 (ssd4):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,f2 (ssd4):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,f1 (ssd5):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,f1 (ssd5):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,f0 (ssd6):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,f0 (ssd6):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,0 (ssd9):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 92160 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f3 (ssd10):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f3 (ssd10):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f2 (ssd11):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f2 (ssd11):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f1 (ssd12):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@1/fp@0,0/ssd@w5006048449af6847,f1 (ssd12):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,f0 (ssd6):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks
WARNING: /pci@8,700000/SUNW,qlc@3/fp@0,0/ssd@w5006048449af6848,f0 (ssd6):
        corrupt label - wrong magic number

 

        Vendor 'EMC', product 'SYMMETRIX', 5760 512 byte blocks

 

panic[cpu0]/thread=300043d86c0: Deadlock: cycle in blocking chain

 

000002a1007f72b0 genunix:turnstile_block+360 (300043d86c0, 0, 10410100, 300043f0008, 0, 30015456ad0)
  %l0-3: 0000000010047d6c 00000300043d6060 0000030002dfdf00 0000000000000000
  %l4-7: 0000000010410100 0000000000000000 000002a100085d20 0000000000000000
000002a1007f7380 unix:mutex_vector_enter+3b8 (30015456ad0, 10415fc0, 10424590, 2a10007dd20, 0, 5)
  %l0-3: fffdbd22e14188b8 0000030015456ad0 0000030002bce020 0000000000000640
  %l4-7: 0000000000000000 0000030002bce020 000002a100085d20 0000000000000000
000002a1007f7440 rpcmod:mir_svc_start+c (300043694a8, 30015456a58, 22, 1, 300043d86c4, 300043d86bc)
  %l0-3: 0000030015456ad0 0000000000000016 0000000000000006 000000001000a408
  %l4-7: 000000000000ff00 0000000001010000 0000000000000000 000002a1007f7450
000002a1007f74f0 rpcmod:svc_cots_kstart+4c (30001e9d880, 30001e9d818, 30004a39cc8, 30001e9d818, 81010100, ff00)
  %l0-3: 000000001018e6fc 0000000000008328 0000030004673260 0000000000000800
  %l4-7: 000000001047ebd0 000002a1007f7664 0000000000000020 0000000000000000
000002a1007f75a0 rpcmod:svc_tli_kcreate+348 (1, 0, 300043694a8, 2a1007f7788, 2a1007f7780, 78031dd8)
  %l0-3: 00000000101990b4 000002a1007f77a0 0000000000000000 0000000000000000
  %l4-7: 0000000000000000 0000000000000000 0000030004673260 0000000000000000
000002a1007f76b0 nfssrv:nfs_svc+198 (30009be45c0, 10, 8328, 30004673260, 100000, 2a1007f79e0)
  %l0-3: 0000000000000000 0000000000100000 0000000000000000 0000000000000000
  %l4-7: 0000000000000000 0000000010411b58 0000000000000000 0000000000000000
000002a1007f7830 nfs:nfssys+d8 (b, ffbefa9c, ffffffff, 0, 0, 0)
  %l0-3: 0000000000000350 0000000000000000 0000000000000000 0000000000000000
  %l4-7: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
000002a1007f7a30 genunix:loadable_syscall+7c (300000785f8, 10435438, 6a, 4482001a02, 0, 0)
  %l0-3: 0000000000000000 0000000000000000 000000000000000b 00000000ffbefa9c
  %l4-7: 00000000ffffffff 0000000000000000 0000000000000000 0000000000000000

 

syncing file systems... 12 9 done
dumping to /dev/dsk/c1t0d0s1, offset 2097610752

 

SolarisCAT(vmcore.5)> panic

 

panic on cpu 0

panic string:   Deadlock: cycle in blocking chain

==== panic user thread: 0x300043d86c0  pid: 404 ====
cmd: /usr/lib/nfs/nfsd -a 16
t_wchan: 0x30015456ad0  sobj: mutex     owner: 0x2a100085d20    (recursive mutex loop)
top mutex owner: 0x300043d86c0  mutex type: unknown
t_stk: 0x2a1007f7af0  sp: 0x10423811  t_stkbase: 0x2a1007f2000
t_pri: 58(TS)  t_epri: 60  t_lwp: 0x300043d6060  machpcb: 0x2a1007f7af0
t_procp: 0x300043f0008  p_as: 0x30002dfe908  hat: 0x300001495b0  cnum: 0x189a
  size: 2007040  rss: 1441792
bound cpuid: 0  last cpuid: 0 
idle: 307 ticks (3.07 seconds)
start: Mon Jan 29 02:47:41 2007
age: 636230 seconds (7 days 8 hours 43 minutes 50 seconds)
stime: 63639305 (0 seconds later)
syscall: nfssys (0x700b0)
tstate: TS_SLEEP - awaiting an event
tflg:   T_PANIC - thread initiated a system panic
tpflg:  TP_TWAIT - wait to be freed by lwp_wait
tsched: TS_LOAD - thread is in memory
        TS_DONT_SWAP - thread/LWP should not be swapped
pflag:  SLOAD - in core
        SULOAD - u-block in core

 

pc: 0x10048b54  unix:panicsys+0x44:     call    unix:setjmp

 

unix:panicsys+0x44 (0x104241e0, 0x2a1007f7338, 0x1014a328, 0x78002000, 0xe3c2e800, 0xb)
unix:vpanic+0xcc (0x1014a328, 0x2a1007f7338, 0x3c, 0x0, 0x242dd1ebe29f0, 0x30002dfdf30)
unix:panic+0x1c (0x1014a328, 0x10410100, 0x3c, 0x2a100085d20, 0x0, 0x8)
genunix:turnstile_block+0x360 (0x300043d86c0, 0x0, 0x10410100, 0x300043f0008, 0x0, 0x30015456ad0)
unix:mutex_vector_enter+0x3b8 (0x30015456ad0, 0x10415fc0, 0x10424590, 0x2a10007dd20, 0x0, 0x5)
unix:mutex_enter - frame recycled
rpcmod:mir_svc_start+0xc (0x300043694a8, 0x30015456a58, 0x22, 0x1, 0x300043d86c4, 0x300043d86bc)
rpcmod:svc_cots_kstart+0x4c (0x30001e9d880, 0x30001e9d818, 0x30004a39cc8, 0x30001e9d818, 0x81010100, 0xff00)
rpcmod:svc_tli_kcreate+0x348 (0x1, 0x0, 0x300043694a8, 0x2a1007f7788, 0x2a1007f7780, 0x78031dd8)
nfssrv:nfs_svc+0x198 (0x30009be45c0, 0x10, 0x8328, 0x30004673260, 0x100000, 0x2a1007f79e0)
unix:nfs_svc - frame recycled
nfs:nfssys+0xd8 (0xb, 0xffbefa9c, 0xffffffff, 0x0, 0x0, 0x0)
genunix:loadable_syscall+0x7c (0x300000785f8, 0x10435438, 0x6a, 0x4482001a02, 0x0, 0x0)
unix:syscall_trap32+0xa8 (0xb, 0xffbefa9c, 0xffffffffffffffff, 0x0, 0x0, 0x0)
-- switch to user thread's user stack --

 

SolarisCAT(vmcore.5)> panic string

 

panic string:   Deadlock: cycle in blocking chain

 

SolarisCAT(vmcore.5)> panic cpu
panic on cpu 0
panic cpu0 @ 0x1041bc00   panic thread 0x300043d86c0      pri: 58  epri: 60  pid: 404 
  cmd: /usr/lib/nfs/nfsd -a 16
UltraSPARC-III+  impl: 0x15 (CHEETAH+)  ver: 0x23
  upaid: 0  freq: 900MHz  ecache_size: 8388608 (8M)
clock: 900MHz  state: ONLINE  entered: Mon Jan 29 02:45:19 2007
running thread:  0x300043d86c0  idle thread:      0x2a10001fd20
dispatch thread: 0x300043d86c0  interrupt thread: 0x2a100045d20
interrupt stack: 0x2a100077f50  dispatch queue:   0x30000072008
nrunnable: 0    disp pri: 110
current lwp: 0x300043d6060      intr_actv: 0x0          in_prom: 0
runrun: 1       kprunrun: 0     on_intr: 0      base_spl: 0
cpuflg: CPU_RUNNING - cpu running
        CPU_READY - cpu ready for cross-calls
        CPU_EXISTS - cpu is configured
        CPU_ENABLE - cpu enabled for interrupts

 

SolarisCAT(vmcore.5)> panic thread

 

==== panic user thread: 0x300043d86c0  pid: 404 ====
cmd: /usr/lib/nfs/nfsd -a 16
t_wchan: 0x30015456ad0  sobj: mutex     owner: 0x2a100085d20    (recursive mutex loop)
top mutex owner: 0x300043d86c0  mutex type: unknown
t_stk: 0x2a1007f7af0  sp: 0x10423811  t_stkbase: 0x2a1007f2000
t_pri: 58(TS)  t_epri: 60  t_lwp: 0x300043d6060  machpcb: 0x2a1007f7af0
t_procp: 0x300043f0008  p_as: 0x30002dfe908  hat: 0x300001495b0  cnum: 0x189a
  size: 2007040  rss: 1441792
bound cpuid: 0  last cpuid: 0 
idle: 307 ticks (3.07 seconds)
start: Mon Jan 29 02:47:41 2007
age: 636230 seconds (7 days 8 hours 43 minutes 50 seconds)
stime: 63639305 (0 seconds later)
syscall: nfssys (0x700b0)
tstate: TS_SLEEP - awaiting an event
tflg:   T_PANIC - thread initiated a system panic
tpflg:  TP_TWAIT - wait to be freed by lwp_wait
tsched: TS_LOAD - thread is in memory
        TS_DONT_SWAP - thread/LWP should not be swapped
pflag:  SLOAD - in core
        SULOAD - u-block in core

 

pc: 0x10048b54  unix:panicsys+0x44:     call    unix:setjmp

 

unix:panicsys+0x44 (0x104241e0, 0x2a1007f7338, 0x1014a328, 0x78002000, 0xe3c2e800, 0xb)
unix:vpanic+0xcc (0x1014a328, 0x2a1007f7338, 0x3c, 0x0, 0x242dd1ebe29f0, 0x30002dfdf30)
unix:panic+0x1c (0x1014a328, 0x10410100, 0x3c, 0x2a100085d20, 0x0, 0x8)
genunix:turnstile_block+0x360 (0x300043d86c0, 0x0, 0x10410100, 0x300043f0008, 0x0, 0x30015456ad0)
unix:mutex_vector_enter+0x3b8 (0x30015456ad0, 0x10415fc0, 0x10424590, 0x2a10007dd20, 0x0, 0x5)
unix:mutex_enter - frame recycled
rpcmod:mir_svc_start+0xc (0x300043694a8, 0x30015456a58, 0x22, 0x1, 0x300043d86c4, 0x300043d86bc)
rpcmod:svc_cots_kstart+0x4c (0x30001e9d880, 0x30001e9d818, 0x30004a39cc8, 0x30001e9d818, 0x81010100, 0xff00)
rpcmod:svc_tli_kcreate+0x348 (0x1, 0x0, 0x300043694a8, 0x2a1007f7788, 0x2a1007f7780, 0x78031dd8)
nfssrv:nfs_svc+0x198 (0x30009be45c0, 0x10, 0x8328, 0x30004673260, 0x100000, 0x2a1007f79e0)
unix:nfs_svc - frame recycled
nfs:nfssys+0xd8 (0xb, 0xffbefa9c, 0xffffffff, 0x0, 0x0, 0x0)
genunix:loadable_syscall+0x7c (0x300000785f8, 0x10435438, 0x6a, 0x4482001a02, 0x0, 0x0)
unix:syscall_trap32+0xa8 (0xb, 0xffbefa9c, 0xffffffffffffffff, 0x0, 0x0, 0x0)
-- switch to user thread's user stack --

 

SolarisCAT(vmcore.5)> trap
  no trap frames found

 

SolarisCAT(vmcore.5)> thread summary

 

        reference clock = panic_lbolt: 0x3cb0f09                             
   13   threads ran since 1 second before current tick (4 user, 9 kernel)
   48   threads ran since 1 minute before current tick (26 user, 22 kernel)

 

    0*  TS_RUN threads
    3   TS_STOPPED threads (0 user, 3 kernel)
   20   TS_FREE threads (0 user, 20 kernel)
    0   !TS_LOAD (swapped) threads

 

    3*  threads sleeping on a mutex (1 user, 2 kernel)
    0   threads sleeping on a rwlock
  180   threads sleeping on a condition variable (66 user, 114 kernel)
    1   threads sleeping on a semaphore (0 user, 1 kernel)
   61   threads sleeping on a user-level sobj (61 user, 0 kernel)
   18   threads sleeping on a shuttle (door) (18 user, 0 kernel)

 

    0   threads in biowait()

 

    0   threads in dispatch queues

 

  288   total threads in allthreads list (146 user, 142 kernel)
  287   nthread

 

SolarisCAT(vmcore.5)> thread 0x3cb0f09

 

address translation failed for thread: 664 bytes @ 0x3cb0f09
SolarisCAT(vmcore.5)> thread 0x10048b54
address translation failed for proc: 2680 bytes @ 0xc458a308c470e2e8
==== user thread: 0x10048b54 ====
cmd: (unknown)
address translation failed for sobj_ops: 32 bytes @ 0xc407a7e3f05c20d8
t_wchan: 0x80a2200022480010  t_wchan0: 0x913f6000d00420d0 
t_stk: 0xe25fa7d7c4146024  sp: 0xfa07a7df7fff1215  t_stkbase: 0x1000000c4042000
t_pri: -31744  t_epri: -24575  t_lwp: 0xf270a1d0050411c9address translation failed for lwp: 880 bytes @ 0xf270a1d0050411c9

 

address translation failed for cpu: 912 bytes @ 0xc426e00084102008
address translation failed for cpu: 912 bytes @ 0xc426e00084102008

 

idle: 1067068913 ticks (1062238183220 days 19 hours 3 minutes 43.21 seconds)
start: Tue Oct 15 15:08:32 .*0(
age: -361432099016249821 seconds (4183241886762 days 3 hours 37 minutes 1 seconds)
swapped out: -8065911703151373877 (933554595279 days 12 hours 22 minutes 11.82 seconds earlier)
interrupted (pinned) thread: 0xc426e004d05c6270
address translation failed for lwp: 880 bytes @ 0xf270a1d0050411c9
tstate: unknown state
tflg:   T_INTR_THREAD - thread is an interrupt thread
        T_WAKEABLE - thread is blocked, signals enabled
        T_TOMASK - use lwp_sigoldmask on return from signal
        T_TALLOCSTK - thread structure allocated from stk
        T_DONTBLOCK - for lockfs
tpflg:  TP_HOLDLWP - hold thread's lwp
        TP_TWAIT - wait to be freed by lwp_wait
        TP_LWPEXIT - LWP has exited
        TP_EXITLWP - terminate this LWP
        TP_PRVSTOP - thread is virtually stopped via /proc
        TP_MSACCT - collect micro-state accounting information
        TP_STOPPING - thread is executing stop()
        TP_WATCHPT - process has watchpoints in effect
tsched: TS_SWAPENQ - swap thread when it reaches a safe point
        TS_CREATE - setrun() by syslwp_create()

 

pc: 0x8538b038c42c602a  instr: ?
startpc: 0x8410a800c4346024     address translation failed for instruction: 4 bytes @ 0x8410a800c4346024

 

-- no stack --

 

SolarisCAT(vmcore.5)> stack 0x10048b54
0x0(jmpl?) (0xfa07a7df7fff1215, 0x913f6000d00420d0, 0x80a2200022480010, 0xc407a7e3f05c20d8, 0x80a620002268000c, 0xc407a7e3c45c20c8)
-- error reading next frame @ 0xd027a7a3c477afc6 --
SolarisCAT(vmcore.5)>

SolarisCAT(vmcore.5)> etcsystem

forceload drv/sd
forceload drv/ssd
forceload drv/emcp
forceload misc/emcpmp
forceload misc/emcpmpc
forceload misc/emcpmpaa
forceload misc/emcpmpap
forceload misc/emcpsapi
forceload misc/emcpcg
forceload misc/emcphr
forceload misc/emcpgpx
forceload misc/emcpdm
forceload misc/emcpioc
set emcp:bPxEnableInit=1
set lwp_default_stksize=24576
set rpcmod:svc_default_stksize=24576
set fcp:ssfcp_enable_auto_configuration=1
set ssd:ssd_max_throttle=20
forceload drv/ssd

SolarisCAT(vmcore.5)> proc

    addr       pid    ppid    uid    size  rss      swresv    time           command
------------- ------ ------ ------ ---------- -------- -------- ------ ---------
0x30004751550   6357      1      0   16621568  7184384  5136384    426 SMSAGENT 
0x3000451b538   6349      1      0   12673024  3276800  1564672    201 MAGENT 
0x30004750030   6316    253      0    3883008  2588672   966656      4 rpc.ttdbserverd
0x30004604020    558    555      0    1908736  1220608   253952      0 /usr/lib/saf/ttymon
0x30004604ab0    556      1      0    1908736  1130496   253952      1 /usr/lib/saf/ttymon -g -h -p compsun2 console login:  -T sun -d /d
ev/console -l
0x30002dcf530    555      1      0    1908736  1155072   294912      2 /usr/lib/saf/sac -t 300
0x30004605540    533      1      0   12337152  2490368  1376256      2 /usr/lib/dmi/snmpXdmid -s compsun2
0x300043bc000    532      1      0   11722752  2031616  1073152      3 /usr/lib/dmi/dmispd
0x30004520aa0    528    525      0   19439616  2162688   860160     84 mibiisa -r -p 32796
0x300043f0a98    525      1      0    2383872  1556480   475136      2 /usr/lib/snmp/snmpdx -y -c /etc/snmp/conf
0x30004520010    408      1      0   17072128  2490368   868352     10 /usr/dt/bin/dtlogin -daemon
0x300043f0008 404    1     0  2007040  1441792   442368   7574 /usr/lib/nfs/nfsd -a 16
0x30004353570    402      1      0   11485184  2228224  1204224      6 /usr/lib/nfs/mountd
0x30002f4a020    369      1      0    1081344   458752   139264      0 /usr/lib/efcode/sparcv9/efdaemon
0x30002ef4018    360    358      0   13705216  2826240  2113536     22 /usr/sbin/nsrexecd
0x300043f1528    358      1      0   13557760  2187264  1982464     19 /usr/sbin/nsrexecd
0x300043bca90    354    350      0   11706368  1818624   475136      4 htt_server -port 9010 -syslog -message_locale C
0x30003decac8    350      1      0    1097728   745472   114688      0 /usr/lib/im/htt -port 9010 -syslog -message_locale C
0x30004352050    341      1      0   12247040  2301952   696320      3 /usr/sbin/vold
    addr       pid    ppid   uid      size      rss     swresv   time  command
------------- ------ ------ ------ ---------- -------- -------- ------ ---------
0x300043bd520    340      1      0    4636672  1818624   753664      4 /usr/lib/sendmail -bd -q15m
0x30004352ae0    333    331      0    1941504   704512   319488      0 /usr/sadm/lib/smc/bin/smcboot
0x30004220ad8    331      1      0    1941504  1384448   319488      1 /usr/sadm/lib/smc/bin/smcboot
0x30004220048    321      1      0    1105920   737280   172032      3 /usr/lib/utmpd
0x30002dce010    313      1      0    9904128   999424   385024      0 /usr/lib/power/powerd
0x30003ce5550    300      1      0    3317760  1024000  1351680      3 /usr/lib/lpsched
0x300041e0ad0    294      1      0   36470784  2473984  1138688     20 /usr/sbin/nscd
0x30004221568    281      1      0    2342912  1695744   475136      2 /usr/sbin/cron
0x300041e0040    273      1      0   29016064  2285568  2129920     42 /usr/sbin/syslogd
0x300041e1560    265    262      0   12386304  2023424   819200      0 /usr/lib/autofs/automountd
0x30002f4aab0    262      1      0   11935744  1482752   507904      0 /usr/lib/autofs/automountd
0x30003ce4ac0    259      1      1   11018240  1900544   876544      0 /usr/lib/nfs/statd
0x30003dec038    258      1      0    2113536  1359872   548864      2 /usr/lib/nfs/lockd
0x30003ded558    253      1      0    2621440  2039808   909312      8 /usr/sbin/inetd -s
0x30003ce4030    226      1      0    2269184  1548288   720896      5 /usr/lib/netsvc/yp/ypbind -broadcast
0x30003220028    216      1      0   12091392  1441792   827392      3 /usr/sbin/keyserv
0x30002f4b540    213      1      0    2506752  1499136   909312    315 /usr/sbin/rpcbind
0x30002ef5538    143      1      0   22036480  3563520  1974272  14859 /usr/lib/picl/picld
0x30003220ab8     69      1      0   11280384  1671168   737280     20 devfsadmd
0x30003221548     64      1      0   19202048  1433600   647168      4 /usr/lib/sysevent/syseventd
   

addr       pid    ppid   uid      size      rss     swresv   time  command
------------- ------ ------ ------ ---------- -------- -------- ------ ---------
0x30002e0c008      3      0      0     Kernel   Kernel   Kernel 748264 fsflush
0x30002e0ca98      2      0      0     Kernel   Kernel   Kernel      0 pageout
0x30002e0d528      1      0      0     901120   335872   180224     46 /etc/init -
   0x104245b0      0      0      0     Kernel   Kernel   Kernel    170 sched

SolarisCAT(vmcore.5)> exit

끝까지 읽어주셔서 감사합니다..^^

[안다미로]

끝까지 못 읽겠네요.... 글씨가 너무 작아서....

[안다미로]

네 ! , 지금 보니까 글씨가 제대로 보입니다.... 감사합니다...