솔라리스 서버 자동 재부팅 현상에 대한 문의를 드립니다.(시스템 정보 및 로그 첨부)

[최형선]

안녕하세요.

갑자기 밤중에 서버 접속이 안되어 UPTIME으로 확인 결과 시스템이 재부팅 된 것으로 보입니다. 아직 제가 로그파일을 정확히

분석할 능력이 안되어 많은 회원님들께 도움을 구합니다. 관련 정보 및 로그 내용을 카피해서 올려드립니다.

도와주세요.

Hardware : IBM 3550 M2

시스템 정보 

             Oracle Solaris 10 9/10 s10x_u9wos_14a X86
           SunOS Zone9 5.10 Generic_142910-17 i86pc i386 i86pc

           64-bit amd64 applications
                sse4.2 sse4.1 ssse3 popcnt tscp cx16 mon sse3 pause sse2 sse fxsr mmx 
                  cmov amd_sysc cx8 tsc fpu 
                  32-bit i386 applications
                  sse4.2 sse4.1 ssse3 popcnt tscp cx16 mon sse3 pause sse2 sse fxsr mmx 
                  cmov sep cx8 tsc fpu
  서비스 명

• Web Server 구성을 위한 UFS Zone 서버 

Sep 19 22:37:25 Zone9 unix: [ID 836849 kern.notice]
Sep 19 22:37:25 Zone9 ^Mpanic[cpu15]/thread=fffffe8000e9cc60:
Sep 19 22:37:25 Zone9 genunix: [ID 761616 kern.notice] turnstile_block(fffffe83b9cbd440): unowned mutex
Sep 19 22:37:25 Zone9 unix: [ID 100000 kern.notice]
Sep 19 22:37:25 Zone9 genunix: [ID 655072 kern.notice] fffffe8000e9caa0 genunix:turnstile_block+3a1 ()
Sep 19 22:37:25 Zone9 genunix: [ID 655072 kern.notice] fffffe8000e9cb00 unix:mutex_vector_enter+249 ()
Sep 19 22:37:25 Zone9 genunix: [ID 655072 kern.notice] fffffe8000e9cb30 ip:squeue_fill+2c ()
Sep 19 22:37:25 Zone9 genunix: [ID 655072 kern.notice] fffffe8000e9cb40 ip:tcp_timer_callback+24 ()
Sep 19 22:37:25 Zone9 genunix: [ID 655072 kern.notice] fffffe8000e9cb70 genunix:callout_list_expire+65 ()
Sep 19 22:37:25 Zone9 genunix: [ID 655072 kern.notice] fffffe8000e9cba0 genunix:callout_expire+4d ()
Sep 19 22:37:25 Zone9 genunix: [ID 655072 kern.notice] fffffe8000e9cbc0 genunix:callout_execute+19 ()
Sep 19 22:37:25 Zone9 genunix: [ID 655072 kern.notice] fffffe8000e9cc40 genunix:taskq_thread+295 ()
Sep 19 22:37:25 Zone9 genunix: [ID 655072 kern.notice] fffffe8000e9cc50 unix:thread_start+8 ()
Sep 19 22:37:25 Zone9 unix: [ID 100000 kern.notice]
Sep 19 22:37:25 Zone9 genunix: [ID 672855 kern.notice] syncing file systems...
Sep 19 22:37:27 Zone9 genunix: [ID 733762 kern.notice]  3836
Sep 19 22:37:30 Zone9 genunix: [ID 733762 kern.notice]  2160
Sep 19 22:37:31 Zone9 genunix: [ID 733762 kern.notice]  3
Sep 19 22:37:53 Zone9 last message repeated 20 times
Sep 19 22:37:54 Zone9 genunix: [ID 622722 kern.notice]  done (not all i/o completed)
Sep 19 22:37:55 Zone9 genunix: [ID 111219 kern.notice] dumping to /dev/dsk/c0t0d0s1, offset 429391872, content: kernel
Sep 19 22:38:03 Zone9 genunix: [ID 100000 kern.notice]
Sep 19 22:38:03 Zone9 genunix: [ID 665016 kern.notice] ^M100% done: 362649 pages dumped,
Sep 19 22:38:03 Zone9 genunix: [ID 851671 kern.notice] dump succeeded
Sep 19 22:41:06 Zone9 genunix: [ID 540533 kern.notice] ^MSunOS Release 5.10 Version Generic_142910-17 64-bit
Sep 19 22:41:06 Zone9 genunix: [ID 877030 kern.notice] Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserve
d.
Sep 19 22:41:06 Zone9 unix: [ID 126719 kern.info] features: 2f7fffdf<sse4_2,sse4_1,ssse3,cpuid,mwait,cmp,cx16,sse3,nx,asysc,htt,s
se2,sse,sep,pat,cx8,pae,mca,mmx,cmov,pge,mtrr,msr,tsc,lgpg>
Sep 19 22:41:06 Zone9 unix: [ID 168242 kern.info] mem = 8377056K (0x1ff4b8000)
Sep 19 22:41:06 Zone9 rootnex: [ID 466748 kern.info] root nexus = i86pc
Sep 19 22:41:06 Zone9 iommulib: [ID 321598 kern.info] NOTICE: iommulib_nexus_register: rootnex-1: Succesfully registered NEXUS i8
6pc nexops=fffffffffbcf58c0
Sep 19 22:41:06 Zone9 rootnex: [ID 349649 kern.info] pseudo0 at root
Sep 19 22:41:06 Zone9 genunix: [ID 936769 kern.info] pseudo0 is /pseudo
Sep 19 22:41:06 Zone9 rootnex: [ID 349649 kern.info] scsi_vhci0 at root
Sep 19 22:41:06 Zone9 genunix: [ID 936769 kern.info] scsi_vhci0 is /scsi_vhci
Sep 19 22:41:06 Zone9 rootnex: [ID 349649 kern.info] isa0 at root
Sep 19 22:41:06 Zone9 pcplusmp: [ID 658230 kern.info] NOTICE: apic: local nmi: 255 0 0 1
Sep 19 22:41:06 Zone9 pcplusmp: [ID 119066 kern.info] pcplusmp: irq 0x9 vector 0x81 ioapic 0x8 intin 0x9 is bound to cpu 1
Sep 19 22:41:06 Zone9 pcplusmp: [ID 119066 kern.info] pcplusmp: irq 0xb vector 0xd1 ioapic 0x8 intin 0xb is bound to cpu 2
Sep 19 22:41:06 Zone9 pseudo: [ID 129642 kern.info] pseudo-device: ppm0
Sep 19 22:41:06 Zone9 genunix: [ID 936769 kern.info] ppm0 is /pseudo/ppm@0
Sep 19 22:41:06 Zone9 rootnex: [ID 349649 kern.info] npe0 at root: space 0 offset 0
Sep 19 22:41:06 Zone9 genunix: [ID 936769 kern.info] npe0 is /pci@0,0
Sep 19 22:41:06 Zone9 npe: [ID 236367 kern.info] PCI Express-device: pci8086,3a40@1c, pcieb4
Sep 19 22:41:06 Zone9 genunix: [ID 936769 kern.info] pcieb4 is /pci@0,0/pci8086,3a40@1c
Sep 19 22:41:06 Zone9 pcplusmp: [ID 272826 kern.info] pcplusmp: pciex1000,79 (mr_sas) instance 0 irq 0x30 vector 0x40 ioapic 0xff
 intin 0xff is bound to cpu 3
Sep 19 22:41:06 Zone9 pcieb: [ID 586369 kern.info] PCIE-device: pci1014,3b2@0, mr_sas0
Sep 19 22:41:06 Zone9 npe: [ID 236367 kern.info] PCI Express-device: pci1014,3b2@0, mr_sas0
Sep 19 22:41:06 Zone9 genunix: [ID 936769 kern.info] mr_sas0 is /pci@0,0/pci8086,3a40@1c/pci1014,3b2@0
Sep 19 22:41:06 Zone9 scsi: [ID 193665 kern.info] sd1 at mr_sas0: target 0 lun 0
Sep 19 22:41:06 Zone9 genunix: [ID 936769 kern.info] sd1 is /pci@0,0/pci8086,3a40@1c/pci1014,3b2@0/sd@0,0
Sep 19 22:41:06 Zone9 genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3a40@1c/pci1014,3b2@0/sd@0,0 (sd1) online
Sep 19 22:41:10 Zone9 genunix: [ID 308332 kern.info] root on /pci@0,0/pci8086,3a40@1c/pci1014,3b2@0/sd@0,0:a fstype ufs
Sep 19 22:41:10 Zone9 unix: [ID 190185 kern.info] SMBIOS v2.5 loaded (3060 bytes)
Sep 19 22:41:10 Zone9 unix: [ID 517869 kern.info] NOTICE: cpu_acpi: _PSS package not found.
Sep 19 22:41:10 Zone9 unix: [ID 388705 kern.warning] WARNING: cpu_acpi: error parsing _PSS for CPU 0
Sep 19 22:41:10 Zone9 unix: [ID 196753 kern.warning] WARNING: cpupm_init: processor 0: unable to initialize P-state support
Sep 19 22:41:10 Zone9 unix: [ID 374799 kern.info] NOTICE: cpu_acpi: _CST eval‎uate failure
Sep 19 22:41:10 Zone9 unix: [ID 518192 kern.warning] WARNING: cpu_acpi: error parsing _CST for CPU 0
Sep 19 22:41:10 Zone9 unix: [ID 726263 kern.info] NOTICE: cpu_idle_init: Failed to cache ACPI C-state data
Sep 19 22:41:10 Zone9 unix: [ID 763091 kern.warning] WARNING: cpupm_init: processor 0: unable to initialize C-state support
Sep 19 22:41:10 Zone9 genunix: [ID 408114 kern.info] /cpus (cpunex0) online
Sep 19 22:41:10 Zone9 pseudo: [ID 129642 kern.info] pseudo-device: dld0
Sep 19 22:41:10 Zone9 genunix: [ID 936769 kern.info] dld0 is /pseudo/dld@0
Sep 19 22:41:10 Zone9 pcplusmp: [ID 272826 kern.info] pcplusmp: pciexclass,060400 (pcieb) instance 0 irq 0x31 vector 0x82 ioapic
0xff intin 0xff is bound to cpu 4
Sep 19 22:41:10 Zone9 npe: [ID 236367 kern.info] PCI Express-device: pci8086,3408@1, pcieb0

[백승찬]

우선 # ls -l /var/crash/`hostname` 디렉토리의 System Crash Dump 파일 부터 확인해 보셔야 할 것 같습니다.

[백승찬]

CPU0 쪽에 문제가 발생해서... System Crash Dump가 떨어진것 같습니다.

[백승찬]

만약 지속적으로 이런 일이 발생이 된다면 CPU와 연결된 MEM도 같이 봐 주시고.... CPU 보드(또는 MEM)를 교체 하시는 것이 좋을 듯합니다.

[백승찬]

우선은 System Crash Dump 파일 부터 분석을 해 보셔야 할 것 같습니다.