[질문] core분석인데 ...의견좀 부탁드립니다.

[김상식]

/var/adm/messages 파일에는 리부팅전의 내용은 없습니다.

SCAT으로 msgbuf를 봤을때 NOTICE: alloc: /usr : file system full 내용이 있던데...이것이 원인이 될수 있는건지 아님 오래전에 발생했고 시스템정리를 했을수 있는 시간이 있었던 건지 (헐 정리가 안됨니다.-- msgbuf로 확인하는 내용이 리부팅 바로 전것만 가지고 있는건지 궁금합니다.)

df -k로 봤을때 /usr은 67%정도로 800M정도 여유가 있었습니다.

리부팅후에 누가 파일정리를 한건쥐는 잘 모르겠습니다.

/usr의 full로 인해 다음과 같은 메시지를 발생시키고 리부팅 될수도 있나요??

 CPU를 교체 해야 맞는 건가요??

-----------------------------------------------

========================= CPUs =========================

                    Run   Ecache   CPU    CPU
Brd  CPU   Module   MHz     MB    Impl.   Mask
---  ---  -------  -----  ------  ------  ----
 3     6     0      400     8.0   US-II    10.0
 3     7     1      400     8.0   US-II    10.0
 5    10     0      400     8.0   US-II    10.0
 5    11     1      400     8.0   US-II    10.0

========================= Memory =========================

                                              Intrlv.  Intrlv.
Brd   Bank   MB    Status   Condition  Speed   Factor   With
---  -----  ----  -------  ----------  -----  -------  -------
 3     0    1024   Active      OK       60ns    2-way     A
 5     0    1024   Active      OK       60ns    2-way     A

-----------------------------------------------------------------

======================================================================
SolarisCAT(vmcore.0)> msgbuf

NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
WARNING: [AFT1] Uncorrectable Memory Error on CPU7 Data access at TL=0, errID 0x002980c6.f457aec3
    AFSR 0x00000000.80200000<PRIV,UE> AFAR 0x00000000.7a4f4758
    AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC 0x1008ca94
    UDBH 0x00a0 UDBH.ESYND 0xa0 UDBL 0x0203<UE> UDBL.ESYND 0x03
    UDBL Syndrome 0x3 Memory Module Board 5 J3100 J3200 J3300 J3400 J3500 J3600 J3700 J3800
WARNING: [AFT1] errID 0x002980c6.f457aec3 Syndrome 0x3 indicates that this may not be a memory module problem
[AFT2] errID 0x002980c6.f457aec3 PA=0x00000000.7a4f4758
    E$tag 0x00000000.0c400f49 E$State: Shared E$parity 0x06
[AFT2] E$Data (0x00): 0x00000300.0207d540
[AFT2] E$Data (0x08): 0x00000000.116b1880
[AFT2] E$Data (0x10): 0x00000000.113a4740
[AFT2] E$Data (0x18): 0x00000000.313a4740 *Bad* PSYND=0x00ff
[AFT2] E$Data (0x20): 0x00000000.113a4740
[AFT2] E$Data (0x28): 0x00000000.113a4740
[AFT2] E$Data (0x30): 0x00000000.00000000
[AFT2] E$Data (0x38): 0x00000000.00000000
WARNING: [AFT1] CP event on CPU6 (caused Data access error on CPU7), errID 0x002980c6.f457aec3
    AFSR 0x00000000.01000008<CP> AFAR 0x00000000.7a4f4758
    AFSR.PSYND 0x0008(Score 95) AFSR.ETS 0x00
    UDBH 0x00a0 UDBH.ESYND 0xa0 UDBL 0x00a0 UDBL.ESYND 0xa0
[AFT2] errID 0x002980c6.f457aec3 PA=0x00000000.7a4f4758
    E$tag 0x00000000.1d400f49 E$State: Owner E$parity 0x0e
[AFT2] E$Data (0x00): 0x00000300.0207d540
[AFT2] E$Data (0x08): 0x00000000.116b1880
[AFT2] E$Data (0x10): 0x00000000.113a4740
[AFT2] E$Data (0x18): 0x00000000.313a4740 *Bad* PSYND=0x0008
[AFT2] E$Data (0x20): 0x00000000.113a4740
[AFT2] E$Data (0x28): 0x00000000.113a4740
[AFT2] E$Data (0x30): 0x00000000.00000000
[AFT2] E$Data (0x38): 0x00000000.00000000
panic[cpu7]/thread=300003396a0: [AFT1] errID 0x002980c6.f457aec3 UE Error(s)
    See previous message(s) for details
syncing file systems... 2panic[cpu7]/thread=2a1000abd60: panic sync timeout
dumping to /dev/dsk/c0t0d0s1, offset 419495936

SolarisCAT(vmcore.0)> panic thread
==== panic kernel thread: 0x300003396a0  pid: 3  on cpu: 7 ====
cmd: fsflush

SolarisCAT(vmcore.0)> panic
panic on cpu 7
panic string:   [AFT1] errID 0x002980c6.f457aec3 UE Error(s)
    See previous message(s) for details
==== panic kernel thread: 0x300003396a0  pid: 3  on cpu: 7 ====
cmd: fsflush

t_stk: 0x2a1001fdaf0  sp: 0x2a1001fca61  t_stkbase: 0x2a1001fa000
t_pri: 60(SYS)  pctcpu: 1.964327  t_lwp: 0x3000033b478  machpcb: 0x2a1001fdaf0
t_procp: 0x30000336008(proc_fsflush)  p_as: 0x10422d60(kas)
last cpuid: 7 
idle: 500 ticks (5.00 seconds)
start: Mon Dec  6 08:13:02 2004
age: 11681725 seconds (135 days 4 hours 55 minutes 25 seconds)
stime: 5097 (135 days 4 hours 52 minutes 18.87 seconds earlier)
syscall: sys#0 (0x0)
tstate: TS_ONPROC - thread is being run on a processor
tflg:   none set
tpflg:  none set
tsched: TS_LOAD - thread is in memory
        TS_DONT_SWAP - thread/LWP should not be swapped
pflag:  SSYS - system resident process
        SLOAD - in core
        SLOCK - process cannot be swapped
        SULOAD - u-block in core
        SNOWAIT - children never become zombies

pc: 0x1001063c  unix:complete_panic+0x20:       call    unix:setjmp

unix:complete_panic+0x20 (0x10462400, 0x2a1001fd7a0, 0xc, 0x0, 0x0, 0x0)
unix:do_panic+0x16c (0x10408000, 0x2a1001fd7a0, 0x0, 0x0, 0x0, 0x1011d7ac)
genunix:vcmn_err+0x18 (0x3, 0x2a1001fd558, 0x2a1001fd7a0, 0x3, 0x81010100, 0xff00)
SUNW,UltraSPARC-II:cpu_aflt_log+0x4f0 (0x2a1001fd55e, 0x1, 0x1011d788, 0x2a1001fd6e8, 0x2a1001fd5ab, 0x1011d7b0)
SUNW,UltraSPARC-II:cpu_async_error+0x884 (0x80200000, 0x7a4f4750, 0x10460ae8, 0x0, 0x80200000, 0x2a1001fd970)
unix:prom_rtt+0x0 (0x113a4740, 0x1, 0x20, 0x0, 0x0, 0x0)
-- prom_rtt regs data  rp: 0x2a1001fd970
pc:  0x1008ca94 genunix:fsflush+0x3f0:  ldub    [%o0 + 0x47], %g4
npc: 0x1008ca98 genunix:fsflush+0x3f4:  andcc   %g4, 0x80       ( btst  %g4, 0x80 )
  global:                       %g1         0x113a46e0
        %g2         0x104a8000  %g3                  0
        %g4             0x8771  %g5         0x11f56680
        %g6                  0  %g7      0x300003396a0
  out:  %o0         0x113a4740  %o1                0x1
        %o2               0x20  %o3                  0
        %o4                  0  %o5                  0
        %sp      0x2a1001fd211  %o7         0x1008ca74
  loc:  %l0      0x30000371fc8  %l1      0x30000371ff8 
        %l2         0x113a4740  %l3              0xbb8 
        %l4      0x30000397fd8  %l5         0x10451a08 
        %l6                0x6  %l7         0x10454098 
  in:   %i0             0x8771  %i1                  0 
        %i2         0x1041e1e0  %i3             0xa27f 
        %i4      0x30004de9c28  %i5         0x113a4740 
        %fp      0x2a1001fd2f1  %i7         0x1002d480 
<trap>genunix:fsflush+0x3f0 (0x8771, 0x0, 0x1041e1e0, 0xa27f, 0x30004de9c28, 0x113a4740)
unix:thread_start+0x4 (0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
-- end of kernel thread's stack --

[조재구]

메모리 error 군요.. 메모리 error 지만 CPU 도 의심이 듭니다. sunVTS 프로그램으로 시스템 하드 웨어를 정밀히 진단해 보세요..

[꼬모]

/usr filesystem full 로 인해 발생한 것은 아니구요.. 메세지를 보니 EPx500 장비인듯 한데요... 제 경험으로 라면, CPU 6번을 교체하심이 맞을 듯 합니다.. 얼핏 봐선 memory 같지만 CPU 6번에 장애로 인해 CPU7번에서 panic이 발생을 했네요.. ^^

[taeo]

6,7번 모두 가는게 좋을듯 싶네요. AFT 0 과 2는 일어날수 잇어요 하루에 3번미만일경우는 교체안해도 되나 1일 3회 발생시 교체권고, AFT 1은 즉시교체요망됩니다.

[우탄이]

cpu 6번 만 교체 하세요. cpu 7번은 6번 애러 때문에 생긴거니까 괜찮을거 같네요

[스머프]

개인적으론 저도 6번만 교체 했으면 음. 만일 제가 담당하는 싸이트라면. CPU 6번, 7(CP Event), Board 5 (메모리모듈)교체 하겠습니다. 음. 모든장애요인은 6번에의해 발생했으나. 제 경험상으론 받드시 또 문제가 있을듯.. 선택은 자유지만 한번에 깔끔하게 끝내려면 ...