|
/var/adm/messages 파일에는 리부팅전의 내용은 없습니다.
SCAT으로 msgbuf를 봤을때 NOTICE: alloc: /usr : file system full 내용이 있던데...이것이 원인이 될수 있는건지 아님 오래전에 발생했고 시스템정리를 했을수 있는 시간이 있었던 건지 (헐 정리가 안됨니다.-- msgbuf로 확인하는 내용이 리부팅 바로 전것만 가지고 있는건지 궁금합니다.)
df -k로 봤을때 /usr은 67%정도로 800M정도 여유가 있었습니다.
리부팅후에 누가 파일정리를 한건쥐는 잘 모르겠습니다.
/usr의 full로 인해 다음과 같은 메시지를 발생시키고 리부팅 될수도 있나요??
CPU를 교체 해야 맞는 건가요??
-----------------------------------------------
========================= CPUs =========================
Run Ecache CPU CPU
Brd CPU Module MHz MB Impl. Mask
--- --- ------- ----- ------ ------ ----
3 6 0 400 8.0 US-II 10.0
3 7 1 400 8.0 US-II 10.0
5 10 0 400 8.0 US-II 10.0
5 11 1 400 8.0 US-II 10.0
========================= Memory =========================
Intrlv. Intrlv.
Brd Bank MB Status Condition Speed Factor With
--- ----- ---- ------- ---------- ----- ------- -------
3 0 1024 Active OK 60ns 2-way A
5 0 1024 Active OK 60ns 2-way A
-----------------------------------------------------------------
======================================================================
SolarisCAT(vmcore.0)> msgbuf
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
NOTICE: alloc: /usr: file system full
WARNING: [AFT1] Uncorrectable Memory Error on CPU7 Data access at TL=0, errID 0x002980c6.f457aec3
AFSR 0x00000000.80200000<PRIV,UE> AFAR 0x00000000.7a4f4758
AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC 0x1008ca94
UDBH 0x00a0 UDBH.ESYND 0xa0 UDBL 0x0203<UE> UDBL.ESYND 0x03
UDBL Syndrome 0x3 Memory Module Board 5 J3100 J3200 J3300 J3400 J3500 J3600 J3700 J3800
WARNING: [AFT1] errID 0x002980c6.f457aec3 Syndrome 0x3 indicates that this may not be a memory module problem
[AFT2] errID 0x002980c6.f457aec3 PA=0x00000000.7a4f4758
E$tag 0x00000000.0c400f49 E$State: Shared E$parity 0x06
[AFT2] E$Data (0x00): 0x00000300.0207d540
[AFT2] E$Data (0x08): 0x00000000.116b1880
[AFT2] E$Data (0x10): 0x00000000.113a4740
[AFT2] E$Data (0x18): 0x00000000.313a4740 *Bad* PSYND=0x00ff
[AFT2] E$Data (0x20): 0x00000000.113a4740
[AFT2] E$Data (0x28): 0x00000000.113a4740
[AFT2] E$Data (0x30): 0x00000000.00000000
[AFT2] E$Data (0x38): 0x00000000.00000000
WARNING: [AFT1] CP event on CPU6 (caused Data access error on CPU7), errID 0x002980c6.f457aec3
AFSR 0x00000000.01000008<CP> AFAR 0x00000000.7a4f4758
AFSR.PSYND 0x0008(Score 95) AFSR.ETS 0x00
UDBH 0x00a0 UDBH.ESYND 0xa0 UDBL 0x00a0 UDBL.ESYND 0xa0
[AFT2] errID 0x002980c6.f457aec3 PA=0x00000000.7a4f4758
E$tag 0x00000000.1d400f49 E$State: Owner E$parity 0x0e
[AFT2] E$Data (0x00): 0x00000300.0207d540
[AFT2] E$Data (0x08): 0x00000000.116b1880
[AFT2] E$Data (0x10): 0x00000000.113a4740
[AFT2] E$Data (0x18): 0x00000000.313a4740 *Bad* PSYND=0x0008
[AFT2] E$Data (0x20): 0x00000000.113a4740
[AFT2] E$Data (0x28): 0x00000000.113a4740
[AFT2] E$Data (0x30): 0x00000000.00000000
[AFT2] E$Data (0x38): 0x00000000.00000000
panic[cpu7]/thread=300003396a0: [AFT1] errID 0x002980c6.f457aec3 UE Error(s)
See previous message(s) for details
syncing file systems... 2panic[cpu7]/thread=2a1000abd60: panic sync timeout
dumping to /dev/dsk/c0t0d0s1, offset 419495936
SolarisCAT(vmcore.0)> panic thread
==== panic kernel thread: 0x300003396a0 pid: 3 on cpu: 7 ====
cmd: fsflush
SolarisCAT(vmcore.0)> panic
panic on cpu 7
panic string: [AFT1] errID 0x002980c6.f457aec3 UE Error(s)
See previous message(s) for details
==== panic kernel thread: 0x300003396a0 pid: 3 on cpu: 7 ====
cmd: fsflush
t_stk: 0x2a1001fdaf0 sp: 0x2a1001fca61 t_stkbase: 0x2a1001fa000
t_pri: 60(SYS) pctcpu: 1.964327 t_lwp: 0x3000033b478 machpcb: 0x2a1001fdaf0
t_procp: 0x30000336008(proc_fsflush) p_as: 0x10422d60(kas)
last cpuid: 7
idle: 500 ticks (5.00 seconds)
start: Mon Dec 6 08:13:02 2004
age: 11681725 seconds (135 days 4 hours 55 minutes 25 seconds)
stime: 5097 (135 days 4 hours 52 minutes 18.87 seconds earlier)
syscall: sys#0 (0x0)
tstate: TS_ONPROC - thread is being run on a processor
tflg: none set
tpflg: none set
tsched: TS_LOAD - thread is in memory
TS_DONT_SWAP - thread/LWP should not be swapped
pflag: SSYS - system resident process
SLOAD - in core
SLOCK - process cannot be swapped
SULOAD - u-block in core
SNOWAIT - children never become zombies
pc: 0x1001063c unix:complete_panic+0x20: call unix:setjmp
unix:complete_panic+0x20 (0x10462400, 0x2a1001fd7a0, 0xc, 0x0, 0x0, 0x0)
unix:do_panic+0x16c (0x10408000, 0x2a1001fd7a0, 0x0, 0x0, 0x0, 0x1011d7ac)
genunix:vcmn_err+0x18 (0x3, 0x2a1001fd558, 0x2a1001fd7a0, 0x3, 0x81010100, 0xff00)
SUNW,UltraSPARC-II:cpu_aflt_log+0x4f0 (0x2a1001fd55e, 0x1, 0x1011d788, 0x2a1001fd6e8, 0x2a1001fd5ab, 0x1011d7b0)
SUNW,UltraSPARC-II:cpu_async_error+0x884 (0x80200000, 0x7a4f4750, 0x10460ae8, 0x0, 0x80200000, 0x2a1001fd970)
unix:prom_rtt+0x0 (0x113a4740, 0x1, 0x20, 0x0, 0x0, 0x0)
-- prom_rtt regs data rp: 0x2a1001fd970
pc: 0x1008ca94 genunix:fsflush+0x3f0: ldub [%o0 + 0x47], %g4
npc: 0x1008ca98 genunix:fsflush+0x3f4: andcc %g4, 0x80 ( btst %g4, 0x80 )
global: %g1 0x113a46e0
%g2 0x104a8000 %g3 0
%g4 0x8771 %g5 0x11f56680
%g6 0 %g7 0x300003396a0
out: %o0 0x113a4740 %o1 0x1
%o2 0x20 %o3 0
%o4 0 %o5 0
%sp 0x2a1001fd211 %o7 0x1008ca74
loc: %l0 0x30000371fc8 %l1 0x30000371ff8
%l2 0x113a4740 %l3 0xbb8
%l4 0x30000397fd8 %l5 0x10451a08
%l6 0x6 %l7 0x10454098
in: %i0 0x8771 %i1 0
%i2 0x1041e1e0 %i3 0xa27f
%i4 0x30004de9c28 %i5 0x113a4740
%fp 0x2a1001fd2f1 %i7 0x1002d480
<trap>genunix:fsflush+0x3f0 (0x8771, 0x0, 0x1041e1e0, 0xa27f, 0x30004de9c28, 0x113a4740)
unix:thread_start+0x4 (0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
-- end of kernel thread's stack --
첫댓글 메모리 error 군요.. 메모리 error 지만 CPU 도 의심이 듭니다. sunVTS 프로그램으로 시스템 하드 웨어를 정밀히 진단해 보세요..
/usr filesystem full 로 인해 발생한 것은 아니구요.. 메세지를 보니 EPx500 장비인듯 한데요... 제 경험으로 라면, CPU 6번을 교체하심이 맞을 듯 합니다.. 얼핏 봐선 memory 같지만 CPU 6번에 장애로 인해 CPU7번에서 panic이 발생을 했네요.. ^^
6,7번 모두 가는게 좋을듯 싶네요. AFT 0 과 2는 일어날수 잇어요 하루에 3번미만일경우는 교체안해도 되나 1일 3회 발생시 교체권고, AFT 1은 즉시교체요망됩니다.
cpu 6번 만 교체 하세요. cpu 7번은 6번 애러 때문에 생긴거니까 괜찮을거 같네요
개인적으론 저도 6번만 교체 했으면 음. 만일 제가 담당하는 싸이트라면. CPU 6번, 7(CP Event), Board 5 (메모리모듈)교체 하겠습니다. 음. 모든장애요인은 6번에의해 발생했으나. 제 경험상으론 받드시 또 문제가 있을듯.. 선택은 자유지만 한번에 깔끔하게 끝내려면 ...