<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16640" name=GENERATOR></HEAD>
<BODY>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>On a two node
cluster I got a reboot (core dump) on the first node.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>The
/var/log/messages doesn't show anything wrong, but running</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>crash on the code
dump shows that ocfs2 panicked the system.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>Nodes are SuSE Sles
10 SP1 systems. I believe node 1 had a </SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>backup routine
(Veritas) running on it at the time. </SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>Any ideas on what
happened ?</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=827040714-10062008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>thank
you,</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>
charlie</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=827040714-10062008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN
class=827040714-10062008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>node 1 system
info:</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=827040714-10062008>----------------------------</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>Jun 4 11:48:03
bustech-bu kernel: OCFS2 Node Manager 1.2.5-SLES-r2997 Tue Mar 27 16:33:19 EDT
2007 (build sles)<BR>Jun 4 11:48:03 bustech-bu kernel: OCFS2 DLM
1.2.5-SLES-r2997 Tue Mar 27 16:33:19 EDT 2007 (build sles)<BR>Jun 4
11:48:03 bustech-bu kernel: OCFS2 DLMFS 1.2.5-SLES-r2997 Tue Mar 27 16:33:19 EDT
2007 (build sles)<BR></SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=827040714-10062008> </DIV></SPAN></FONT></SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>node 1 boot.msg
file. looks like the reboot was on June 4, 2008 at
11:40</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=827040714-10062008>---------------------------------------------------------------------------------------------------------</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>INIT: <BR>Boot
logging started on /dev/tty1(/dev/console) at Wed Jun 4 11:40:23
2008</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>Master Resource
Control: previous runlevel: N, switching to runlevel: 1<BR>Starting irqbalance
unused<BR>Saving 1979 MB crash dump to /var/log/dump/2008-06-04-11:40
...<BR>Entering runlevel: 1</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN
class=827040714-10062008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>node 1
/var/log/messages</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=827040714-10062008>-------------------------------------</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>Jun 4 08:37:21
bustech-bu syslog-ng[3986]: STATS: dropped 0<BR>Jun 4 09:15:01 bustech-bu
run-crons[12180]: time.cron returned 1<BR>Jun 4 09:37:21 bustech-bu
syslog-ng[3986]: STATS: dropped 0<BR>Jun 4 10:15:01 bustech-bu
run-crons[14123]: time.cron returned 1<BR>Jun 4 10:37:21 bustech-bu
syslog-ng[3986]: STATS: dropped 0<BR>Jun 4 11:15:01 bustech-bu
run-crons[16066]: time.cron returned 1<BR>Jun 4 11:37:21 bustech-bu
syslog-ng[3986]: STATS: dropped 0</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=827040714-10062008>
**reboot here. note no previous errors**</DIV>
<DIV>Jun 4 11:46:22 bustech-bu syslog-ng[4018]: syslog-ng version 1.6.8
starting <BR>Jun 4 11:46:22 bustech-bu ifup:
lo <BR></DIV></SPAN></FONT>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008></SPAN></FONT><FONT
face=Arial size=2><SPAN class=827040714-10062008></SPAN></FONT><FONT face=Arial
size=2><SPAN class=827040714-10062008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>node 1 crash
info</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=827040714-10062008>------------------------</SPAN></FONT></DIV><FONT><SPAN
class=827040714-10062008>
<DIV><FONT face=Arial size=2>crash> bt<BR>PID: 13
TASK: dff1f670 CPU: 3 COMMAND: "events/3"<BR> #0
[dff21f08] crash_kexec at c013bb1a<BR> #1 [dff21f4c] panic at
c0120172<BR> #2 [dff21f68] o2quo_fence_self at fb8cc399<BR> #3
[dff21f70] run_workqueue at c012de27<BR> #4 [dff21f8c] worker_thread at
c012e754<BR> #5 [dff21fcc] kthread at c0130e77<BR> #6 [dff21fe8]
kernel_thread_helper at c0102003<BR>crash> </FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial><FONT size=2>N<SPAN class=827040714-10062008>ode 2
/var/log/messages. it looks like this node saw node 1 go away and come
back</SPAN></FONT></FONT></DIV>
<DIV><FONT face=Arial><FONT size=2><SPAN
class=827040714-10062008>-----------------------------------------------------------------------------------------------------------------------------</SPAN></FONT></FONT></SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008>Jun 4 10:30:01
CN2 run-crons[1587]: time.cron returned 1<BR>Jun 4 11:07:27 CN2
syslog-ng[4054]: STATS: dropped 0<BR>Jun 4 11:30:01 CN2 run-crons[3546]:
time.cron returned 1<BR>Jun 4 11:41:33 CN2 kernel: o2net: connection to
node bustech-bu (num 0) at 192.168.200.10:7777 has been idle for 120.0 seconds,
shutting it down.<BR>Jun 4 11:41:33 CN2 kernel:
(0,1):o2net_idle_timer:1426 here are some times that might help debug the
situation: (tmr 1212604773.841590 now 1212604893.850342 dr 1212604773.841583 adv
1212604773.841590:1212604773.841591 func (04f07b3d:505)
1212595634.502257:1212595634.502260)<BR>Jun 4 11:41:33 CN2 kernel: o2net:
no longer connected to node bustech-bu (num 0) at
192.168.200.10:7777<BR>Jun 4 11:41:38 CN2 kernel:
(5717,1):dlm_get_lock_resource:920 5CA2BC69EF1C446B97521FEB7175EF1C:$RECOVERY:
at least one node (0) torecover before lock mastery can begin<BR>Jun 4
11:41:38 CN2 kernel: (5717,1):dlm_get_lock_resource:954
5CA2BC69EF1C446B97521FEB7175EF1C: recovery map is not empty, but must master
$RECOVERY lock now<BR>Jun 4 11:41:38 CN2 kernel:
(5728,3):dlm_get_lock_resource:920 8057F00ED41A4507A24B6A4EF0211F1D:$RECOVERY:
at least one node (0) torecover before lock mastery can begin<BR>Jun 4
11:41:38 CN2 kernel: (5728,3):dlm_get_lock_resource:954
8057F00ED41A4507A24B6A4EF0211F1D: recovery map is not empty, but must master
$RECOVERY lock now<BR>Jun 4 11:41:39 CN2 kernel:
(6117,3):dlm_get_lock_resource:920 mas:$RECOVERY: at least one node (0)
torecover before lock mastery can begin<BR>Jun 4 11:41:39 CN2 kernel:
(6117,3):dlm_get_lock_resource:954 mas: recovery map is not empty, but must
master $RECOVERY lock now<BR>Jun 4 11:41:40 CN2 kernel:
(5706,1):dlm_get_lock_resource:920 6FFB00A1F4F94113B6748BC33CA47F83:$RECOVERY:
at least one node (0) torecover before lock mastery can begin<BR>Jun 4
11:41:40 CN2 kernel: (5706,1):dlm_get_lock_resource:954
6FFB00A1F4F94113B6748BC33CA47F83: recovery map is not empty, but must master
$RECOVERY lock now<BR>Jun 4 11:41:40 CN2 kernel:
(5770,3):dlm_get_lock_resource:920 E2A008B35C664DDC9FF850F59B0E122F:$RECOVERY:
at least one node (0) torecover before lock mastery can begin<BR>Jun 4
11:41:40 CN2 kernel: (5770,3):dlm_get_lock_resource:954
E2A008B35C664DDC9FF850F59B0E122F: recovery map is not empty, but must master
$RECOVERY lock now<BR>Jun 4 11:41:41 CN2 kernel:
(5759,3):dlm_get_lock_resource:920 DD202255EE9C419781F4E61DE6E33CFE:$RECOVERY:
at least one node (0) torecover before lock mastery can begin<BR>Jun 4
11:41:41 CN2 kernel: (5759,3):dlm_get_lock_resource:954
DD202255EE9C419781F4E61DE6E33CFE: recovery map is not empty, but must master
$RECOVERY lock now<BR>Jun 4 11:48:08 CN2 kernel: o2net: connected to node
bustech-bu (num 0) at 192.168.200.10:7777<BR>Jun 4 11:48:11 CN2 kernel:
ocfs2_dlm: Node 0 joins domain 6FFB00A1F4F94113B6748BC33CA47F83<BR>Jun 4
11:48:11 CN2 kernel: ocfs2_dlm: Nodes in domain
("6FFB00A1F4F94113B6748BC33CA47F83"): 0 1 <BR>Jun 4 11:48:15 CN2 kernel:
ocfs2_dlm: Node 0 joins domain 5CA2BC69EF1C446B97521FEB7175EF1C<BR>Jun 4
11:48:15 CN2 kernel: ocfs2_dlm: Nodes in domain
("5CA2BC69EF1C446B97521FEB7175EF1C"): 0 1 <BR>Jun 4 11:48:20 CN2 kernel:
ocfs2_dlm: Node 0 joins domain 8057F00ED41A4507A24B6A4EF0211F1D<BR>Jun 4
11:48:20 CN2 kernel: ocfs2_dlm: Nodes in domain
("8057F00ED41A4507A24B6A4EF0211F1D"): 0 1 <BR>Jun 4 11:48:24 CN2 kernel:
ocfs2_dlm: Node 0 joins domain DD202255EE9C419781F4E61DE6E33CFE<BR>Jun 4
11:48:24 CN2 kernel: ocfs2_dlm: Nodes in domain
("DD202255EE9C419781F4E61DE6E33CFE"): 0 1 <BR>Jun 4 11:48:28 CN2 kernel:
ocfs2_dlm: Node 0 joins domain E2A008B35C664DDC9FF850F59B0E122F<BR>Jun 4
11:48:28 CN2 kernel: ocfs2_dlm: Nodes in domain
("E2A008B35C664DDC9FF850F59B0E122F"): 0 1 <BR>Jun 4 11:49:07 CN2 kernel:
ocfs2_dlm: Node 0 joins domain mas<BR>Jun 4 11:49:07 CN2 kernel:
ocfs2_dlm: Nodes in domain ("mas"): 0 1 <BR>Jun 4 12:07:28 CN2
syslog-ng[4054]: STATS: dropped 0<BR>Jun 4 12:30:01 CN2 run-crons[5552]:
time.cron returned 1<BR></SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=827040714-10062008></SPAN></FONT><FONT
face=Arial size=2><SPAN class=827040714-10062008></SPAN></FONT><FONT face=Arial
size=2><SPAN class=827040714-10062008></SPAN></FONT><FONT face=Arial
size=2><SPAN class=827040714-10062008></SPAN></FONT> </DIV></BODY></HTML>