[Ocfs2-users] mystery reboot

Tue Jun 10 09:50:53 PDT 2008

http://oss.oracle.com/bugzilla/show_bug.cgi?id=919

Fixed in 1.2.9-1. SUSE has the fix checked into their
tree. It should be out soon with sles10 sp1.

Charlie Sharkey wrote:
>  
> On a two node cluster I got a reboot (core dump) on the first node.
> The /var/log/messages doesn't show anything wrong, but running
> crash on the code dump shows that ocfs2 panicked the system.
> Nodes are SuSE Sles 10 SP1 systems. I believe node 1 had a
> backup routine (Veritas) running on it at the time.
> Any ideas on what happened ?
>  
> thank you,
>   charlie
>  
>  
> node 1 system info:
> ----------------------------
> Jun  4 11:48:03 bustech-bu kernel: OCFS2 Node Manager 1.2.5-SLES-r2997 
> Tue Mar 27 16:33:19 EDT 2007 (build sles)
> Jun  4 11:48:03 bustech-bu kernel: OCFS2 DLM 1.2.5-SLES-r2997 Tue Mar 
> 27 16:33:19 EDT 2007 (build sles)
> Jun  4 11:48:03 bustech-bu kernel: OCFS2 DLMFS 1.2.5-SLES-r2997 Tue 
> Mar 27 16:33:19 EDT 2007 (build sles)
>  
> node 1 boot.msg file. looks like the reboot was on June 4, 2008  at 11:40
> ---------------------------------------------------------------------------------------------------------
> INIT:
> Boot logging started on /dev/tty1(/dev/console) at Wed Jun  4 11:40:23 
> 2008
>  
> Master Resource Control: previous runlevel: N, switching to runlevel: 1
> Starting irqbalance unused
> Saving 1979 MB crash dump to /var/log/dump/2008-06-04-11:40 ...
> Entering runlevel: 1
>  
>  
> node 1 /var/log/messages
> -------------------------------------
> Jun  4 08:37:21 bustech-bu syslog-ng[3986]: STATS: dropped 0
> Jun  4 09:15:01 bustech-bu run-crons[12180]: time.cron returned 1
> Jun  4 09:37:21 bustech-bu syslog-ng[3986]: STATS: dropped 0
> Jun  4 10:15:01 bustech-bu run-crons[14123]: time.cron returned 1
> Jun  4 10:37:21 bustech-bu syslog-ng[3986]: STATS: dropped 0
> Jun  4 11:15:01 bustech-bu run-crons[16066]: time.cron returned 1
> Jun  4 11:37:21 bustech-bu syslog-ng[3986]: STATS: dropped 0
>          **reboot here. note no previous errors**
> Jun  4 11:46:22 bustech-bu syslog-ng[4018]: syslog-ng version 1.6.8 
> starting 
> Jun  4 11:46:22 bustech-bu ifup:     lo       
>  
> node 1 crash info
> ------------------------
> crash> bt
> PID: 13     TASK: dff1f670  CPU: 3   COMMAND: "events/3"
>  #0 [dff21f08] crash_kexec at c013bb1a
>  #1 [dff21f4c] panic at c0120172
>  #2 [dff21f68] o2quo_fence_self at fb8cc399
>  #3 [dff21f70] run_workqueue at c012de27
>  #4 [dff21f8c] worker_thread at c012e754
>  #5 [dff21fcc] kthread at c0130e77
>  #6 [dff21fe8] kernel_thread_helper at c0102003
> crash>  
>  
> Node 2 /var/log/messages. it looks like this node saw node 1 go away 
> and come back
> -----------------------------------------------------------------------------------------------------------------------------
> Jun  4 10:30:01 CN2 run-crons[1587]: time.cron returned 1
> Jun  4 11:07:27 CN2 syslog-ng[4054]: STATS: dropped 0
> Jun  4 11:30:01 CN2 run-crons[3546]: time.cron returned 1
> Jun  4 11:41:33 CN2 kernel: o2net: connection to node bustech-bu (num 
> 0) at 192.168.200.10:7777 has been idle for 120.0 seconds, shutting it 
> down.
> Jun  4 11:41:33 CN2 kernel: (0,1):o2net_idle_timer:1426 here are some 
> times that might help debug the situation: (tmr 1212604773.841590 now 
> 1212604893.850342 dr 1212604773.841583 adv 
> 1212604773.841590:1212604773.841591 func (04f07b3d:505) 
> 1212595634.502257:1212595634.502260)
> Jun  4 11:41:33 CN2 kernel: o2net: no longer connected to node 
> bustech-bu (num 0) at 192.168.200.10:7777
> Jun  4 11:41:38 CN2 kernel: (5717,1):dlm_get_lock_resource:920 
> 5CA2BC69EF1C446B97521FEB7175EF1C:$RECOVERY: at least one node (0) 
> torecover before lock mastery can begin
> Jun  4 11:41:38 CN2 kernel: (5717,1):dlm_get_lock_resource:954 
> 5CA2BC69EF1C446B97521FEB7175EF1C: recovery map is not empty, but must 
> master $RECOVERY lock now
> Jun  4 11:41:38 CN2 kernel: (5728,3):dlm_get_lock_resource:920 
> 8057F00ED41A4507A24B6A4EF0211F1D:$RECOVERY: at least one node (0) 
> torecover before lock mastery can begin
> Jun  4 11:41:38 CN2 kernel: (5728,3):dlm_get_lock_resource:954 
> 8057F00ED41A4507A24B6A4EF0211F1D: recovery map is not empty, but must 
> master $RECOVERY lock now
> Jun  4 11:41:39 CN2 kernel: (6117,3):dlm_get_lock_resource:920 
> mas:$RECOVERY: at least one node (0) torecover before lock mastery can 
> begin
> Jun  4 11:41:39 CN2 kernel: (6117,3):dlm_get_lock_resource:954 mas: 
> recovery map is not empty, but must master $RECOVERY lock now
> Jun  4 11:41:40 CN2 kernel: (5706,1):dlm_get_lock_resource:920 
> 6FFB00A1F4F94113B6748BC33CA47F83:$RECOVERY: at least one node (0) 
> torecover before lock mastery can begin
> Jun  4 11:41:40 CN2 kernel: (5706,1):dlm_get_lock_resource:954 
> 6FFB00A1F4F94113B6748BC33CA47F83: recovery map is not empty, but must 
> master $RECOVERY lock now
> Jun  4 11:41:40 CN2 kernel: (5770,3):dlm_get_lock_resource:920 
> E2A008B35C664DDC9FF850F59B0E122F:$RECOVERY: at least one node (0) 
> torecover before lock mastery can begin
> Jun  4 11:41:40 CN2 kernel: (5770,3):dlm_get_lock_resource:954 
> E2A008B35C664DDC9FF850F59B0E122F: recovery map is not empty, but must 
> master $RECOVERY lock now
> Jun  4 11:41:41 CN2 kernel: (5759,3):dlm_get_lock_resource:920 
> DD202255EE9C419781F4E61DE6E33CFE:$RECOVERY: at least one node (0) 
> torecover before lock mastery can begin
> Jun  4 11:41:41 CN2 kernel: (5759,3):dlm_get_lock_resource:954 
> DD202255EE9C419781F4E61DE6E33CFE: recovery map is not empty, but must 
> master $RECOVERY lock now
> Jun  4 11:48:08 CN2 kernel: o2net: connected to node bustech-bu (num 
> 0) at 192.168.200.10:7777
> Jun  4 11:48:11 CN2 kernel: ocfs2_dlm: Node 0 joins domain 
> 6FFB00A1F4F94113B6748BC33CA47F83
> Jun  4 11:48:11 CN2 kernel: ocfs2_dlm: Nodes in domain 
> ("6FFB00A1F4F94113B6748BC33CA47F83"): 0 1
> Jun  4 11:48:15 CN2 kernel: ocfs2_dlm: Node 0 joins domain 
> 5CA2BC69EF1C446B97521FEB7175EF1C
> Jun  4 11:48:15 CN2 kernel: ocfs2_dlm: Nodes in domain 
> ("5CA2BC69EF1C446B97521FEB7175EF1C"): 0 1
> Jun  4 11:48:20 CN2 kernel: ocfs2_dlm: Node 0 joins domain 
> 8057F00ED41A4507A24B6A4EF0211F1D
> Jun  4 11:48:20 CN2 kernel: ocfs2_dlm: Nodes in domain 
> ("8057F00ED41A4507A24B6A4EF0211F1D"): 0 1
> Jun  4 11:48:24 CN2 kernel: ocfs2_dlm: Node 0 joins domain 
> DD202255EE9C419781F4E61DE6E33CFE
> Jun  4 11:48:24 CN2 kernel: ocfs2_dlm: Nodes in domain 
> ("DD202255EE9C419781F4E61DE6E33CFE"): 0 1
> Jun  4 11:48:28 CN2 kernel: ocfs2_dlm: Node 0 joins domain 
> E2A008B35C664DDC9FF850F59B0E122F
> Jun  4 11:48:28 CN2 kernel: ocfs2_dlm: Nodes in domain 
> ("E2A008B35C664DDC9FF850F59B0E122F"): 0 1
> Jun  4 11:49:07 CN2 kernel: ocfs2_dlm: Node 0 joins domain mas
> Jun  4 11:49:07 CN2 kernel: ocfs2_dlm: Nodes in domain ("mas"): 0 1
> Jun  4 12:07:28 CN2 syslog-ng[4054]: STATS: dropped 0
> Jun  4 12:30:01 CN2 run-crons[5552]: time.cron returned 1
>  
> ------------------------------------------------------------------------
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users