<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=NO-BOK link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Hi,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>We are running a 8 nodes cluster on RHEL 2.6.18-128 64-bit. Yesterday the server/san guys exchanged the ocfs2 disks to another SAN, by mirroring and synchronizing the disks. When they rebooted the servers, one of the nodes, tos-dipsprod-07 wasn’t able to start Oracle Grid Infrastructure, the voting disk was not found. Then we tried to reboot that node, causing all nodes to reboot. Time round about 02:25. When examine the /var/log/messages I discovered a BUG message on one of the node that rebooted unexpectedly, tos-dipsprod-02. I’ve tried to google it, but I couldn’t find any solution. Is this a well known bug? Does any body have a solution to this problem?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Below is a extract of o2net and ocfs2 messages from the /var/log/message file.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>/var/log/messages til tos-dipsprod-07:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 02:08:49 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-06 (num 3) at 192.168.7.105:7777 has been idle for 10.0 seconds, shutting it down.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 02:25:25 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-01 (num 0) at 192.168.7.100:7777 has been idle for 10.0 seconds, shutting it down.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 02:25:35 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-02 (num 1) at 192.168.7.101:7777 has been idle for 10.0 seconds, shutting it down.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 02:25:40 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-03 (num 2) at 192.168.7.102:7777 has been idle for 10.0 seconds, shutting it down.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 02:25:45 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-06 (num 3) at 192.168.7.105:7777 has been idle for 10.0 seconds, shutting it down.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 02:25:54 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-04 (num 5) at 192.168.7.103:7777 has been idle for 10.0 seconds, shutting it down.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 04:03:17 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-06 (num 3) at 192.168.7.105:7777 has been idle for 10.0 seconds, shutting it down.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 04:06:32 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-01 (num 0) at 192.168.7.100:7777 has been idle for 10.0 seconds, shutting it down.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 04:06:37 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-02 (num 1) at 192.168.7.101:7777 has been idle for 10.0 seconds, shutting it down.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 04:06:47 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-03 (num 2) at 192.168.7.102:7777 has been idle for 10.0 seconds, shutting it down.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Mar 21 06:04:25 tos-dipsprod-07 kernel: o2net: connection to node tos-dipsprod-02 (num 1) at 192.168.7.101:7777 has been idle for 10.0 seconds, shutting it down.</span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>Og her fra tos-dipsprod-02:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>10474-Mar 21 02:25:15 tos-dipsprod-02 kernel: (o2net,7452,5):dlm_begin_reco_handler:2730 992D008CD522447C8333FC34BD46F8CD: dead_node previously set to 7, node 3 changing it to 7<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>10646-Mar 21 02:25:25 tos-dipsprod-02 kernel: (o2net,7452,5):dlm_finalize_reco_handler:2839 ERROR: node 6 sent recovery finalize msg, but node 3 is supposed to be the new master, dead=7<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>10826:Mar 21 02:25:25 tos-dipsprod-02 kernel: Kernel BUG at ...shran/BUILD/ocfs2-1.4.7/fs/ocfs2/dlm/dlmrecovery.c:2840<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>10939-Mar 21 02:43:01 tos-dipsprod-02 syslogd 1.4.1: restart.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>10995-Mar 21 02:43:02 tos-dipsprod-02 modprobe: FATAL: Module ocfs2_stackglue not found.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>--<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>17537-Mar 21 04:06:19 tos-dipsprod-02 kernel: (o2net,7472,1):dlm_begin_reco_handler:2730 992D008CD522447C8333FC34BD46F8CD: dead_node previously set to 6, node 6 changing it to 7<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>17709-Mar 21 04:06:29 tos-dipsprod-02 kernel: (o2net,7472,1):dlm_finalize_reco_handler:2839 ERROR: node 6 sent recovery finalize msg, but node 255 is supposed to be the new master, dead=7<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>17891:Mar 21 04:06:29 tos-dipsprod-02 kernel: Kernel BUG at ...shran/BUILD/ocfs2-1.4.7/fs/ocfs2/dlm/dlmrecovery.c:2840<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>18004-Mar 21 04:38:04 tos-dipsprod-02 syslogd 1.4.1: restart.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>18060-Mar 21 04:41:33 tos-dipsprod-02 modprobe: FATAL: Module ocfs2_stackglue not found.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><table class=MsoNormalTable border=0 cellpadding=0 width="209%" style='width:209.64%;margin-left:-.75pt'><tr><td style='padding:0cm 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:115%'><b><span style='font-size:7.5pt;line-height:115%;font-family:"Verdana","sans-serif";color:#1F497D'>Morten Kristiansen </span></b><span style='font-size:7.5pt;line-height:115%;font-family:"Verdana","sans-serif";color:#1F497D'>| <i>Counsellor </i><br><b>Helse Nord IKT</b> | Departement of Serviceproduction </span><span style='color:#1F497D'><o:p></o:p></span></p></td></tr><tr><td style='padding:0cm 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:115%'><b><span lang=EN-US style='font-size:7.5pt;line-height:115%;font-family:"Verdana","sans-serif";color:#1F497D'>Tlf:</span></b><span lang=EN-US style='font-size:7.5pt;line-height:115%;font-family:"Verdana","sans-serif";color:#1F497D'> +47 76 16 61 81 | <b>Mob:</b> +47 906 52 903 <o:p></o:p></span></p><p class=MsoNormal style='line-height:115%'><b><span lang=EN-US style='font-size:7.5pt;line-height:115%;font-family:"Verdana","sans-serif";color:#1F497D'>Office address:</span></b><span lang=EN-US style='font-size:7.5pt;line-height:115%;font-family:"Verdana","sans-serif";color:#1F497D'> Amtmann Worsøes gate 63, 8012 Bodø, Norway<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:115%'><span style='font-size:8.0pt;line-height:115%;font-family:"Verdana","sans-serif";color:gray'>Quality - Safety – Respect</span><span style='font-size:7.5pt;line-height:115%;font-family:"Verdana","sans-serif";color:#1F497D'><o:p></o:p></span></p></td></tr></table><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p></div></body></html>