<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Century Gothic";
panose-1:2 11 5 2 2 2 2 2 2 4;}
@font-face
{font-family:"Times\000D\000A New Roma";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Times\000D\000A New Roman";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Courier New \;";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Times New Roman \;";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";
color:black;}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
p
{mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman";
color:black;}
pre
{margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Century Gothic";
color:black;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Century Gothic";
color:black;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Century Gothic";
color:black;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Century Gothic";
color:black;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
@page Section1
{size:595.3pt 841.9pt;
margin:70.9pt 2.0cm 70.9pt 2.0cm;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1029" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor=white lang=EN-AU link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>Ah! I didn’t
realise that!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>If I get any information
from netconsole, I will update this thread.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>Many thanks Sunil.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
color=black face="Times New Roman"><span lang=EN-US style='font-size:12.0pt;
color:windowtext'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 color=black face=Tahoma><span lang=EN-US
style='font-size:10.0pt;font-family:Tahoma;color:windowtext;font-weight:bold'>From:</span></font></b><font
size=2 color=black face=Tahoma><span lang=EN-US style='font-size:10.0pt;
font-family:Tahoma;color:windowtext'> Sunil Mushran
[mailto:sunil.mushran@oracle.com] <br>
<b><span style='font-weight:bold'>Sent:</span></b> Wednesday, 8 December 2010
10:59 AM<br>
<b><span style='font-weight:bold'>To:</span></b> Neil Campbell<br>
<b><span style='font-weight:bold'>Cc:</span></b> <st1:PersonName w:st="on">ocfs2-users@oss.oracle.com</st1:PersonName><br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [Ocfs2-users] Random
node reboots</span></font><font color=black><span lang=EN-US style='color:windowtext'><o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt'>It means that the network heartbeat has not heard from
that node for 60 secs.<br>
If disk heartbeat also agrees, then that node is assumed to be dead and<br>
recovery follows.<br>
<br>
nointr has no role to play in all this.<br>
<br>
On 12/07/2010 03:56 PM, Neil Campbell wrote: <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u1:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="PersonName"><!--[if gte mso 9]><xml>
<u1:shapedefaults u2:ext="edit" spidmax="1029"/>
</xml><![endif]--><!--[if gte mso 9]><xml>
<u1:shapelayout u3:ext="edit">
<u1:idmap u3:ext="edit" data="1"/>
</u1:shapelayout>
</xml><![endif]-->Thanks Sunil.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>OK, so when another node
says </span></font><font size=2 face="Courier New ;"><span style='font-size:
10.0pt;font-family:"Courier New \;"'>“…...been idle for 60.0
seconds, shutting it down”<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>It doesn’t actually
mean it is going to shut down the other node, instead it means<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>something like it is not
including it in the list of available nodes ?<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>Do you think I should add
the nointr mount option for this as a general purpose filesystem? <u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>In the meantime, I will
look at setting up netconsole.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>Thanks<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>Neil<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u1:p> </u1:p></span></font><o:p></o:p></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
color=black face="Times New Roma"><span lang=EN-US
style='font-size:12.0pt;font-family:"Times\000D\000A New Roma";
color:windowtext'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 color=black face=Tahoma><span lang=EN-US
style='font-size:10.0pt;font-family:Tahoma;color:windowtext;font-weight:bold'>From:</span></font></b><font
size=2 color=black face=Tahoma><span lang=EN-US style='font-size:10.0pt;
font-family:Tahoma;color:windowtext'> Sunil Mushran [<a
href="mailto:sunil.mushran@oracle.com">mailto:sunil.mushran@oracle.com</a>] <br>
<b><span style='font-weight:bold'>Sent:</span></b> Wednesday, 8 December 2010
10:50 AM<br>
<b><span style='font-weight:bold'>To:</span></b> Neil Campbell<br>
<b><span style='font-weight:bold'>Cc:</span></b> <st1:personname u4:st="on"><a
href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</a></st1:personname><br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [Ocfs2-users] Random
node reboots</span></font><o:p></o:p></p>
</div>
<u1:p></u1:p>
<p class=MsoNormal><font size=3 color=black
face="Times New Roman"><span style='font-size:12.0pt;
font-family:"Times\000D\000A New Roman"'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=3 color=black
face="Times New Roman"><span style='font-size:12.0pt;
font-family:"Times\000D\000A New Roman"'>It's the other way round.
These message indicate that the other node<br>
died. When these nodes detect that, they need to do recovery/cleanup.<br>
The messages all relate to that activity.<br>
<br>
The reason as to why the first node died will only be known once we have<br>
the oops trace. And that is caught by netdump/netconsole.<br>
<br>
On 12/07/2010 03:37 PM, Neil Campbell wrote: <u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u5:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="PersonName"><!--[if gte mso 9]><xml>
<u5:shapedefaults u2:ext="edit" spidmax="1029"/>
</xml><![endif]--><!--[if gte mso 9]><xml>
<u5:shapelayout u3:ext="edit">
<u5:idmap u3:ext="edit" data="1"/>
</u5:shapelayout>
</xml><![endif]-->Hi Sunil,<u5:p></u5:p></u5:smarttagtype></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u5:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>Thanks for responding.<u5:p></u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u5:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>It is being evicted by
the other nodes in the OCFS2 cluster isn’t it?<u5:p></u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u5:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>/var/log/messages from
dcapp02 shows <u5:p></u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u5:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New"'>Dec 8 01:56:01 dcapp02
kernel: o2net: connection to node dcapp01 (num 0) at 10.255.255.1:10007 has
been idle for 60.0 seconds, shutting it down</span></font>.<u1:p></u1:p><o:p></o:p></p>
<u5:p></u5:p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u5:p><u6:p></u6:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>This appears in all the
other messages files of the nodes in the cluster (except the one that got
re-booted).<u5:p></u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u5:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>So that is why dcapp01
got shutdown isn’t it? What I think I need help with is why ?<u5:p></u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u5:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>Many thanks<u5:p></u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'>Neil<u5:p></u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<p class=MsoNormal><font size=2 color=black face="Century Gothic"><span
style='font-size:10.0pt;font-family:"Century Gothic"'><u5:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
color=black face="Times New Roma"><span lang=EN-US
style='font-size:12.0pt;font-family:"Times\000D\000A New Roma"'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 color=black face=Tahoma><span lang=EN-US
style='font-size:10.0pt;font-family:Tahoma;color:windowtext;font-weight:bold'>From:</span></font></b><font
size=2 color=black face=Tahoma><span lang=EN-US style='font-size:10.0pt;
font-family:Tahoma;color:windowtext'> Sunil Mushran [<a
href="mailto:sunil.mushran@oracle.com" moz-do-not-send=true>mailto:sunil.mushran@oracle.com</a>]
<br>
<b><span style='font-weight:bold'>Sent:</span></b> Wednesday, 8 December 2010
10:32 AM<br>
<b><span style='font-weight:bold'>To:</span></b> Neil Campbell<br>
<b><span style='font-weight:bold'>Cc:</span></b> <st1:personname u7:st="on"><a
href="mailto:ocfs2-users@oss.oracle.com" moz-do-not-send=true>ocfs2-users@oss.oracle.com</a></st1:personname><br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [Ocfs2-users] Random
node reboots</span></font><u1:p></u1:p><o:p></o:p></p>
</div>
<u5:p></u5:p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman ;"><span
style='font-size:12.0pt;font-family:"Times New Roman \;"'><u5:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<pre><font size=2 color=black face="Courier New"><span style='font-size:10.0pt'>==> /var/log/messages from the node being rebooted doesn’t show anything, just the to reboot shows the following<u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u5:p> </u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>That's how Linux works. You should configure netconsole or netdump<u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>to capture the logs. Only then we'll know as to why the node is panic-ing.<u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=3 color=black face="Times New Roman ;"><span
style='font-size:12.0pt;font-family:"Times New Roman \;"'><br>
On 12/07/2010 03:27 PM, Neil Campbell wrote: <u5:p></u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<pre><font size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="PersonName"><!--[if gte mso 9]><xml>
<u8:shapedefaults u2:ext="edit" spidmax="1029"/>
</xml><![endif]--><!--[if gte mso 9]><xml>
<u8:shapelayout u3:ext="edit">
<u8:idmap u3:ext="edit" data="1"/>
</u8:shapelayout>
</xml><![endif]-->Hi all,<u8:p></u8:p><u5:p></u5:p></u8:smarttagtype><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>I keep getting node reboots across my cluster, it seems random in that the node being evicted changes<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>and in that it happens every now an then. I’m running RHEL 4 kernel 2.6.89.0.26.ELsmp, <u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>and OCFS is OCFS2 1.2.9 Mon Jun 21 20:03:07 PDT 2010 (build 5e8325ec7f66b5189c65c7a8710fe8cb)<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>I am using OCFS2 as a general purpose filesystem (i.e not for <st1:personname u6:st="on"><st1:personname u7:st="on"><st1:personname u4:st="on"><st1:PersonName
w:st="on">Oracle</st1:personname></st1:personname></st1:personname></st1:PersonName> datafiles or OCR etc),<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>with the following entries in /etc/fstab <u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>/dev/emcpowera1 /u01/cfs ocfs2 _netdev 0 0<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>As a general purpose filesystem, should I be using the nointr mount option?<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>/etc/init.d/o2cb status<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>Module "configfs": Loaded<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>Filesystem "configfs": Mounted<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>Module "ocfs2_nodemanager": Loaded<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>Module "ocfs2_dlm": Loaded<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>Module "ocfs2_dlmfs": Loaded<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>Filesystem "ocfs2_dlmfs": Mounted<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>Checking O2CB cluster UATocfs2: Online<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'> Heartbeat dead threshold: 61<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'> Network idle timeout: 60000<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'> Network keepalive delay: 2000<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'> Network reconnect delay: 2000<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre
style='margin-left:36.0pt'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>Checking O2CB heartbeat: Active<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>/var/log/messages from the node being rebooted doesn’t show anything, just the to reboot shows the following<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'> Dec 8 00:59:02 dcapp01 syslogd 1.4.1: restart.<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>On the other nodes, I see the following entries<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:56:01 dcapp02 kernel: o2net: connection to node dcapp01 (num 0) at 10.255.255.1:10007 has been idle for 60.0 seconds, shutting it down.<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:56:01 dcapp02 kernel: (0,3):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1291733701.691575 now 1291733761.692608 dr 1291733701.690949 adv 1291733701.696965:1291733701.696967 func (d399da91:500) 1291733701.691576:1291733701.696950)<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:56:01 dcapp02 kernel: o2net: no longer connected to node dcapp01 (num 0) at 10.255.255.1:10007<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:57:01 dcapp02 kernel: (16082,3):o2net_connect_expired:1585 ERROR: no connection established with node 0 after 60.0 seconds, giving up and returning errors.<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:57:01 dcapp02 kernel: (4215,2):dlm_send_remote_convert_request:398 ERROR: status = -107<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:57:01 dcapp02 kernel: (4215,2):dlm_wait_for_node_death:365 C5C06C9B675D41B99B60DE2EB28CE0F7: waiting 5000ms for notification of death of node 0<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:57:04 dcapp02 kernel: (16082,3):ocfs2_dlm_eviction_cb:119 device (120,1): dlm has evicted node 0<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:57:05 dcapp02 kernel: (4269,0):dlm_send_remote_convert_request:398 ERROR: status = -107<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:57:05 dcapp02 kernel: (4269,0):dlm_wait_for_node_death:365 D43AF814A25845F7B103EBBEA440BA18: waiting 5000ms for notification of death of node 0<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:57:05 dcapp02 kernel: (16082,3):ocfs2_dlm_eviction_cb:119 device (120,66): dlm has evicted node 0<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 01:57:06 dcapp02 kernel: (16082,3):ocfs2_dlm_eviction_cb:119 device (120,65): dlm has evicted node 0<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 02:00:15 dcapp02 kernel: o2net: connected to node dcapp01 (num 0) at 10.255.255.1:10007<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 02:00:29 dcapp02 kernel: ocfs2_dlm: Node 0 joins domain C5C06C9B675D41B99B60DE2EB28CE0F7<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 02:00:29 dcapp02 kernel: ocfs2_dlm: Nodes in domain ("C5C06C9B675D41B99B60DE2EB28CE0F7"): 0 1 2 3 6 7 8 9 10 11 14 15<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 02:00:35 dcapp02 kernel: ocfs2_dlm: Node 0 joins domain 97F22666B5A6494AAF38C53909275DB2<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 02:00:35 dcapp02 kernel: ocfs2_dlm: Nodes in domain ("97F22666B5A6494AAF38C53909275DB2"): 0 1 2 3<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 02:00:39 dcapp02 kernel: ocfs2_dlm: Node 0 joins domain D43AF814A25845F7B103EBBEA440BA18<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Dec 8 02:00:39 dcapp02 kernel: ocfs2_dlm: Nodes in domain ("D43AF814A25845F7B103EBBEA440BA18"): 0 1 2 3<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>I would really appreciate some help with this, as I’m not sure where to go from here.<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Thanks<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Neil<u8:p></u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u8:p> </u8:p><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre>
<div class=MsoNormal align=center style='text-align:center'><font size=2
color=black face=Arial><span style='font-size:10.0pt;font-family:Arial'>
<hr size=2 width="100%" align=center>
</span></font></div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial'>Downer<br>
This message is for the named person's use only. It may contain confidential,
proprietary or legally privileged information. No confidentiality or privilege
is waived or lost by any mistransmission. If you receive this message in error,
please immediately delete it and all copies of it from your system, destroy any
hard copies of it and notify the sender. You must not, directly or indirectly,
use, disclose, distribute, print, or copy any part of this message if you are
not the intended recipient. Downer EDI and any of its subsidiaries each reserve
the right to monitor all e-mail communications through its networks. Any views
expressed in this message are those of the individual sender, except where the
message states otherwise and the sender is authorized to state them to be the
views of any such entity. <u5:p></u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<div class=MsoNormal align=center style='text-align:center'><font size=2
color=black face=Arial><span style='font-size:10.0pt;font-family:Arial'>
<hr size=2 width="100%" align=center>
</span></font></div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman ;"><span
style='font-size:12.0pt;font-family:"Times New Roman \;"'><u5:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<pre wrap=""><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'><u5:p> </u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><fieldset class="mimeAttachmentHeader"></fieldset><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u5:p> </u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>_______________________________________________<u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Ocfs2-users mailing list<u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><a
href="mailto:Ocfs2-users@oss.oracle.com" moz-do-not-send=true>Ocfs2-users@oss.oracle.com</a><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><a
href="http://oss.oracle.com/mailman/listinfo/ocfs2-users" moz-do-not-send=true>http://oss.oracle.com/mailman/listinfo/ocfs2-users</a><u5:p></u5:p><u1:p></u1:p><o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=3 color=black face="Times New Roman ;"><span
style='font-size:12.0pt;font-family:"Times New Roman \;"'><u5:p> </u5:p></span></font><u1:p></u1:p><o:p></o:p></p>
<div class=MsoNormal align=center style='text-align:center'><font size=2
color=black face=Arial><span style='font-size:10.0pt;font-family:Arial'>
<hr size=2 width="100%" align=center>
</span></font></div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial'>Downer<br>
This message is for the named person's use only. It may contain confidential,
proprietary or legally privileged information. No confidentiality or privilege
is waived or lost by any mistransmission. If you receive this message in error,
please immediately delete it and all copies of it from your system, destroy any
hard copies of it and notify the sender. You must not, directly or indirectly,
use, disclose, distribute, print, or copy any part of this message if you are
not the intended recipient. Downer EDI and any of its subsidiaries each reserve
the right to monitor all e-mail communications through its networks. Any views
expressed in this message are those of the individual sender, except where the
message states otherwise and the sender is authorized to state them to be the
views of any such entity. <u1:p></u1:p></span></font><o:p></o:p></p>
<div class=MsoNormal align=center style='text-align:center'><font size=2
color=black face=Arial><span style='font-size:10.0pt;font-family:Arial'>
<hr size=2 width="100%" align=center>
</span></font></div>
<p class=MsoNormal><font size=3 color=black
face="Times New Roman"><span style='font-size:12.0pt;
font-family:"Times\000D\000A New Roman"'><u1:p> </u1:p></span></font><o:p></o:p></p>
<pre wrap=""><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'><u1:p> </u1:p><o:p></o:p></span></font></pre><pre><fieldset class="mimeAttachmentHeader"></fieldset><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><u1:p> </u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>_______________________________________________<u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Ocfs2-users mailing list<u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><a
href="mailto:Ocfs2-users@oss.oracle.com" moz-do-not-send=true>Ocfs2-users@oss.oracle.com</a><u1:p></u1:p><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><a
href="http://oss.oracle.com/mailman/listinfo/ocfs2-users" moz-do-not-send=true>http://oss.oracle.com/mailman/listinfo/ocfs2-users</a><u1:p></u1:p><o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=3 color=black
face="Times New Roman"><span style='font-size:12.0pt;
font-family:"Times\000D\000A New Roman"'><u1:p> </u1:p></span></font><o:p></o:p></p>
<div class=MsoNormal align=center style='text-align:center'><font size=2
color=black face=Arial><span style='font-size:10.0pt;font-family:Arial'>
<hr size=2 width="100%" align=center>
</span></font></div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial'>Downer<br>
This message is for the named person's use only. It may contain confidential,
proprietary or legally privileged information. No confidentiality or privilege
is waived or lost by any mistransmission. If you receive this message in error,
please immediately delete it and all copies of it from your system, destroy any
hard copies of it and notify the sender. You must not, directly or indirectly,
use, disclose, distribute, print, or copy any part of this message if you are
not the intended recipient. Downer EDI and any of its subsidiaries each reserve
the right to monitor all e-mail communications through its networks. Any views
expressed in this message are those of the individual sender, except where the
message states otherwise and the sender is authorized to state them to be the
views of any such entity. <o:p></o:p></span></font></p>
<div class=MsoNormal align=center style='text-align:center'><font size=2
color=black face=Arial><span style='font-size:10.0pt;font-family:Arial'>
<hr size=2 width="100%" align=center>
</span></font></div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt'><o:p> </o:p></span></font></p>
<pre wrap=""><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><fieldset class="mimeAttachmentHeader"></fieldset><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>_______________________________________________<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Ocfs2-users mailing list<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><a
href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><a
href="http://oss.oracle.com/mailman/listinfo/ocfs2-users">http://oss.oracle.com/mailman/listinfo/ocfs2-users</a><o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt'><o:p> </o:p></span></font></p>
</u1:smarttagtype></div>
<P><FONT size=2 face=Arial>
<HR>
</FONT><FONT size=2 face=Arial>Downer<BR>This message is for the named person's
use only. It may contain confidential, proprietary or legally privileged
information. No confidentiality or privilege is waived or lost by any
mistransmission. If you receive this message in error, please immediately delete
it and all copies of it from your system, destroy any hard copies of it and
notify the sender. You must not, directly or indirectly, use, disclose,
distribute, print, or copy any part of this message if you are not the intended
recipient. Downer EDI and any of its subsidiaries each reserve the right to
monitor all e-mail communications through its networks. Any views expressed in
this message are those of the individual sender, except where the message states
otherwise and the sender is authorized to state them to be the views of any such
entity.</FONT><FONT size=2 face=Arial>
<HR>
<BR>
<P></P></FONT>
</body>
</html>