<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:#606420;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink="#606420">
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>We are having trouble getting the 2<sup>nd</sup> node in our
2 node RAC configuration to have an active O2CB heartbeat. We have our OCR and
voting disks on an OCFS2 mount point, so we cannot bring up Clusterware on this
node.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I’m at a loss as to what the issue is. It was
running fine for a few weeks, then we had a reboot and we cannot get the
heartbeat active and we cannot mount any OCFS2 filesystems on the 2<sup>nd</sup>
node.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Any ideas are greatly appreciated.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Dmesg errors are at the bottom.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Here are the rpm and status details:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>[root@nyclx1 ~]# rpm -qa | grep ocfs2<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2-tools-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2console-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2-2.6.18-92.el5-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2-2.6.18-92.el5debug-1.2.8-2.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2-2.6.18-92.el5-debuginfo-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2-tools-debuginfo-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>[root@nyclx1 ~]# /etc/init.d/o2cb status<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Driver for "configfs": Loaded<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Filesystem "configfs": Mounted<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Driver for "ocfs2_dlmfs": Loaded<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Filesystem "ocfs2_dlmfs": Mounted<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Checking O2CB cluster tiaa: Online<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Heartbeat dead threshold = 31<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'> Network idle timeout: 30000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'> Network keepalive delay: 2000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'> Network reconnect delay: 2000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Checking O2CB heartbeat: Active<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>[root@nyclx2 ~]# rpm -qa | grep ocfs2<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2-tools-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2-2.6.18-92.el5-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2-tools-debuginfo-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2console-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2-2.6.18-92.el5debug-1.2.8-2.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2-2.6.18-92.el5-debuginfo-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>[root@nyclx2 ~]# /etc/init.d/o2cb status<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Driver for "configfs": Loaded<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Filesystem "configfs": Mounted<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Driver for "ocfs2_dlmfs": Loaded<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Filesystem "ocfs2_dlmfs": Mounted<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Checking O2CB cluster tiaa: Online<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Heartbeat dead threshold = 31<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'> Network idle timeout: 30000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'> Network keepalive delay: 2000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'> Network reconnect delay: 2000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>Checking O2CB heartbeat: Not active<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>OCFS2 1.4.1 Wed Jul 23 12:05:34 PDT 2008 (build
3fc82af4b5669945497b322b6aabd031)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(11296,1):o2net_connect_expired:1637 ERROR: no
connection established with node 0 after 30.0 seconds, giving up and returning
errors.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14212,1):dlm_request_join:1033 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14212,1):dlm_try_to_join_domain:1207 ERROR: status
= -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14212,1):dlm_join_domain:1485 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14212,1):dlm_register_domain:1732 ERROR: status =
-107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14212,1):ocfs2_dlm_init:2662 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14212,1):ocfs2_mount_volume:1251 ERROR: status =
-107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2: Unmounting device (253,2) on (node 1)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(11296,1):o2net_connect_expired:1637 ERROR: no
connection established with node 0 after 30.0 seconds, giving up and returning
errors.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14350,1):dlm_request_join:1033 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14350,1):dlm_try_to_join_domain:1207 ERROR: status
= -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14350,1):dlm_join_domain:1485 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14350,1):dlm_register_domain:1732 ERROR: status =
-107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14350,1):ocfs2_dlm_init:2662 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(14350,1):ocfs2_mount_volume:1251 ERROR: status =
-107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2: Unmounting device (253,3) on (node 1)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(11296,1):o2net_connect_expired:1637 ERROR: no
connection established with node 0 after 30.0 seconds, giving up and returning
errors.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4347,1):dlm_request_join:1033 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4347,1):dlm_try_to_join_domain:1207 ERROR: status =
-107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4347,1):dlm_join_domain:1485 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4347,1):dlm_register_domain:1732 ERROR: status =
-107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4347,1):ocfs2_dlm_init:2662 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4347,1):ocfs2_mount_volume:1251 ERROR: status =
-107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2: Unmounting device (253,3) on (node 1)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(11296,1):o2net_connect_expired:1637 ERROR: no
connection established with node 0 after 30.0 seconds, giving up and returning
errors.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4948,1):dlm_request_join:1033 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4948,1):dlm_try_to_join_domain:1207 ERROR: status =
-107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4948,1):dlm_join_domain:1485 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4948,1):dlm_register_domain:1732 ERROR: status =
-107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4948,1):ocfs2_dlm_init:2662 ERROR: status = -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>(4948,1):ocfs2_mount_volume:1251 ERROR: status =
-107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>ocfs2: Unmounting device (253,3) on (node 1)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>OCFS2 Node Manager 1.4.1 Wed Jul 23 12:05:37 PDT
2008 (build 0f78045c75c0174e50e4cf0934bf9eae)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>OCFS2 DLM 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build
4ce8fae327880c466761f40fb7619490)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>OCFS2 DLMFS 1.4.1 Wed Jul 23 12:05:37 PDT 2008
(build 4ce8fae327880c466761f40fb7619490)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>OCFS2 User DLM kernel interface loaded<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>[root@nyclx2 ~]#<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Reid McKinley</span></font><o:p></o:p></p>
</div>
<pre>
********************************************************************************************
This message, including any attachments, contains confidential information intended
for a specific individual and purpose, and is protected by law. If you are not the intended
recipient, please contact the sender immediately by reply e-mail and destroy all copies.
You are hereby notified that any disclosure, copying, or distribution of this message, or
the taking of any action based on it, is strictly prohibited.
TIAA-CREF
********************************************************************************************
</pre></body>
</html>