[Ocfs2-users] O2CB heartbeat not active on 2nd node

[McKinley, Reid]

No, iptables is shutdown and disabled.  No firewalls.

[root at nyclx1 ~]# service iptables status
Firewall is stopped.

-----Original Message-----
From: Sunil Mushran [mailto:sunil.mushran at oracle.com] 
Sent: Wednesday, June 03, 2009 12:57 PM
To: McKinley, Reid
Cc: ocfs2-users at oss.oracle.com
Subject: Re: [Ocfs2-users] O2CB heartbeat not active on 2nd node

The connect requests are not getting through. Do you
have any firewalls setup? Is iptables running? If so, either
shut it down or allow traffic on the o2cb port.

McKinley, Reid wrote:
>
> We are having trouble getting the 2^nd node in our 2 node RAC 
> configuration to have an active O2CB heartbeat. We have our OCR and 
> voting disks on an OCFS2 mount point, so we cannot bring up 
> Clusterware on this node.
>
> I'm at a loss as to what the issue is. It was running fine for a few 
> weeks, then we had a reboot and we cannot get the heartbeat active and

> we cannot mount any OCFS2 filesystems on the 2^nd node.
>
> Any ideas are greatly appreciated.
>
> Dmesg errors are at the bottom.
>
> Here are the rpm and status details:
>
> [root at nyclx1 ~]# rpm -qa | grep ocfs2
>
> ocfs2-tools-1.4.1-1.el5
>
> ocfs2console-1.4.1-1.el5
>
> ocfs2-2.6.18-92.el5-1.4.1-1.el5
>
> ocfs2-2.6.18-92.el5debug-1.2.8-2.el5
>
> ocfs2-2.6.18-92.el5-debuginfo-1.4.1-1.el5
>
> ocfs2-tools-debuginfo-1.4.1-1.el5
>
> [root at nyclx1 ~]# /etc/init.d/o2cb status
>
> Driver for "configfs": Loaded
>
> Filesystem "configfs": Mounted
>
> Driver for "ocfs2_dlmfs": Loaded
>
> Filesystem "ocfs2_dlmfs": Mounted
>
> Checking O2CB cluster tiaa: Online
>
> Heartbeat dead threshold = 31
>
> Network idle timeout: 30000
>
> Network keepalive delay: 2000
>
> Network reconnect delay: 2000
>
> Checking O2CB heartbeat: Active
>
> [root at nyclx2 ~]# rpm -qa | grep ocfs2
>
> ocfs2-tools-1.4.1-1.el5
>
> ocfs2-2.6.18-92.el5-1.4.1-1.el5
>
> ocfs2-tools-debuginfo-1.4.1-1.el5
>
> ocfs2console-1.4.1-1.el5
>
> ocfs2-2.6.18-92.el5debug-1.2.8-2.el5
>
> ocfs2-2.6.18-92.el5-debuginfo-1.4.1-1.el5
>
> [root at nyclx2 ~]# /etc/init.d/o2cb status
>
> Driver for "configfs": Loaded
>
> Filesystem "configfs": Mounted
>
> Driver for "ocfs2_dlmfs": Loaded
>
> Filesystem "ocfs2_dlmfs": Mounted
>
> Checking O2CB cluster tiaa: Online
>
> Heartbeat dead threshold = 31
>
> Network idle timeout: 30000
>
> Network keepalive delay: 2000
>
> Network reconnect delay: 2000
>
> Checking O2CB heartbeat: Not active
>
> OCFS2 1.4.1 Wed Jul 23 12:05:34 PDT 2008 (build 
> 3fc82af4b5669945497b322b6aabd031)
>
> (11296,1):o2net_connect_expired:1637 ERROR: no connection established 
> with node 0 after 30.0 seconds, giving up and returning errors.
>
> (14212,1):dlm_request_join:1033 ERROR: status = -107
>
> (14212,1):dlm_try_to_join_domain:1207 ERROR: status = -107
>
> (14212,1):dlm_join_domain:1485 ERROR: status = -107
>
> (14212,1):dlm_register_domain:1732 ERROR: status = -107
>
> (14212,1):ocfs2_dlm_init:2662 ERROR: status = -107
>
> (14212,1):ocfs2_mount_volume:1251 ERROR: status = -107
>
> ocfs2: Unmounting device (253,2) on (node 1)
>
> (11296,1):o2net_connect_expired:1637 ERROR: no connection established 
> with node 0 after 30.0 seconds, giving up and returning errors.
>
> (14350,1):dlm_request_join:1033 ERROR: status = -107
>
> (14350,1):dlm_try_to_join_domain:1207 ERROR: status = -107
>
> (14350,1):dlm_join_domain:1485 ERROR: status = -107
>
> (14350,1):dlm_register_domain:1732 ERROR: status = -107
>
> (14350,1):ocfs2_dlm_init:2662 ERROR: status = -107
>
> (14350,1):ocfs2_mount_volume:1251 ERROR: status = -107
>
> ocfs2: Unmounting device (253,3) on (node 1)
>
> (11296,1):o2net_connect_expired:1637 ERROR: no connection established 
> with node 0 after 30.0 seconds, giving up and returning errors.
>
> (4347,1):dlm_request_join:1033 ERROR: status = -107
>
> (4347,1):dlm_try_to_join_domain:1207 ERROR: status = -107
>
> (4347,1):dlm_join_domain:1485 ERROR: status = -107
>
> (4347,1):dlm_register_domain:1732 ERROR: status = -107
>
> (4347,1):ocfs2_dlm_init:2662 ERROR: status = -107
>
> (4347,1):ocfs2_mount_volume:1251 ERROR: status = -107
>
> ocfs2: Unmounting device (253,3) on (node 1)
>
> (11296,1):o2net_connect_expired:1637 ERROR: no connection established 
> with node 0 after 30.0 seconds, giving up and returning errors.
>
> (4948,1):dlm_request_join:1033 ERROR: status = -107
>
> (4948,1):dlm_try_to_join_domain:1207 ERROR: status = -107
>
> (4948,1):dlm_join_domain:1485 ERROR: status = -107
>
> (4948,1):dlm_register_domain:1732 ERROR: status = -107
>
> (4948,1):ocfs2_dlm_init:2662 ERROR: status = -107
>
> (4948,1):ocfs2_mount_volume:1251 ERROR: status = -107
>
> ocfs2: Unmounting device (253,3) on (node 1)
>
> OCFS2 Node Manager 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build 
> 0f78045c75c0174e50e4cf0934bf9eae)
>
> OCFS2 DLM 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build 
> 4ce8fae327880c466761f40fb7619490)
>
> OCFS2 DLMFS 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build 
> 4ce8fae327880c466761f40fb7619490)
>
> OCFS2 User DLM kernel interface loaded
>
> [root at nyclx2 ~]#
>
> Reid McKinley
>
>
************************************************************************
********************
> This message, including any attachments, contains confidential
information intended 
> for a specific individual and purpose, and is protected by law. If you
are not the intended 
> recipient, please contact the sender immediately by reply e-mail and
destroy all copies.
> You are hereby notified that any disclosure, copying, or distribution
of this message, or
> the taking of any action based on it, is strictly prohibited.
>
> TIAA-CREF
>
************************************************************************
********************
>   
>
------------------------------------------------------------------------
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


********************************************************************************************
This message, including any attachments, contains confidential information intended 
for a specific individual and purpose, and is protected by law. If you are not the intended 
recipient, please contact the sender immediately by reply e-mail and destroy all copies.
You are hereby notified that any disclosure, copying, or distribution of this message, or
the taking of any action based on it, is strictly prohibited.

TIAA-CREF
********************************************************************************************