[Ocfs2-users] O2CB heartbeat not active on 2nd node

[McKinley, Reid]

We are having trouble getting the 2nd node in our 2 node RAC
configuration to have an active O2CB heartbeat. We have our OCR and
voting disks on an OCFS2 mount point, so we cannot bring up Clusterware
on this node.

 

I'm at a loss as to what the issue is.  It was running fine for a few
weeks, then we had a reboot and we cannot get the heartbeat active and
we cannot mount any OCFS2 filesystems on the 2nd node.

 

Any ideas are greatly appreciated.

 

Dmesg errors are at the bottom.

 

Here are the rpm and status details:

[root at nyclx1 ~]# rpm -qa | grep ocfs2

ocfs2-tools-1.4.1-1.el5

ocfs2console-1.4.1-1.el5

ocfs2-2.6.18-92.el5-1.4.1-1.el5

ocfs2-2.6.18-92.el5debug-1.2.8-2.el5

ocfs2-2.6.18-92.el5-debuginfo-1.4.1-1.el5

ocfs2-tools-debuginfo-1.4.1-1.el5

[root at nyclx1 ~]# /etc/init.d/o2cb status

Driver for "configfs": Loaded

Filesystem "configfs": Mounted

Driver for "ocfs2_dlmfs": Loaded

Filesystem "ocfs2_dlmfs": Mounted

Checking O2CB cluster tiaa: Online

Heartbeat dead threshold = 31

  Network idle timeout: 30000

  Network keepalive delay: 2000

  Network reconnect delay: 2000

Checking O2CB heartbeat: Active

 

[root at nyclx2 ~]# rpm -qa | grep ocfs2

ocfs2-tools-1.4.1-1.el5

ocfs2-2.6.18-92.el5-1.4.1-1.el5

ocfs2-tools-debuginfo-1.4.1-1.el5

ocfs2console-1.4.1-1.el5

ocfs2-2.6.18-92.el5debug-1.2.8-2.el5

ocfs2-2.6.18-92.el5-debuginfo-1.4.1-1.el5

[root at nyclx2 ~]# /etc/init.d/o2cb status

Driver for "configfs": Loaded

Filesystem "configfs": Mounted

Driver for "ocfs2_dlmfs": Loaded

Filesystem "ocfs2_dlmfs": Mounted

Checking O2CB cluster tiaa: Online

Heartbeat dead threshold = 31

  Network idle timeout: 30000

  Network keepalive delay: 2000

  Network reconnect delay: 2000

Checking O2CB heartbeat: Not active

 

OCFS2 1.4.1 Wed Jul 23 12:05:34 PDT 2008 (build
3fc82af4b5669945497b322b6aabd031)

(11296,1):o2net_connect_expired:1637 ERROR: no connection established
with node 0 after 30.0 seconds, giving up and returning errors.

(14212,1):dlm_request_join:1033 ERROR: status = -107

(14212,1):dlm_try_to_join_domain:1207 ERROR: status = -107

(14212,1):dlm_join_domain:1485 ERROR: status = -107

(14212,1):dlm_register_domain:1732 ERROR: status = -107

(14212,1):ocfs2_dlm_init:2662 ERROR: status = -107

(14212,1):ocfs2_mount_volume:1251 ERROR: status = -107

ocfs2: Unmounting device (253,2) on (node 1)

(11296,1):o2net_connect_expired:1637 ERROR: no connection established
with node 0 after 30.0 seconds, giving up and returning errors.

(14350,1):dlm_request_join:1033 ERROR: status = -107

(14350,1):dlm_try_to_join_domain:1207 ERROR: status = -107

(14350,1):dlm_join_domain:1485 ERROR: status = -107

(14350,1):dlm_register_domain:1732 ERROR: status = -107

(14350,1):ocfs2_dlm_init:2662 ERROR: status = -107

(14350,1):ocfs2_mount_volume:1251 ERROR: status = -107

ocfs2: Unmounting device (253,3) on (node 1)

(11296,1):o2net_connect_expired:1637 ERROR: no connection established
with node 0 after 30.0 seconds, giving up and returning errors.

(4347,1):dlm_request_join:1033 ERROR: status = -107

(4347,1):dlm_try_to_join_domain:1207 ERROR: status = -107

(4347,1):dlm_join_domain:1485 ERROR: status = -107

(4347,1):dlm_register_domain:1732 ERROR: status = -107

(4347,1):ocfs2_dlm_init:2662 ERROR: status = -107

(4347,1):ocfs2_mount_volume:1251 ERROR: status = -107

ocfs2: Unmounting device (253,3) on (node 1)

(11296,1):o2net_connect_expired:1637 ERROR: no connection established
with node 0 after 30.0 seconds, giving up and returning errors.

(4948,1):dlm_request_join:1033 ERROR: status = -107

(4948,1):dlm_try_to_join_domain:1207 ERROR: status = -107

(4948,1):dlm_join_domain:1485 ERROR: status = -107

(4948,1):dlm_register_domain:1732 ERROR: status = -107

(4948,1):ocfs2_dlm_init:2662 ERROR: status = -107

(4948,1):ocfs2_mount_volume:1251 ERROR: status = -107

ocfs2: Unmounting device (253,3) on (node 1)

OCFS2 Node Manager 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build
0f78045c75c0174e50e4cf0934bf9eae)

OCFS2 DLM 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build
4ce8fae327880c466761f40fb7619490)

OCFS2 DLMFS 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build
4ce8fae327880c466761f40fb7619490)

OCFS2 User DLM kernel interface loaded

[root at nyclx2 ~]#

 

Reid McKinley


********************************************************************************************
This message, including any attachments, contains confidential information intended 
for a specific individual and purpose, and is protected by law. If you are not the intended 
recipient, please contact the sender immediately by reply e-mail and destroy all copies.
You are hereby notified that any disclosure, copying, or distribution of this message, or
the taking of any action based on it, is strictly prohibited.

TIAA-CREF
********************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090603/ff17f145/attachment-0001.html