<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman, new york, times, serif;font-size:10pt">Hi,<br><br>We recently upgraded ocfs2 to 1.2.8 from 1.2.3 on our 4 node RAC production systems.<br><br>On one of the nodes, we notice the following in the logs<br><br><font size="3">Jun 18 02:00:57 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 02:00:57 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 02:01:02 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 02:01:02 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 02:01:07 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 02:01:07 db0 kernel:
(6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 02:01:12 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 02:01:12 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 02:01:17 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 02:01:17 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 02:01:22 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 02:01:22 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 02:01:28 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status =
-107<br>Jun 18 02:01:28 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 02:01:33 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 02:01:33 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 02:01:38 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 02:01:38 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 02:01:43 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 02:01:43 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br></font><font size="3">Jun 18 00:09:00 db0 kernel:
(15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107<br></font><font size="2"><font color="#39275f"></font></font><font size="3">Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel:
(15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107<br>Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107</font><br><font size="3"><br>We are suspecting that a backup that was scheduled to happen right around 2 am did not complete as a result of these errors.<br>The backup process is hung and we can still see it in the process list. <br><br>We are not able to access the /orabackup folder (ocfs2 mounted) from any of the nodes either.<br><br>Right now we see the following in the logs<br><br></font>Jun 18 14:56:27 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status
= -107<br>Jun 18 14:56:27 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 14:56:32 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 14:56:32 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 14:56:37 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 14:56:37 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 14:56:42 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 14:56:42 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 14:56:48 db0 kernel:
(6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 14:56:48 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 14:56:53 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 14:56:53 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 14:56:58 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 14:56:58 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2<br>Jun 18 14:57:03 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Jun 18 14:57:03 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node
2<br><font size="3"><br>We need to fix this issue before the backup runs again at 2 am. Please advice what we should do to fix this.<br><br>Thanks,<br>Sincerely,<br>Saranya<br><br><br><br></font></div><br>
</body></html>