[Ocfs2-users] ocfs2 1.2.8 issues

Wed Jun 18 13:35:53 PDT 2008

Hi,

We recently upgraded ocfs2 to 1.2.8 from 1.2.3 on our 4 node RAC production systems.

On one of the nodes, we notice the following in the logs

Jun 18 02:00:57 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 02:00:57 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 02:01:02 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 02:01:02 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 02:01:07 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 02:01:07 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 02:01:12 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 02:01:12 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 02:01:17 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 02:01:17 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 02:01:22 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 02:01:22 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 02:01:28 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 02:01:28 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 02:01:33 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 02:01:33 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 02:01:38 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 02:01:38 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 02:01:43 db0 kernel: (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 02:01:43 db0 kernel: (6327,7):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: status = -107
Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: status = -107

We are suspecting that a backup that was scheduled to happen right around 2 am did not complete as a result of these errors.
The backup process is hung and we can still see it in the process list. 

We are not able to access the /orabackup folder (ocfs2 mounted) from any of the nodes either.

Right now we see the following in the logs

Jun 18 14:56:27 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 14:56:27 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 14:56:32 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 14:56:32 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 14:56:37 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 14:56:37 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 14:56:42 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 14:56:42 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 14:56:48 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 14:56:48 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 14:56:53 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 14:56:53 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 14:56:58 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 14:56:58 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2
Jun 18 14:57:03 db0 kernel: (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107
Jun 18 14:57:03 db0 kernel: (6327,3):dlm_wait_for_node_death:365 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of death of node 2

We need to fix this issue before the backup runs again at 2 am. Please advice what we should do to fix this.

Thanks,
Sincerely,
Saranya

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20080618/4b643a14/attachment.html 