<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman, new york, times, serif;font-size:10pt">Hi <br>Thanks for the quick response.<br>Would reboot the nodes solve the issue temporarily before we schedule an upgrade?<br><br>Thanks,<br>Sincerely,<br>Saranya Sivakumar<br><div><div style="font-family: times new roman,new york,times,serif; font-size: 10pt;"><br><br><div style="font-family: times new roman,new york,times,serif; font-size: 12pt;">----- Original Message ----<br>From: Sunil Mushran <Sunil.Mushran@oracle.com><br>To: Saranya Sivakumar <sarlavk@yahoo.com><br>Cc: ocfs2-users@oss.oracle.com<br>Sent: Wednesday, June 18, 2008 3:54:31 PM<br>Subject: Re: [Ocfs2-users] ocfs2 1.2.8 issues<br><br>
<a href="http://oss.oracle.com/projects/ocfs2/news/article_18.html" target="_blank">http://oss.oracle.com/projects/ocfs2/news/article_18.html</a><br><br>This is oss bugzilla#919 that has been fixed in 1.2.9-1.<br><br>Saranya Sivakumar wrote:<br>> Hi,<br>><br>> We recently upgraded ocfs2 to 1.2.8 from 1.2.3 on our 4 node RAC <br>> production systems.<br>><br>> On one of the nodes, we notice the following in the logs<br>><br>> Jun 18 02:00:57 db0 kernel: <br>> (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 02:00:57 db0 kernel: (6327,7):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 02:01:02 db0 kernel: <br>> (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 02:01:02 db0 kernel: (6327,7):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting
5000ms for notification of <br>> death of node 2<br>> Jun 18 02:01:07 db0 kernel: <br>> (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 02:01:07 db0 kernel: (6327,7):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 02:01:12 db0 kernel: <br>> (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 02:01:12 db0 kernel: (6327,7):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 02:01:17 db0 kernel: <br>> (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 02:01:17 db0 kernel: (6327,7):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 02:01:22 db0 kernel: <br>>
(6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 02:01:22 db0 kernel: (6327,7):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 02:01:28 db0 kernel: <br>> (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 02:01:28 db0 kernel: (6327,7):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 02:01:33 db0 kernel: <br>> (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 02:01:33 db0 kernel: (6327,7):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 02:01:38 db0 kernel: <br>> (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 02:01:38 db0 kernel:
(6327,7):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 02:01:43 db0 kernel: <br>> (6327,7):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 02:01:43 db0 kernel: (6327,7):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel:
(15652,1):dlm_purge_lockres:189 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: <br>> status = -107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_drop_lockres_ref:2284 ERROR: <br>> status =
-107<br>> Jun 18 00:09:00 db0 kernel: (15652,1):dlm_purge_lockres:189 ERROR: <br>> status = -107<br>><br>> We are suspecting that a backup that was scheduled to happen right <br>> around 2 am did not complete as a result of these errors.<br>> The backup process is hung and we can still see it in the process list.<br>><br>> We are not able to access the /orabackup folder (ocfs2 mounted) from <br>> any of the nodes either.<br>><br>> Right now we see the following in the logs<br>><br>> Jun 18 14:56:27 db0 kernel: <br>> (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 14:56:27 db0 kernel: (6327,3):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 14:56:32 db0 kernel: <br>> (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 14:56:32 db0 kernel:
(6327,3):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 14:56:37 db0 kernel: <br>> (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 14:56:37 db0 kernel: (6327,3):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 14:56:42 db0 kernel: <br>> (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 14:56:42 db0 kernel: (6327,3):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 14:56:48 db0 kernel: <br>> (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 14:56:48 db0 kernel: (6327,3):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for
notification of <br>> death of node 2<br>> Jun 18 14:56:53 db0 kernel: <br>> (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 14:56:53 db0 kernel: (6327,3):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 14:56:58 db0 kernel: <br>> (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 14:56:58 db0 kernel: (6327,3):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>> Jun 18 14:57:03 db0 kernel: <br>> (6327,3):dlm_send_remote_convert_request:398 ERROR: status = -107<br>> Jun 18 14:57:03 db0 kernel: (6327,3):dlm_wait_for_node_death:365 <br>> 2CED57AE61DE47BA8D2EECE680EFFA6C: waiting 5000ms for notification of <br>> death of node 2<br>><br>> We need to fix this issue before the backup runs
again at 2 am. Please <br>> advice what we should do to fix this.<br>><br>> Thanks,<br>> Sincerely,<br>> Saranya<br>><br>><br>><br>><br>> ------------------------------------------------------------------------<br>><br>> _______________________________________________<br>> Ocfs2-users mailing list<br>> <a ymailto="mailto:Ocfs2-users@oss.oracle.com" href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a><br>> <a href="http://oss.oracle.com/mailman/listinfo/ocfs2-users" target="_blank">http://oss.oracle.com/mailman/listinfo/ocfs2-users</a><br><br></div></div></div></div><br>
</body></html>