<div dir="ltr">1.2.5 is 6+ year old release. You may want to use something more current.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jan 14, 2013 at 12:06 PM, Bill Zha <span dir="ltr"><<a href="mailto:lfl2000us@yahoo.com" target="_blank">lfl2000us@yahoo.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Sunil and All,<br>
<br>
We have a 10 Redhat4.2-node OCFS cluster running on version 1.2.5-6. One of the node started to rebooted almost everyday since last week. The entire cluster had been stable for the past 1 year or so. I captured the following console output, can you or someone had the similar issue let me know what the possible cause of these reboots?<br>
<br>
(25271,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156758.101016 now 1358156788.97593 dr 1358156758.101008 adv 1358156758.101022:1358156758.101024 func (5d21e188:507) 1357953447.247097:1357953447.247100)<br>
(25267,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156758.666788 now 1358156788.663604 dr 1358156760.666794 adv 1358156758.666793:1358156758.666795 func (5d21e188:505) 1357953453.107343:1357953453.107349)<br>
(25267,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156758.848933 now 1358156788.953367 dr 1358156760.847939 adv 1358156758.848939:1358156758.848941 func (0e6eb1eb:505) 1357965605.352156:1357965605.352162)<br>
(25267,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156759.108373 now 1358156789.243003 dr 1358156761.108392 adv 1358156759.108376:1358156759.108378 func (af22ae1f:502) 1357914301.741127:1357914301.741130)<br>
(25275,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156759.626366 now 1358156789.623629 dr 1358156789.622319 adv 1358156759.626369:1358156759.626371 func (abd851aa:505) 1357965605.363679:1357965605.363685)<br>
(25275,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156759.656350 now 1358156789.913330 dr 1358156761.656039 adv 1358156759.656354:1358156759.656355 func (0e6eb1eb:502) 1357907401.318584:1357907401.318587)<br>
(25275,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156759.663467 now 1358156790.203323 dr 1358156761.662745 adv 1358156759.663470:1358156759.663472 func (7dcded64:502) 1357875986.764566:1357875986.764568)<br>
(25275,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156759.987324 now 1358156790.493342 dr 1358156761.987117 adv 1358156759.987327:1358156759.987329 func (6bcd2bc6:502) 1357875995.222247:1357875995.222255)<br>
(25,7):o2hb_write_timeout:269 ERROR: Heartbeat write timeout to device dm-14 after 18 milliseconds<br>
Heartbeat thread (25) printing last 24 blocking operations (cur = 11):<br>
Heartbeat thread stuck at msleep, stuffing current time into that blocker (index 11)<br>
Index 12: took 0 ms to do allocating bios for read<br>
Index 13: took 0 ms to do bio alloc read<br>
Index 14: took 0 ms to do bio add page read<br>
Index 15: took 0 ms to do bio add page read<br>
Index 16: took 0 ms to do submit_bio for read<br>
Index 17: took 0 ms to do waiting for read completion<br>
Index 18: took 0 ms to do bio alloc write<br>
Index 19: took 0 ms to do bio add page write<br>
Index 20: took 0 ms to do submit_bio for write<br>
Index 21: took 0 ms to do checking slots<br>
Index 22: took 0 ms to do waiting for write completion<br>
Index 23: took 100897 ms to do msleep<br>
Index 0: took 0 ms to do allocating bios for read<br>
Index 1: took 0 ms to do bio alloc read<br>
Index 2: took 0 ms to do bio add page read<br>
Index 3: took 0 ms to do bio add page read<br>
Index 4: took 0 ms to do submit_bio for read<br>
Index 5: took 0 ms to do waiting for read completion<br>
Index 6: took 0 ms to do bio alloc write<br>
Index 7: took 0 ms to do bio add page write<br>
Index 8: took 0 ms to do submit_bio for write<br>
Index 9: took 0 ms to do checking slots<br>
Index 10: took 0 ms to do waiting for write completion<br>
Index 11: took 313 ms to do msleep<br>
*** ocfs2 is very sorry to be fencing this system by restarting ***<br>
<br>
<br>
Thank you so much for your help!<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
Bill<br>
</font></span></blockquote></div><br></div>