[Ocfs2-users] ocfs cluster node keeps rebooting

[Bill Zha]

Hi Sunil and All,

We have a 10 Redhat4.2-node OCFS cluster running on version 1.2.5-6.  One of the node started to rebooted almost everyday since last week.  The entire cluster had been stable for the past 1 year or so.  I captured the following console output, can you or someone had the similar issue let me know what the possible cause of these reboots?

(25271,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156758.101016 now 1358156788.97593 dr 1358156758.101008 adv 1358156758.101022:1358156758.101024 func (5d21e188:507) 1357953447.247097:1357953447.247100)
(25267,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156758.666788 now 1358156788.663604 dr 1358156760.666794 adv 1358156758.666793:1358156758.666795 func (5d21e188:505) 1357953453.107343:1357953453.107349)
(25267,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156758.848933 now 1358156788.953367 dr 1358156760.847939 adv 1358156758.848939:1358156758.848941 func (0e6eb1eb:505) 1357965605.352156:1357965605.352162)
(25267,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156759.108373 now 1358156789.243003 dr 1358156761.108392 adv 1358156759.108376:1358156759.108378 func (af22ae1f:502) 1357914301.741127:1357914301.741130)
(25275,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156759.626366 now 1358156789.623629 dr 1358156789.622319 adv 1358156759.626369:1358156759.626371 func (abd851aa:505) 1357965605.363679:1357965605.363685)
(25275,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156759.656350 now 1358156789.913330 dr 1358156761.656039 adv 1358156759.656354:1358156759.656355 func (0e6eb1eb:502) 1357907401.318584:1357907401.318587)
(25275,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156759.663467 now 1358156790.203323 dr 1358156761.662745 adv 1358156759.663470:1358156759.663472 func (7dcded64:502) 1357875986.764566:1357875986.764568)
(25275,4):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1358156759.987324 now 1358156790.493342 dr 1358156761.987117 adv 1358156759.987327:1358156759.987329 func (6bcd2bc6:502) 1357875995.222247:1357875995.222255)
(25,7):o2hb_write_timeout:269 ERROR: Heartbeat write timeout to device dm-14 after 18 milliseconds
Heartbeat thread (25) printing last 24 blocking operations (cur = 11):
Heartbeat thread stuck at msleep, stuffing current time into that blocker (index 11)
Index 12: took 0 ms to do allocating bios for read
Index 13: took 0 ms to do bio alloc read
Index 14: took 0 ms to do bio add page read
Index 15: took 0 ms to do bio add page read
Index 16: took 0 ms to do submit_bio for read
Index 17: took 0 ms to do waiting for read completion
Index 18: took 0 ms to do bio alloc write
Index 19: took 0 ms to do bio add page write
Index 20: took 0 ms to do submit_bio for write
Index 21: took 0 ms to do checking slots
Index 22: took 0 ms to do waiting for write completion
Index 23: took 100897 ms to do msleep
Index 0: took 0 ms to do allocating bios for read
Index 1: took 0 ms to do bio alloc read
Index 2: took 0 ms to do bio add page read
Index 3: took 0 ms to do bio add page read
Index 4: took 0 ms to do submit_bio for read
Index 5: took 0 ms to do waiting for read completion
Index 6: took 0 ms to do bio alloc write
Index 7: took 0 ms to do bio add page write
Index 8: took 0 ms to do submit_bio for write
Index 9: took 0 ms to do checking slots
Index 10: took 0 ms to do waiting for write completion
Index 11: took 313 ms to do msleep
*** ocfs2 is very sorry to be fencing this system by restarting ***


Thank you so much for your help!


Bill