<div dir="ltr">Hi Sunil,<br>I checked the grub.conf file on the machine that reboots and there is no (deadline) reference to the io scheduler. I will check when back at work on Monday, but I suspect that we are just using the default io scheduler which would be cfq.<br>
<br>Just to briefly elaborate, our ocfs2 cluster consists of three nodes (one node (or its backup) mounts the ocfs2 filesystem read/write, while two other nodes mount the ocfs2 read only. It is always the read/write node that automatically reboots (fences as we know now) (though sometimes but not always the other systems need to be rebooted to get the system working properly.) The problem could be load-related but it is difficult to be sure.<br>
<br>I will discuss with my colleagues about whether to try the deadline option and/or set up a private network for the ocfs2 members. The deadline option is very easy to try (involving a small change to the grub.conf, and a reboot), while setting up the private network is a little bit more work but not hard.<br>
.<br>rgds<br>Derek<br><br><div class="gmail_quote">2008/8/24 Sunil Mushran <span dir="ltr"><<a href="mailto:sunil.mushran@oracle.com">sunil.mushran@oracle.com</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Which io scheduler are you using? On el4, it is best to use deadline.<br>
cfq is the default. Check the faq for details on using deadline.<br>
<br>
Derek Hazell wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div></div><div class="Wj3C7c">
<br>
Hi Ocfs2 user<br>
We got some relevant log messages (via a serial console) and via a putty session logged on a root.<br>
I suspect we need to set up a private network between the ocfs2 cluster members, is this right? Anything else we might need to do?<br>
regards, I appreciate your help<br>
<br>
Derek<br>
########################################################<br>
CURRENT O2CB CONFIG<br>
[root@sysname fs]# /etc/init.d/o2cb configure<br>
Configuring the O2CB driver.<br>
This will configure the on-boot properties of the O2CB driver.<br>
The following questions will determine whether the driver is loaded on<br>
boot. The current values will be shown in brackets ('[]'). Hitting<br>
<ENTER> without typing an answer will keep that current value. Ctrl-C<br>
will abort.<br>
Load O2CB driver on boot (y/n) [y]:<br>
Cluster to start on boot (Enter "none" to clear) [ocfs2]:<br>
Specify heartbeat dead threshold (>=7) [61]:<br>
Specify network idle timeout in ms (>=5000) [60000]: 120000<br>
Specify network keepalive delay in ms (>=1000) [2000]:<br>
Specify network reconnect delay in ms (>=2000) [2000]:<br>
Writing O2CB configuration: OK<br>
O2CB cluster ocfs2 already online<br>
[root@sysname fs]#<br>
##################<br>
TRACE OF ROOT PUTTY LOGIN<br>
<br>
[root@sysname ~]#<br></div></div>
Message from syslogd@sysname <mailto:<a href="mailto:syslogd@sysname" target="_blank">syslogd@sysname</a>> at Fri Aug 22 23:12:03 2008 ...<div class="Ih2E3d"><br>
sysname kernel: Heartbeat thread (11) printing last 24 blocking operations (cur = 8):<br>
<br></div>
Message from syslogd@sysname <mailto:<a href="mailto:syslogd@sysname" target="_blank">syslogd@sysname</a>> at Fri Aug 22 23:12:03 2008 ...<div class="Ih2E3d"><br>
sysname kernel: Heartbeat thread stuck at waiting for read completion, stuffing current time into that blocker (index 8)<br>
<br></div>
Message from syslogd@sysname <mailto:<a href="mailto:syslogd@sysname" target="_blank">syslogd@sysname</a>> at Fri Aug 22 23:12:03 2008 ...<div class="Ih2E3d"><br>
sysname kernel: Index 9: took 0 ms to do bio alloc read<br>
<br>
.<br>
.<br>
.<br>
<br></div>
Message from syslogd@sysname <mailto:<a href="mailto:syslogd@sysname" target="_blank">syslogd@sysname</a>> at Fri Aug 22 23:12:04 2008 ...<div class="Ih2E3d"><br>
sysname kernel: Index 3: took 5240 ms to do waiting for write completion<br>
<br></div>
Message from syslogd@sysname <mailto:<a href="mailto:syslogd@sysname" target="_blank">syslogd@sysname</a>> at Fri Aug 22 23:12:04 2008 ...<div class="Ih2E3d"><br>
sysname kernel: Index 4: took 0 ms to do allocating bios for read<br>
<br></div>
Message from syslogd@sysname <mailto:<a href="mailto:syslogd@sysname" target="_blank">syslogd@sysname</a>> at Fri Aug 22 23:12:04 2008 ...<div class="Ih2E3d"><br>
sysname kernel: Index 5: took 0 ms to do bio alloc read<br>
<br></div>
Message from syslogd@sysname <mailto:<a href="mailto:syslogd@sysname" target="_blank">syslogd@sysname</a>> at Fri Aug 22 23:12:04 2008 ...<div class="Ih2E3d"><br>
sysname kernel: Index 6: took 0 ms to do bio add page read<br>
<br></div>
Message from syslogd@sysname <mailto:<a href="mailto:syslogd@sysname" target="_blank">syslogd@sysname</a>> at Fri Aug 22 23:12:04 2008 ...<div class="Ih2E3d"><br>
sysname kernel: Index 7: took 0 ms to do submit_bio for read<br>
<br></div>
Message from syslogd@sysname <mailto:<a href="mailto:syslogd@sysname" target="_blank">syslogd@sysname</a>> at Fri Aug 22 23:12:04 2008 ...<div><div></div><div class="Wj3C7c"><br>
sysname kernel: Index 8: took 120303 ms to do waiting for read completion<br>
#############<br>
TRACE OF SERIAL CONSOLE:<br>
(11,1):o2hb_write_timeout:269 ERROR: Heartbeat write timeout to device emcpowerb1 after 120000 milliseconds<br>
Heartbeat thread (11) printing last 24 blocking operations (cur = 8):<br>
Heartbeat thread stuck at waiting for read completion, stuffing current time into that blocker (index 8)<br>
Index 9: took 0 ms to do bio alloc read<br>
Index 10: took 0 ms to do bio add page read<br>
Index 11: took 0 ms to do submit_bio for read<br>
Index 12: took 3025 ms to do waiting for read completion<br>
Index 13: took 0 ms to do bio alloc write<br>
Index 14: took 0 ms to do bio add page write<br>
Index 15: took 0 ms to do submit_bio for write<br>
Index 16: took 0 ms to do checking slots<br>
Index 17: took 7221 ms to do waiting for write completion<br>
Index 18: took 0 ms to do allocating bios for read<br>
Index 19: took 0 ms to do bio alloc read<br>
Index 20: took 0 ms to do bio add page read<br>
Index 21: took 0 ms to do submit_bio for read<br>
Index 22: took 3892 ms to do waiting for read completion<br>
Index 23: took 0 ms to do bio alloc write<br>
Index 0: took 0 ms to do bio add page write<br>
Index 1: took 0 ms to do submit_bio for write<br>
Index 2: took 0 ms to do checking slots<br>
Index 3: took 5240 ms to do waiting for write completion<br>
Index 4: took 0 ms to do allocating bios for read<br>
Index 5: took 0 ms to do bio alloc read<br>
Index 6: took 0 ms to do bio add page read<br>
Index 7: took 0 ms to do submit_bio for read<br>
Index 8: took 120303 ms to do waiting for read completion<br>
*** ocfs2 is very sorry to be fencing this system by restarting ***<br>
Bootdata ok (command line is ro root=/dev/VolGroup_ID_12182/LogVol1 console=ttyS0,9600n8)<br>
################################################################################<br>
-----Original Message-----<br></div></div><div class="Ih2E3d">
From: <a href="mailto:ocfs2-users-bounces@oss.oracle.com" target="_blank">ocfs2-users-bounces@oss.oracle.com</a> <mailto:<a href="mailto:ocfs2-users-bounces@oss.oracle.com" target="_blank">ocfs2-users-bounces@oss.oracle.com</a>> [mailto:<a href="mailto:ocfs2-users-bounces@oss.oracle.com" target="_blank">ocfs2-users-bounces@oss.oracle.com</a> <mailto:<a href="mailto:ocfs2-users-bounces@oss.oracle.com" target="_blank">ocfs2-users-bounces@oss.oracle.com</a>>] On Behalf Of Sunil Mushran<br>
Sent: Tuesday, 19 August 2008 3:56 AM<br>
To: _Derek Hazell (Internet)<br></div><div class="Ih2E3d">
Cc: <a href="mailto:ocfs2-users@oss.oracle.com" target="_blank">ocfs2-users@oss.oracle.com</a> <mailto:<a href="mailto:ocfs2-users@oss.oracle.com" target="_blank">ocfs2-users@oss.oracle.com</a>><br>
Subject: Re: [Ocfs2-users] ocfs2 issue? : unexplained reboots of RHEL 4 server (kernel:2.6.9-42.0.2.ELs)<br>
<br>
Configure a netdump or netconsole server. It will catch the relevant<br>
<br>
messages.<br>
<br>
################################################################################<br>
<br></div><div class="Ih2E3d">
------------------------------------------------------------------------<br>
<br>
_______________________________________________<br>
Ocfs2-users mailing list<br>
<a href="mailto:Ocfs2-users@oss.oracle.com" target="_blank">Ocfs2-users@oss.oracle.com</a><br>
<a href="http://oss.oracle.com/mailman/listinfo/ocfs2-users" target="_blank">http://oss.oracle.com/mailman/listinfo/ocfs2-users</a><br>
</div></blockquote>
<br>
</blockquote></div><br><br clear="all"><br>-- <br> best wishes<br><br>Derek<br><br><br><br>Psalm 71:14 "But as for me, I will always have hope; I will praise you more and more". (NIV)<br>########################<br>
new home ph: 02-9701-0841<br>new mobile ph: 0458-588-821<br>(or +61-458-588-821 from overseas)<br>email : <a href="mailto:derek.hazell@gmail.com">derek.hazell@gmail.com</a><br>skype : dereklife2005<br>msn : <a href="mailto:derek_hazell@yahoo.com">derek_hazell@yahoo.com</a><br>
yahoo messenger : derek_hazell<br>########################<br>
</div>