Another update on the situation.<br><br>Yesterday, we have a chance to unplug system from services, so I have tried a little experiment with the system with iozone. The results are interesting.<br><br>- The "lock-up" didn't just response to HTTP load. After first-mount, if I ran "iozone -s 1M -r 1 -O" on the OCFS2-mounted path, the lock-up will occur immediately. It will freeze for about a minute. After the freeze, iozone will run quite fast after that. There are slight lock-up (1-2 seconds) when I did "for i in `seq 1 50`; do iozone -s 1M -r 1 -O; done.<br>
<br>- During lock-up, if I run iostat -x 1 | grep sdg (the SAN disk), it appear that the %util is almost 100%, and the service time greatly increase in twice or thrice the amount. The I/O operation per second is only about 150-160 at the time, but %util reach almost 99%. <br>
<br>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util<br>sdg1 0.00 0.00 161.00 0.00 1288.00 0.00 8.00 0.96 5.99 5.97 96.10<br><br> This is very odd, since the normal operation (servign HTTP load), the result of iostat is much better<br>
<br>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util<br>sdg1 4.00 617.00 427.00 25.00 21496.00 5136.00 58.92 2.21 4.90 1.77 80.10<br><br> But I think it now proved one thing. The system is not actually lock-up, it just busy doing I/O to backend storage. Could this mean that there is something wrong with the storage (driver + SAN configuration)?. Note that we are using HP MSA2312fc.<br>
<br>- This happened on both machine connect to SAN. It would happened almost all the time if we did the iozone on both machine at the same time. The lock-up would occur on each machine in turn. <br><br>- Another interesting point is that, the amount of time it lock-up is usually exactly the same on both machine. Approximately 145 seconds, counted by looking to the number of lines in "iostat -x 1" output, starting when %util ramped up and finish when it cooled down. <br>
<br>- I tried appending "pci=nomsi", according to bug report on this <a href="http://bugzilla.kernel.org/show_bug.cgi?id=11646">http://bugzilla.kernel.org/show_bug.cgi?id=11646</a>, but it didn't help at all.<br>
<br>- I just found out that the FC HBA Card we installed on the second machine is operating at the different speed than the first. The 1st run at 4Gb and 2nd run at 2Gb. Anyway this is just temporary card, vendor will install the newer card (presumably the same model) soon. Note that both machine are the same model, Dell PowerEdge 2950 DualxQuad Cores Intel Xeon 2.66GHz. <br>
<br>Our kernel and driver information<br><br>uname -a output<br><div style="margin-left: 40px;">Linux <a href="http://host1.mydomain.com">host1.mydomain.com</a> 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 16:12:36 EST 2009 x86_64 x86_64 x86_64 GNU/Linux<br>
<br>Linux <a href="http://host2.mydomain.com">host2.mydomain.com</a> 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010 x86_64 x86_64 x86_64 GNU/Linux<br></div><br>(NOTE: the different in kernel version, do we need to run exactly the same version on both machine?)<br>
<br>OCFS version<br><div style="margin-left: 40px;">ocfs2-tools-1.4.3-1.el5<br>ocfs2console-1.4.3-1.el5<br>ocfs2-2.6.18-164.6.1.el5-1.4.4-1.el5 <-- 1st machine<br>ocfs2-2.6.18-164.11.1.el5-1.4.4-1.el5 <-- 2nd machine<br>
</div><br>Both using qla2xxx driver.<br><div style="margin-left: 40px;">filename: /lib/modules/2.6.18-164.6.1.el5/kernel/drivers/scsi/qla2xxx/qla2xxx.ko<br>version: 8.03.00.1.05.05-k<br>license: GPL<br>
description: QLogic Fibre Channel HBA Driver<br>author: QLogic Corporation<br>srcversion: 109451851724AE603A18126<br></div><br><br><div class="gmail_quote">2010/1/29 Somsak Sriprayoonsakul <span dir="ltr"><<a href="mailto:somsaks@gmail.com">somsaks@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">An update for the situation.<br><br>It seems that the lockup situation when we brought in the new server is not quite exactly the same behavior as before.<br>
<br>Running "iozone -O" on the new machine, which mount OCFS2 without serving the content yet (the load is very low), could leads to OCFS2 lockup for 60-90 seconds. This happened very frequently (running iozone 3 times and this happened once). On the other hand, I run "iozone -O 50" times on the older server without having this lockup. Sometime it slow (few seconds), but not this slow.<br>
<br>When the lockup occur, both machine would hang. Even simple ls just hang. The funny thing is that, sometime, iozone command is actually not yet running. It just start to spawn the command and lockup occurred, so strace show nothing suspicious. In case it hang in the middle of iozone, the system call the takes long time vary from sync, read, write, and lseek.<br>
<br>I tried simpler things like repeatedly do "ls" or just "echo hello world > test" but it didn't cause this problem. The command that will cause this lockup almost certainly is just "iozone -O" and it must be called on the second server.<br>
<br><br><div class="gmail_quote">2010/1/28 Somsak Sriprayoonsakul <span dir="ltr"><<a href="mailto:somsaks@gmail.com" target="_blank">somsaks@gmail.com</a>></span><div><div></div><div class="h5"><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
For our case log file shouldn't be the problem since it's store locally. But fragmentation might be the cause. We serve a lot of small files (mostly html and jpg) out of OCFS2. And the file come in and go out very frequently, old topic will be paged out to archive directory out of OCFS2, while new topic are created on OCFS2. Thanks for pointing the bug report then. <br>
<br>Right now we are using 4 slots and planning to serving about 4 in near future. I think we shouldn't decrease the number of slot right now. <br><br>Are there any other statistics that we should observe for? <br><br>
<div class="gmail_quote">2010/1/28 Brad Plant <span dir="ltr"><<a href="mailto:bplant@iinet.net.au" target="_blank">bplant@iinet.net.au</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div></div><div>
Hi Somsak,<br>
<br>
I observed high loads and apache slowness when there was insufficient contiguous free space due to fragmentation. I believe it was because apache couldn't write it's log files efficiently. We had 2 apache nodes and I found that stopping apache on the problem node resolved the problem until I deleted lots of unused files.<br>
<br>
My symptoms don't seem to suggest a slow open() syscall like your strace results are showing, but I certainly got the high load and poor apache performance. It might be worth checking out anyway. There is a bug report and we're just waiting for the patch to get reviewed and made publicly available.<br>
<br>
<a href="http://oss.oracle.com/bugzilla/show_bug.cgi?id=1189" target="_blank">http://oss.oracle.com/bugzilla/show_bug.cgi?id=1189</a><br>
<br>
Cheers,<br>
<font color="#888888"><br>
Brad<br>
</font><div><div></div><div><br>
<br>
<br>
On Tue, 19 Jan 2010 16:12:07 +0700<br>
Somsak Sriprayoonsakul <<a href="mailto:somsaks@gmail.com" target="_blank">somsaks@gmail.com</a>> wrote:<br>
<br>
> Hello,<br>
><br>
> We are using OCFS2 version 1.4.3 on CentOS5, x86_64 with 8GB memory. The<br>
> underlying storage is HP 2312fc smart array equipped with 12 SAS 15K rpm,<br>
> configured as RAID10 using 10 HDDs + 2 spares. The array has about 4GB<br>
> cache. Communication is 4Gbps FC, through HP StorageWorks 8/8 Base e-port<br>
> SAN Switch. Right now we only have this machine connect to the SAN through<br>
> switch, but we plan to add more machine to utilize this SAN system.<br>
><br>
> Our application is apache version 1.3.41, mostly serving static HTML file +<br>
> few PHP. Note that, we have to downgrade to 1.3.41 due to our application<br>
> requirement. Apache is configured on has 500 MaxClients.<br>
><br>
> The storage OCFS2 are formatted with mkfs.ocfs2 without any special option<br>
> on. It run directly from multipath'ed SAN storage without LVM or software<br>
> RAID. We mount OCFS2 with noatime, commit=15, and data=writeback (as well as<br>
> heartbeat=local). Our cluster.conf is like this<br>
><br>
> cluster:<br>
> node_count = 1<br>
> name = mycluster<br>
><br>
> node:<br>
> ip_port = 7777<br>
> ip_address = 203.123.123.123<br>
> number = 1<br>
> name = <a href="http://mycluster.mydomain.com" target="_blank">mycluster.mydomain.com</a><br>
> cluster = mycluster<br>
><br>
> (NOTE: Some details are neglected here, such as hostname and IP address).<br>
><br>
> Periodically, we found that the file system work very slow. I think that it<br>
> happened once every few minutes. When the file system slow, httpd process<br>
> CPU utilization will goes much higher to about 50% or above. I tried to<br>
> debug this slow by creating a small script that periodically do<br>
><br>
> strace -f dd if=/dev/zero of=/san/testfile bs=1k count=1<br>
><br>
> And time the speed of dd, usually dd will finish within subsecond, but<br>
> periodically dd will be much slower to about 30-60 seconds. Strace output<br>
> show this.<br>
><br>
> 0.000026 open("/san/testfile", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 1<br>
> 76.418696 rt_sigaction(SIGUSR1, NULL, {SIG_DFL, [], 0}, 8) = 0<br>
><br>
> So I presume that this mean the open system call is periodically very slow.<br>
> I did about 5-10 tests which yield similar strace'd results (ranging from<br>
> just 5-7 seconds to 80 seconds).<br>
><br>
> So my question is, what could be the cause of this slowness? How could I<br>
> debug this deeper? On which point should we optimize the file system?<br>
><br>
> We are in the process of purchasing and adding more web servers to the<br>
> system and use reverse proxy to load balance between two servers. We just<br>
> want to make sure that this will not make situation worst.<br>
</div></div><br></div></div>_______________________________________________<br>
Ocfs2-users mailing list<br>
<a href="mailto:Ocfs2-users@oss.oracle.com" target="_blank">Ocfs2-users@oss.oracle.com</a><br>
<a href="http://oss.oracle.com/mailman/listinfo/ocfs2-users" target="_blank">http://oss.oracle.com/mailman/listinfo/ocfs2-users</a><br></blockquote></div><br>
</blockquote></div></div></div><br>
</blockquote></div><br>