<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 31 March 2016 at 11:44, Eric Ren <span dir="ltr"><<a href="mailto:zren@suse.com" target="_blank">zren@suse.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hi,<div><div class="h5"><br>
<br>
On 03/31/2016 01:52 PM, Graeme Donaldson wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
On 31 March 2016 at 04:17, Eric Ren <<a href="mailto:zren@suse.com" target="_blank">zren@suse.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
Hi,<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
How did you perform the testing? It really matters. If you write a file on<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
shared disk from one node, and read this file from another node, without,<br>
or with very little interval, the writing IO speed could decrease by ~20<br>
times according my previous testing(just as a reference). It's a<br>
extremely<br>
bad situation for 2 nodes cluster, isn't?<br>
<br>
But it's incredible that in your case writing speed drop by >3000 times!<br>
<br>
</blockquote>
<br>
<br>
I simply used 'dd' to create a file with /dev/zero as a source. If there<br>
is<br>
a better way to do this I am all ears.<br>
<br>
</blockquote>
<br>
Alright, you just did a local IO on ocfs2, then the performance shouldn't<br>
be that bad. I guess the ocfs2 volume has been used over 60%? or seriously<br>
fragmented?<br>
Please give info with `df -h`, and super block with debugfs.ocfs2, and<br>
also the exact `dd` command you performed. Additionally, perform `dd` on<br>
each node.<br>
<br>
You know, ocfs2 is a shared disk fs. So 3 basic testing cases I can think<br>
of are:<br>
1. only one node of cluster do IO;<br>
2. more than one nodes of cluster perform IO, but each nodes just<br>
read/write its own file on shared disk;<br>
3. like 2), but some nodes read and some write a same file on shared disk.<br>
<br>
The above model is much theoretically simplified though. The practical<br>
scenarios could be much more complicated, like fragmentation issue that<br>
your case much likely is.<br>
</blockquote>
<br>
<br>
Here is all the output requested: <a href="http://pastebin.com/raw/BnJAQv9T" rel="noreferrer" target="_blank">http://pastebin.com/raw/BnJAQv9T</a><br>
</blockquote>
<br></div></div>
Paste directly in email is fine, that way all info would be archived;-)<br>
<br>
Hah, your disk is too small for ocfs2. What's your using scenario? it does really really matters. I guess files on ocfs2 volume are usually small ones? If so, 4k block size and 4k cluster size is little big? For<br>
more info to get optimal format parameter, please see mkfs.ocfs2 manuall, user maillist or google for the topic about ocfs2 formatting. I'm not good at it.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
It's interesting to me that you guessed the usage is over 60%. It is indeed<br>
sitting at 65%. Is the solution as simple as ensuring that an OCFS2<br>
filesystem doesn't go over the 60% usage mark? Or am I getting ahead of<br>
myself a little?<br>
</blockquote>
<br></span>
Yes, so use ocfs2 on top cLVM is good idea if you want it to get resilience. I'm not sure if tune.ocfs2 can change block size suchlike offline. FWIW, fragmentation is always evil;-)</blockquote><div><br></div><div>The files are the code and images, etc. that make up the customer's website. I ran something to show me the distributions of file sizes and only around 10% are under 4KB in size, so I wouldn't think that 4K block/cluster ought to be an issue. Perhaps it just down to the size. We're going to see if re-creating the filesystem with a 1K block (cluster cannot be smaller than 4K) and making it larger makes the issue go away.</div><div><br></div><div>For interest's sake, this is the size distribution on the volume. The first column is the size in bytes and the second column is the count of files that fall in the size range, so there are 545 files of 0 bytes, there are 265 files between 16 bytes and 32 bytes, etc.</div><div><br></div><div><div> 0 545</div><div> 1 12</div><div> 2 1</div><div> 8 9</div><div> 16 51</div><div> 32 265</div><div> 64 593</div><div> 128 899</div><div> 256 6902</div><div> 512 1247</div><div> 1024 10290</div><div> 2048 21719</div><div> 4096 46908</div><div> 8192 53438</div><div> 16384 42749</div><div> 32768 68509</div><div> 65536 62462</div><div> 131072 32245</div><div> 262144 13349</div><div> 524288 5458</div><div> 1048576 2193</div><div> 2097152 245</div><div> 4194304 66</div><div> 8388608 15</div><div> 67108864 3</div><div> 268435456 1</div><div> 536870912 1</div></div><div><br></div><div>Graeme</div></div></div></div>