<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 30 March 2016 at 14:24, Eric Ren <span dir="ltr"><<a href="mailto:zren@suse.com" target="_blank">zren@suse.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
We're seeing very poor write performance on a cluster that was built<br>
roughly a year ago. I am by no means an expert on OCFS2, nor the DRBD layer<br>
that we have under it. We do have several clusters that are configured in<br>
much the same way via our puppet infrastructure, yet this particular one<br>
gives us write speeds around the 15 kilobyte/sec mark, where some of our<br>
other clients do 55 megabytes/sec on similar hardware.<br>
</blockquote>
<br></span>
How did you perform the testing? It really matters. If you write a file on shared disk from one node, and read this file from another node, without, or with very little interval, the writing IO speed could decrease by ~20 times according my previous testing(just as a reference). It's a extremely bad situation for 2 nodes cluster, isn't?<br>
<br>
But it's incredible that in your case writing speed drop by >3000 times!</blockquote><div><br></div><div>I simply used 'dd' to create a file with /dev/zero as a source. If there is a better way to do this I am all ears.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I realise that this is all very vague, so for now I am just hoping for<br>
general pointers on where to start in diagnosing this, from which I can do<br>
more research and then hopefully revisit the thread with more detailed<br>
questions and data.<br>
<br>
Some basic info to get started:<br>
<br>
O/S: Debian Wheezy<br>
Kernel: Linux hostname 3.2.0-4-amd64 #1 SMP Debian 3.2.73-2+deb7u3 x86_64<br>
GNU/Linux<br>
ocfs2-tools: 1.6.4-1+deb7u1<br>
2 servers in the cluster. OCFS2 filesystem lives on a DRBD dual-primary<br>
device, which itself is built on an LVM volume, whose VG lives on a RAID1<br>
pair of 1TB SATA HDDs.<br>
</blockquote>
<br></span>
Could you firstly do test on LVM, then DRBD, and then OCFS2? Let's blame on them more fairly.<br>
<br>
Eric<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
</span></blockquote></blockquote></div><br></div><div class="gmail_extra">If I do a similar write of a file to a directory that exists on a LVM LV I get roughly 100 megabytes/sec.</div><div class="gmail_extra"><br></div><div class="gmail_extra">I can't write straight to the DRBD device, as that would entail wiping the customer's OCFS2 filesystem, which I cannot do.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Graeme</div></div>