<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>
<div><font size="3">Hello,</font></div><div><font size="3"><br></font></div><div><font size="3">A month ago, following the advice from Srini, we modified our localalloc to be 2048. It made sense at that time because our disk fragmentation was showing many "2048 clusters" of contiguous space available.</font></div><div><br></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><font face="Courier New"># ./stat_sysdir-analyze.sh stat_sysdir-output.txt-201210260843 </font></div></div><div><div><font face="Courier New">Number |</font></div></div><div><div><font face="Courier New">of |</font></div></div><div><div><font face="Courier New">clust. | Contiguous cluster size</font></div></div><div><div><font face="Courier New">--------------------------------</font></div></div><div><div><font face="Courier New"> 3981 510 and smaller</font></div></div><div><div><font face="Courier New"> 4929 511</font></div></div><div><div><font face="Courier New"> 2393 1024</font></div></div><div><div><font face="Courier New"> 63400 2048</font></div></div><div><div><font face="Courier New"> 2682 4096</font></div></div><div><div><font face="Courier New"> 2415 8192</font></div></div><div><div><font face="Courier New"> 430 16384 and bigger</font></div></div></blockquote><div><br></div><div><font size="3">It all worked very well and saved the day!</font></div><div><font size="3"><br></font></div><div><font size="3"><br></font></div><div><font size="3">Today, we began to have more issues with our disk writes hanging. This time though it might be a bit different ...</font></div><div><font size="3"><br></font></div><div><font size="3">First of all, while the filesystem is even more fragmented than before, there are still many "2048 clusters" available:</font></div><div><br></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><font face="Courier New"># ./stat_sysdir-analyze.sh stat_sysdir-output.txt-201211191829 </font></div></div><div><div><font face="Courier New">Number |</font></div></div><div><div><font face="Courier New">of |</font></div></div><div><div><font face="Courier New">clust. | Contiguous cluster size</font></div></div><div><div><font face="Courier New">--------------------------------</font></div></div><div><div><font face="Courier New"> 4407 510 and smaller</font></div></div><div><div><font face="Courier New"> 50270 511</font></div></div><div><div><font face="Courier New"> 13609 1024</font></div></div><div><div><font face="Courier New"> 10338 2048</font></div></div><div><div><font face="Courier New"> 649 4096</font></div></div><div><div><font face="Courier New"> 347 8192</font></div></div><div><div><font face="Courier New"> 641 16384 and bigger</font></div></div></blockquote><div><br></div><div><font size="3">and then, strangely, the problem appeared only every 20 minutes, lasting 2-3 minutes each time.</font></div><div><font size="3"><br></font></div><div><font size="3">Could the problem be related with some internal process in OCFS2 running every 20 minutes? Orphan scans?</font></div><div><font size="3"><br></font></div><div><font size="3">Or could the problem be due to the excessive fragmentation? Should I change the localalloc size to be lower than 511 clusters instead of 2048 (i.e. localalloc=3)? Would that actually worsen the fragmentation? For information, I don't think the filesystem is over-utilized. There are only 12TB used on a 18TB filesystem so that should leave enough space to properly deal with fragmentation.</font></div><div><font size="3"><br></font></div><div><font size="3"><br></font></div><div><font size="3">.... and finally, the problem suddenly disappeared after I shutdown 2 of the 3 nodes of the cluster. I re-enabled the 2 nodes 30 minutes later and the problem didn't come back. On the other hand, most of the employees using that filesystem finished their work day so the load is quite lower now.</font></div><div><font size="3"><br></font></div><div><font size="3"><br></font></div><div><font size="3">Any idea of what happened?</font></div><div><font size="3"><br></font></div><div><font size="3">Thanks in advance,</font></div><div><font size="3"><br></font></div><font size="3">Jeff<br><br>p.s. I attached the small script I wrote to analyze the data gathered from Srini's script (https://oss.oracle.com/~seeda/misc/stat_sysdir.sh)</font><div><font size="3"><br></font></div><div><font size="3"><br></font><br><br><br><br><div><div id="SkyDrivePlaceholder"></div><hr id="stopSpelling">From: SKempinski@sjrwmd.com<br>To: jpaterson23@hotmail.com; ocfs2-users@oss.oracle.com<br>Subject: RE: [Ocfs2-users] OCFS2 hanging on writes -- SOLVED<br>Date: Wed, 7 Nov 2012 23:44:06 +0000<br><br>
<style><!--
.ExternalClass .ecxhmmessage p
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Tahoma;}
--></style><style id="ecxowaParaStyle">
</style>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">Thank you for providing this write-up, Jeff. I think you explained it very well.<br>
<br>
This could be the issue we are experiencing, so I've implemented the localalloc mount option. Additionally, we have begun to move some of the data off to a new filesystem. For the past few days we have not seen the slowdown, so, for now at least, we have
a solution.<br>
<br>
Thank you, again.<br>
<br>
Scott<br>
<br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr tabindex="-1">
<div style="direction:ltr" id="ecxdivRpF98638"><font color="#000000" face="Tahoma" size="2"><b>From:</b> ocfs2-users-bounces@oss.oracle.com [ocfs2-users-bounces@oss.oracle.com] on behalf of Jeff Paterson [jpaterson23@hotmail.com]<br>
<b>Sent:</b> Wednesday, October 31, 2012 9:24 PM<br>
<b>To:</b> Scott Kempinski; ocfs2-users@oss.oracle.com<br>
<b>Subject:</b> Re: [Ocfs2-users] OCFS2 hanging on writes -- SOLVED<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">Hello Scott,
<div><br>
</div>
<div>I had help from an Oracle developer, Srinivas, and he fixed my issue. Thanks again Srini !!</div>
<div><br>
</div>
<div><br>
</div>
<div><i>Disclaimer: I am not technically knowledgable of the OCFS2 filesystem so I will explain what I understood from a discussion with Srini.</i></div>
<div><br>
</div>
<div><br>
</div>
<div>The main issue was that my filesystem is getting more and more fragmented and the default write pre-allocation window (<span style="background-color:rgb(255,255,255);color:rgb(42,42,42);line-height:17px;font-size:10pt">localalloc bitmap) </span><span style="font-size:10pt">was
set too big for such fragmented filesystem. For what I understood, w</span><span style="font-size:10pt">hen you are doing writes on a OCFS2 filesystem, the filesystem reserves a chunk of space before beginning to write onto the filesystem. Even if you are
writing a very small file, the filesystem will always reserve that chunk of space (I assume this helps reduce fragmentation ?!)</span></div>
<div><span style="font-size:10pt"><br>
</span></div>
<div><span style="font-size:10pt">According to my filesystem setup, the size of the pre-allocated chunks was set at 136 MB so it meant the filesystem needed to find 136 MB of contiguous space every time a write was being done. That caused delays because it
had hard time finding them ...</span></div>
<div><span style="font-size:10pt"><br>
</span></div>
<div><span style="font-size:10pt">Srini showed me how to reduce the pre-allocated chunks size (localalloc bitmap) to a smaller size (16 MB instead of 136 MB) and, since then, everything works as new. The solution to my problem was to </span><span style="font-size:10pt">add </span><span style="font-size:10pt;background-color:rgb(255,255,255);color:rgb(42,42,42);line-height:17px"><b>localalloc=16</b></span><span style="font-size:10pt"> to
my filesystem mount options, umount/mount the filesystem and everything was fixed.</span></div>
<div><span style="font-size:10pt"><br>
</span></div>
<blockquote style="border:none;padding:0px">
<div>
<div><font face="Courier New" size="2">[root@fileserv01 ~]# grep tier2-ocfs2 /etc/fstab </font></div>
</div>
<div>
<div><font size="2"><font face="Courier New">LABEL=tier2-ocfs2 /tier2-ocfs2 ocfs2 </font><span style="font-family:'Courier New'">_netdev,nodev,noatime,errors=panic,data=writeback,noacl,nouser_xattr,commit=60,</span><b style="font-family:'Courier New'">localalloc=16</b><span style="font-family:'Courier New'"> </span><span style="font-family:'Courier New'">0
0</span></font></div>
</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
<div>For info, you can view your current localalloc setting <span style="font-size:10pt">by looking at the fs_state in the debugfs.</span></div>
<div><br>
</div>
<div>You first need to mount the virtual debugfs filesystem if it's not already mounted: </div>
<div><br>
</div>
<blockquote style="border:none;padding:0px">
<div>
<div><font face="Courier New" size="2">[root@fileserv01 ~]# grep debugfs /etc/fstab </font></div>
</div>
<div>
<div><font face="Courier New" size="2">debugfs /sys/kernel/debug debugfs 0 0</font></div>
</div>
</blockquote>
<div><br>
</div>
<div>My localalloc settings before the change:</div>
<div><br>
</div>
<blockquote style="border:none;padding:0px"><font size="2"><span style="font-family:'Courier New'">[root@fileserv01 ~]# </span><span style="font-family:'Courier New'">grep "LocalAlloc =" /sys/kernel/debug/ocfs2/*/fs_state</span></font>
<div><font face="Courier New" size="2">LocalAlloc => State: 1 Descriptor: 0 Size:
<b>17441</b> bits Default: 29696 bits</font></div>
</blockquote>
<div><br>
</div>
<div>My localalloc settings after the change</div>
<div><br>
</div>
<div>
<blockquote style="margin-right:0px;margin-left:40px;padding-right:0px;padding-left:0px;border:none">
<font size="2"><span style="font-family:'Courier New'">[root@fileserv01 ~]# </span><span style="font-family:'Courier New'">grep "LocalAlloc =" /sys/kernel/debug/ocfs2/*/fs_state</span></font>
<div><font face="Courier New" size="2">LocalAlloc => State: 1 Descriptor: 0 Size:
<b>2048</b> bits Default: 2048 bits</font></div>
<div><font face="Courier New"><br>
</font></div>
</blockquote>
</div>
<div><br>
</div>
<div>What you will be missing from my above post is the analysis from Srini where he found that they were not many "136 MB" chunks of contiguous space on my filesystem and therefore that tuning was definitely going to help.</div>
<div><br>
</div>
<div>I hope this post may help you and others.</div>
<div><br>
</div>
<div>Jeff</div>
<div><br>
</div>
<div>p.s. sadly, there is currently no defragmentation tool for OCFS2</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<br>
<div>
<div id="ecxSkyDrivePlaceholder"></div>
<hr id="ecxstopSpelling">
From: SKempinski@sjrwmd.com<br>
To: jpaterson23@hotmail.com; ocfs2-users@oss.oracle.com<br>
Subject: RE: [Ocfs2-users] OCFS2 hanging on writes<br>
Date: Wed, 31 Oct 2012 12:30:00 +0000<br>
<br>
<style><!--
.ExternalClass .ecxhmmessage p
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Tahoma;}
--></style><style id="ecxowaParaStyle">
</style>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">Jeff,
<br>
<br>
Have you found a resolution to this issue?<br>
<br>
Lately we've been experiencing intermittent freezing, so I'm curious to hear more about your issue.<br>
<br>
Thanks, Scott<br>
<br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr tabindex="-1">
<div id="ecxdivRpF60210" style="direction:ltr"><font color="#000000" face="Tahoma" size="2"><b>From:</b> ocfs2-users-bounces@oss.oracle.com [ocfs2-users-bounces@oss.oracle.com] on behalf of Jeff Paterson [jpaterson23@hotmail.com]<br>
<b>Sent:</b> Thursday, October 25, 2012 9:32 PM<br>
<b>To:</b> ocfs2-users@oss.oracle.com<br>
<b>Subject:</b> [Ocfs2-users] OCFS2 hanging on writes<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr"><font size="2"><span style="white-space:nowrap;color:rgb(34,34,34);font-family:arial,sans-serif">Hello,</span><br>
</font>
<div>
<div dir="ltr">
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap">I would need help with our OCFS2 (1.8.0) filesystem. We are having problems with it since a couple days. When we write onto it, it hangs.</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap">The "hanging pattern" is easily reproductible. If I write a 1GB file on the filesystem, it does the following:</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - write ~200 MB of data on the disk in 1 second</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - freeze for about 10 seconds</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - write ~200 MB of data on the disk in 1 second</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - freeze for about 10 seconds</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - write ~200 MB of data on the disk in 1 second</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - freeze for about 10 seconds</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> (and so on)</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap">When the freezes occur:</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - other writes operations (from other processes) on the same node also freeze</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - writes operations on other nodes are not affected by the freezes on another node</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> </span></font></div>
<div><font size="2"><font color="#222222" face="arial, sans-serif"><span style="white-space:nowrap">Read operations (on any cluster node, even the one with frozen writes) don't seem to be affected by the freezes. One sure thing, read operations alone d</span></font><span style="white-space:nowrap;color:rgb(34,34,34);font-family:arial,sans-serif">on't
cause the filesystem freeze.</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap">
<div>For info, before the problem began to appear we could sustain 640 MB/s writes without any freeze.</div>
<div><br>
</div>
<div>I tried to mount the filesystem on a single node to avoid issues that could happen with inter-node communications and the problem was still there.</div>
<div><br>
</div>
<div><br>
</div>
<div><b><u>Filesystem details</u></b></div>
<div>
<ul>
<li>The filesystem has 18 TB and it is currently 72% full.</li><li>Mount options are the following: rw,nodev,_netdev,noatime,errors=panic,data=writeback,noacl,nouser_xattr,commit=60,heartbeat=local</li><li>All Features: backup-super strict-journal-super sparse extended-slotmap inline-data metaecc indexed-dirs refcount discontig-bg unwritten</li></ul>
</div>
<div><br>
</div>
<div><br>
</div>
<div>There is nothing special in the systems logs beside application errors caused by the freezes.</div>
<div><br>
</div>
<div><br>
</div>
<div>Would a fsck.ocfs2 help? How long would it take for 18 TB?</div>
<div><br>
</div>
<div>Is there a flag I can enable in debugfs.ocfs2 to get a better idea of what is happening and why it is freezing like that?</div>
<div><br>
</div>
<div><br>
</div>
<div>Any help would be greatly appreciated.</div>
<div><br>
</div>
<div>Thanks in advance,</div>
<div><br>
</div>
<div>Jeff</div>
</span></font></div>
</div>
</div>
<style><!--
.ExternalClass .ecxhmmessage p
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Tahoma;}
.ExternalClass
{direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt;}
.ExternalClass p
{margin-bottom:0;}
.ExternalClass
{direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt;}
.ExternalClass P
{margin-bottom:0;}
.ExternalClass
{scrollbar-base-color:undefined;scrollbar-highlight-color:undefined;scrollbar-darkshadow-color:undefined;scrollbar-track-color:undefined;scrollbar-arrow-color:undefined;}
--></style></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div></div> </div></body>
</html>