<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>
Hello Scott,<div><br></div><div>I had help from an Oracle developer, Srinivas, and he fixed my issue. Thanks again Srini !!</div><div><br></div><div><br></div><div><i>Disclaimer: I am not technically knowledgable of the OCFS2 filesystem so I will explain what I understood from a discussion with Srini.</i></div><div><br></div><div><br></div><div>The main issue was that my filesystem is getting more and more fragmented and the default write pre-allocation window (<span style="background-color: rgb(255, 255, 255); color: rgb(42, 42, 42); line-height: 17px; font-size: 10pt;">localalloc bitmap) </span><span style="font-size: 10pt;">was set too big for such fragmented filesystem. For what I understood, w</span><span style="font-size: 10pt;">hen you are doing writes on a OCFS2 filesystem, the filesystem reserves a chunk of space before beginning to write onto the filesystem. Even if you are writing a very small file, the filesystem will always reserve that chunk of space (I assume this helps reduce fragmentation ?!)</span></div><div><span style="font-size: 10pt;"><br></span></div><div><span style="font-size: 10pt;">According to my filesystem setup, the size of the pre-allocated chunks was set at 136 MB so it meant the filesystem needed to find 136 MB of contiguous space every time a write was being done. That caused delays because it had hard time finding them ...</span></div><div><span style="font-size: 10pt;"><br></span></div><div><span style="font-size: 10pt;">Srini showed me how to reduce the pre-allocated chunks size (localalloc bitmap) to a smaller size (16 MB instead of 136 MB) and, since then, everything works as new. The solution to my problem was to </span><span style="font-size: 10pt;">add </span><span style="font-size: 10pt; background-color: rgb(255, 255, 255); color: rgb(42, 42, 42); line-height: 17px;"><b>localalloc=16</b></span><span style="font-size: 10pt;"> to my filesystem mount options, umount/mount the filesystem and everything was fixed.</span></div><div><span style="font-size: 10pt;"><br></span></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><font face="Courier New" size="2">[root@fileserv01 ~]# grep tier2-ocfs2 /etc/fstab </font></div></div><div><div><font size="2"><font face="Courier New">LABEL=tier2-ocfs2 /tier2-ocfs2 ocfs2 </font><span style="font-family: 'Courier New';">_netdev,nodev,noatime,errors=panic,data=writeback,noacl,nouser_xattr,commit=60,</span><b style="font-family: 'Courier New';">localalloc=16</b><span style="font-family: 'Courier New';"> </span><span style="font-family: 'Courier New';">0 0</span></font></div></div></blockquote><div><br></div><div><br></div><div>For info, you can view your current localalloc setting <span style="font-size: 10pt;">by looking at the fs_state in the debugfs.</span></div><div><br></div><div>You first need to mount the virtual debugfs filesystem if it's not already mounted: </div><div><br></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><font face="Courier New" size="2">[root@fileserv01 ~]# grep debugfs /etc/fstab </font></div></div><div><div><font face="Courier New" size="2">debugfs /sys/kernel/debug debugfs 0 0</font></div></div></blockquote><div><br></div><div>My localalloc settings before the change:</div><div><br></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><font size="2"><span style="font-family: 'Courier New';">[root@fileserv01 ~]# </span><span style="font-family: 'Courier New';">grep "LocalAlloc =" /sys/kernel/debug/ocfs2/*/fs_state</span></font><div><font face="Courier New" size="2">LocalAlloc => State: 1 Descriptor: 0 Size: <b>17441</b> bits Default: 29696 bits</font></div></blockquote><div><br></div><div>My localalloc settings after the change</div><div><br></div><div><blockquote style="margin-right: 0px; margin-left: 40px; padding-right: 0px; padding-left: 0px; border: none;"><font size="2"><span style="font-family: 'Courier New';">[root@fileserv01 ~]# </span><span style="font-family: 'Courier New';">grep "LocalAlloc =" /sys/kernel/debug/ocfs2/*/fs_state</span></font><div><font face="Courier New" size="2">LocalAlloc => State: 1 Descriptor: 0 Size: <b>2048</b> bits Default: 2048 bits</font></div><div><font face="Courier New"><br></font></div></blockquote></div><div><br></div><div>What you will be missing from my above post is the analysis from Srini where he found that they were not many "136 MB" chunks of contiguous space on my filesystem and therefore that tuning was definitely going to help.</div><div><br></div><div>I hope this post may help you and others.</div><div><br></div><div>Jeff</div><div><br></div><div>p.s. sadly, there is currently no defragmentation tool for OCFS2</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br><br><div><div id="SkyDrivePlaceholder"></div><hr id="stopSpelling">From: SKempinski@sjrwmd.com<br>To: jpaterson23@hotmail.com; ocfs2-users@oss.oracle.com<br>Subject: RE: [Ocfs2-users] OCFS2 hanging on writes<br>Date: Wed, 31 Oct 2012 12:30:00 +0000<br><br>
<style><!--
.ExternalClass .ecxhmmessage p
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Tahoma;}
--></style><style id="ecxowaParaStyle">
</style>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">Jeff,
<br>
<br>
Have you found a resolution to this issue?<br>
<br>
Lately we've been experiencing intermittent freezing, so I'm curious to hear more about your issue.<br>
<br>
Thanks, Scott<br>
<br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr tabindex="-1">
<div style="direction:ltr" id="ecxdivRpF60210"><font color="#000000" face="Tahoma" size="2"><b>From:</b> ocfs2-users-bounces@oss.oracle.com [ocfs2-users-bounces@oss.oracle.com] on behalf of Jeff Paterson [jpaterson23@hotmail.com]<br>
<b>Sent:</b> Thursday, October 25, 2012 9:32 PM<br>
<b>To:</b> ocfs2-users@oss.oracle.com<br>
<b>Subject:</b> [Ocfs2-users] OCFS2 hanging on writes<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr"><font size="2"><span style="white-space:nowrap;color:rgb(34,34,34);font-family:arial,sans-serif">Hello,</span><br>
</font>
<div>
<div dir="ltr">
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap">I would need help with our OCFS2 (1.8.0) filesystem. We are having problems with it since a couple days. When we write onto it, it hangs.</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap">The "hanging pattern" is easily reproductible. If I write a 1GB file on the filesystem, it does the following:</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - write ~200 MB of data on the disk in 1 second</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - freeze for about 10 seconds</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - write ~200 MB of data on the disk in 1 second</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - freeze for about 10 seconds</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - write ~200 MB of data on the disk in 1 second</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - freeze for about 10 seconds</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> (and so on)</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap">When the freezes occur:</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - other writes operations (from other processes) on the same node also freeze</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> - writes operations on other nodes are not affected by the freezes on another node</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"> </span></font></div>
<div><font size="2"><font color="#222222" face="arial, sans-serif"><span style="white-space:nowrap">Read operations (on any cluster node, even the one with frozen writes) don't seem to be affected by the freezes. One sure thing, read operations alone d</span></font><span style="white-space:nowrap;color:rgb(34,34,34);font-family:arial,sans-serif">on't
cause the filesystem freeze.</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap"><br>
</span></font></div>
<div><font color="#222222" face="arial, sans-serif" size="2"><span style="white-space:nowrap">
<div>For info, before the problem began to appear we could sustain 640 MB/s writes without any freeze.</div>
<div><br>
</div>
<div>I tried to mount the filesystem on a single node to avoid issues that could happen with inter-node communications and the problem was still there.</div>
<div><br>
</div>
<div><br>
</div>
<div><b><u>Filesystem details</u></b></div>
<div>
<ul>
<li>The filesystem has 18 TB and it is currently 72% full.</li><li>Mount options are the following: rw,nodev,_netdev,noatime,errors=panic,data=writeback,noacl,nouser_xattr,commit=60,heartbeat=local</li><li>All Features: backup-super strict-journal-super sparse extended-slotmap inline-data metaecc indexed-dirs refcount discontig-bg unwritten</li></ul>
</div>
<div><br>
</div>
<div><br>
</div>
<div>There is nothing special in the systems logs beside application errors caused by the freezes.</div>
<div><br>
</div>
<div><br>
</div>
<div>Would a fsck.ocfs2 help? How long would it take for 18 TB?</div>
<div><br>
</div>
<div>Is there a flag I can enable in debugfs.ocfs2 to get a better idea of what is happening and why it is freezing like that?</div>
<div><br>
</div>
<div><br>
</div>
<div>Any help would be greatly appreciated.</div>
<div><br>
</div>
<div>Thanks in advance,</div>
<div><br>
</div>
<div>Jeff</div>
</span></font></div>
</div>
</div>
<style><!--
.ExternalClass .ecxhmmessage p
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Tahoma;}
.ExternalClass
{direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt;}
.ExternalClass P
{margin-bottom:0;}
--></style></div>
</div>
</div>
</div></div></div> </div></body>
</html>