<p class="MsoNormal">We have an application that has many processing threads writing more than a billion files ranging from 2KB – 50KB, <span style>with 50% under 8KB</span> (currently there are 700 million files). The files are never deleted or modified –
they are written once, and read infrequently.
The files are hashed so that they are evenly distributed across ~1,000,000
subdirectories up to 3 levels deep, with up to 1000 files per directory. The directories are structured like this:</p><p class="MsoNormal"></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">0/00/00</p>
<p class="MsoNormal">0/00/01</p>
<p class="MsoNormal">…</p>
<p class="MsoNormal">F/FF/FE</p>
<p class="MsoNormal">F/FF/FF</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">The files need to be readable and writable across a number
of servers. The NetApp filer we purchased for this project has both NFS
and iSCSI capabilities.</p><p class="MsoNormal"></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">We first tried doing this via NFS. After writing 700 million files (12 TB) into
a single NetApp volume, file-write performance became abysmally slow. We can't create more than 200 files per
second on the NetApp volume, which is about 20% of our required performance
target of 1000 files per second. It
appears that most of the file-write time is going towards stat and inode-create
operations.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">So I now I’m trying the same thing with OCFS2 over
iSCSI. I created 16 luns on the NetApp. The 16 luns became 16 OCFS2 filesystems with
16 different mount points on our servers.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">With this configuration I was initially able to write ~1800
files per second. Now that I have completed
100 million files, performance has dropped to ~1500 files per second.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I’m using OEL 6.1 (2.6.32-100 kernel) with OCFS2 version
1.6. The application servers have 128GB of
memory. I created my OCFS2 filesystems
as follows:</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal" style="margin-left:.5in">mkfs.ocfs2 –T mail –b 4k –C 4k –L <my
label> --fs-features=indexed-dirs –fs-feature-level=max-features
/dev/mapper/<my device></p>
<p class="MsoNormal"> </p><p class="MsoNormal">And I mount them with these options:</p><p class="MsoNormal"></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal" style="margin-left:.5in">_netdev,commit=30,noatime,localflocks,localalloc=32</p>
<p class="MsoNormal" style="margin-left:.5in"> </p>
<p class="MsoNormal">So my questions are these:</p><p class="MsoNormal"><span style><br></span></p><p class="MsoNormal"><span style>1) Given a billion files sized 2KB – 50KB, </span><span style>with 50% under 8KB</span><span style>, do I have the optimal OCFS2 filesystem and mount-point configurations?</span></p>
<p class="MsoNormal"><span style><br></span></p><p class="MsoNormal"><span style>2) Should I split the files across even more
filesystems? Currently I have them split
across 16 OCFS2 filesystems.</span></p><p class="MsoNormal"> </p>
<p class="MsoNormal">Thanks a billion!</p>