<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Greetings,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’m rather new to OCFS2, so please forgive any glaringly ignorant statements.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’m evaluating file systems and storage layout for a simple 2 node mail cluster using Maildir email directories. I have created a 2 node cluster with related tutorials. The problem I’m seeing is that general file access using cp, find, du, ls, etc. is a significant factor slower on ocfs2 than alternative local and remote disk configurations. I’m hoping someone can clue me into whether this behavior is normal, or if I’m missing something in my lab.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>*Hosts are identical CentOS 5.5 virtual machines (VMware) with 2.6.18-238.19.1.el5. (2 ESXi hosts)<o:p></o:p></p><p class=MsoNormal>*OCFS2 build is ocfs2-2.6.18-238.19.1.el5-1.4.7-1.el5 (tools v 1.4.4-1.el5).<o:p></o:p></p><p class=MsoNormal>*SAN is an EMC Clariion. LUN is accessed via iSCSI with EMC PowerPath 5.5.0.00.00-275<o:p></o:p></p><p class=MsoNormal>*Nodes share a gigabit network for their private interconnect via two interconnected switches (ESXi host into each).<o:p></o:p></p><p class=MsoNormal>*Test data is a 181MB Maildir directory (~12K emails) copied to various types of storage.<o:p></o:p></p><p class=MsoNormal>*Tests involve simple bash scripts running (bash) time with the mentioned command line utilities and strace inspection.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The OCFS2 file system was created with the following (mount cannot load xattr or extended-slotmap added with max-features):<o:p></o:p></p><p class=MsoNormal>mkfs.ocfs2 -N 2 -T mail --fs-features=backup-super,sparse,unwritten,inline-data -v /dev/emcpowera<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Mount options are limited to ‘_netdev’ at the moment. I’ve read a bit about changing ‘data’ from ordered to writeback, but that seems to be related to waits on flushing cache to disk. So far, I’m just focusing on reads/lstats.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>With a maildir in place, any operation that must inspect all files takes quite a while to complete without cached entries. The alarming thing is the discrepancy between my OCFS2 data and identical data on local, NFS and iSCSI mounts.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Here’s some simple data that should illustrate my problem and my confusion:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Command: ‘du –hs /path/to/maildir/on/various/mounts<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Storage Real time to complete Min:Sec<o:p></o:p></p><p class=MsoNormal>----------------------------------------------------------------------<o:p></o:p></p><p class=MsoNormal>Local disk 0:0.078<o:p></o:p></p><p class=MsoNormal>NFS 0:2<o:p></o:p></p><p class=MsoNormal>iSCSI (EXT3) 0:1.7<o:p></o:p></p><p class=MsoNormal>iSCSI (OCFS2) 4:24<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Other tests including recursive chowns or chmods, and ls report similar results.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Most telling is perhaps strace output. There I can see system calls on individual Maildir files. Times between each call/operation take far longer on OCFS2 and there is no hint of externally derived waits. Nor are there any indicators of load issues from competing processes; nothing else (significant) is going on and du has full reign of the OS resources.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Output from strace with –tt –T using du –hs against the Maildir on my EXT3 iSCSI LUN (/dev/emcpowerb1)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>18:03:17.572879 lstat("1313705228.000737.mbox:2,S", {st_mode=S_IFREG|0644, st_size=715, ...}) = 0 <0.000018><o:p></o:p></p><p class=MsoNormal>18:03:17.572944 lstat("1313705228.008426.mbox:2,S", {st_mode=S_IFREG|0644, st_size=2779, ...}) = 0 <0.000024><o:p></o:p></p><p class=MsoNormal>18:03:17.573016 lstat("1313705228.006345.mbox:2,S", {st_mode=S_IFREG|0644, st_size=2703, ...}) = 0 <0.000020><o:p></o:p></p><p class=MsoNormal>18:03:17.573083 lstat("1313705228.001305.mbox:2,S", {st_mode=S_IFREG|0644, st_size=1831, ...}) = 0 <0.000017><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Output from the same trace against the OCFS2 store<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>18:06:52.876713 lstat("1313707554.003441.mbox:2,S", {st_mode=S_IFREG|0644, st_size=2322, ...}) = 0 <0.040896><o:p></o:p></p><p class=MsoNormal>18:06:52.917723 lstat("1313707554.003442.mbox:2,S", {st_mode=S_IFREG|0644, st_size=2316, ...}) = 0 <0.040663><o:p></o:p></p><p class=MsoNormal>18:06:52.958473 lstat("1313707554.003443.mbox:2,S", {st_mode=S_IFREG|0644, st_size=2899, ...}) = 0 <0.000938><o:p></o:p></p><p class=MsoNormal>18:06:52.959471 lstat("1313707554.003444.mbox:2,S", {st_mode=S_IFREG|0644, st_size=2522, ...}) = 0 <0.001106><o:p></o:p></p><p class=MsoNormal>18:06:52.960641 lstat("1313707554.003445.mbox:2,S", {st_mode=S_IFREG|0644, st_size=3451, ...}) = 0 <0.039904><o:p></o:p></p><p class=MsoNormal>18:06:53.000644 lstat("1313707554.003446.mbox:2,S", {st_mode=S_IFREG|0644, st_size=3150, ...}) = 0 <0.041060><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Is this normal behavior for a current kernel and the most recent 1.4.7 code? Does someone suspect I’ve blundered somewhere along the way? I’ve seen many posts to this list related to a mail cluster setup like mine. Is anyone on the list running a production mail cluster with OCFS2? I apologize for the length of this email. Thanks.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>-Nick Geron<o:p></o:p></p></div></body></html>