<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:1889608463;
mso-list-type:hybrid;
mso-list-template-ids:-2100236488 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>I want to throw in my experience here:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>You WILL have issues if you have reasonably high load, with
multiple nodes performing writes to log files. It will hang all processes
waiting on the write lock. In the case of apache, especially when using prefork,
this is a VERY BAD idea. It is highly highly recommended to move your log files
outside of the cluster FS. We run ocfs2 in production, hundreds of gigs of
files, with 5 nodes per cluster, 4 clusters. Sustained throughput of 200mbit/s
being served by these nodes. All this is on a fiber channel SAN with very fast
arrays. Even so, when we stumble across the occasional log file contention, and
remove it (say, a php file writing to a log file, or apache, etc), load drops
considerably. When we had apache writing log files to the cluster FS with ¼ the
current load, apache choked and died, it couldn’t handle the contention.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>It will continue to escalate in a non-linear fashion. My
recommendations:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Move all logs, and any files which are written to
concurrently off to local system (yes, it makes administration harder, but
systems happier, and was a MUST for our environment, and we’ve managed to
push OCFS2 scaling a lot farther by doing so)<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Switch away from prefork if you are using that<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>3.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Make absolutely sure you’re mounted with noatime<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>4.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>If possible, switch away from apache and utilize lighttpd
(we recently did this, and were able to gain another 20-30% out of each box)<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Relevant lighttpd settings:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>server.max-fds = 2048<o:p></o:p></p>
<p class=MsoNormal>server.max-write-idle = 120<o:p></o:p></p>
<p class=MsoNormal>server.max-worker = 8<o:p></o:p></p>
<p class=MsoNormal>server.network-backend = "linux-sendfile"<o:p></o:p></p>
<p class=MsoNormal>server.stat-cache-engine = "simple"<o:p></o:p></p>
<p class=MsoNormal>server.max-keep-alive-requests = 10<o:p></o:p></p>
<p class=MsoNormal>server.max-keep-alive-idle = 8<o:p></o:p></p>
<p class=MsoNormal>server.event-handler = "linux-sysepoll"<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Again, we’ve been using OCFS2 in production for 2
years, throughout many problems, and I highly recommend moving the log files to
local storage, OR piping them to a network cable syslog. (Also a possible
solution).<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Michael<o:p></o:p></p>
</div>
</body>
</html>