<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>David,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>Again, I recommend local storage for the logs. I know I’ve
said it many times, but I’ve run into these SAME issues, and local was
the way to go. For log parsing across nodes, or for tailing them all at the
same time, we utilize Capistrano (a ruby-based project), which easily allows me
to tail log files across multiple servers (or perform any operations for that
matter) all to one window.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>Really though, try local storage for your logs.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>In regards to the MPM, prefork has benefits, and also severe
disadvantages. I’m not certain that the worker MPM buffers the logs, and
of course, you’ll want multiple worker processes (and threads), but it
certainly is an improvement over prefork. However, if you have non-threadsafe
modules (just curious, which ones?), as you stated, you may have trouble. This
is one of the reasons that where we can, we switched to lighttpd.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>In regards to non-threadsafe modules, if you are specifically
speaking of handlers (php, ruby, perl, etc), you could simply migrate those to
use fastcgi (I recommend mod_fcgid). It’s a considerable improvement, and
works great with non-threadsafe modules/libraries.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>Your best solution however, is still to use local log files. Again,
it’s easy to test, and if the lags go away, you’ll know.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>Michael<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span class=apple-style-span><b><span style='color:black'>David
Johle</span></b></span><span class=apple-converted-space><span
style='color:black'> </span></span><span class=apple-style-span><span
style='color:black'><a
href="mailto:ocfs2-users%40oss.oracle.com?Subject=Re:%20%5BOcfs2-users%5D%20ocfs2%20hangs%20during%20webserver%20usage&In-Reply-To=%3C200901282147.n0SLlGB9025423%40mix.industrialinfo.com%3E"
title="[Ocfs2-users] ocfs2 hangs during webserver usage">djohle at
industrialinfo.com<span class=apple-converted-space><span style='color:blue;
text-decoration:none'> </span></span></a></span></span><span
style='color:black'><br>
<span class=apple-style-span><i>Wed Jan 28 13:40:53 PST 2009</i></span></span> <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>Okay, that was easy to test, and fail -- still plenty of lag <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>situations. As expected, the results agree with another test I
did <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>which was to shut down one of the two nodes to the point of <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>unmounting the shared filesystem.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>I believe the problem here is that actual chid processes of Apache
<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>locally sharing the logfile is just as bad as sharing across <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>nodes. Maybe perhaps that is the root cause of the problem, and
the <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>fact that additional nodes are affected is just consequential.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>Michael Moody suggested not using prefork MPM, presumably for this
<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>reason. I wonder, does the worker MPM "consolidate" all
the writes <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>from the threads within a worker into a single buffered write <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>stream? If so, I can see how having 32 threads per worker would
not <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>reduce the amount of writes to the logfile, but could reduce the <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>amount of contending I/O requests by a factor of 32. Is this the
case?<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>Way back when we first transitioned to Apache 2.x we looked into <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>using a better MPM, but had some necessary modules installed that <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>were not thread-safe, and thus would only work with prefork. I am
<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>pretty sure this is still a constraint for us at this point.<o:p></o:p></span></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>