[Ocfs2-users] ocfs2 hangs during webserver usage

[David Johle]

At 06:32 PM 1/27/2009, jmoseley at corp.xanadoo.com wrote:
>As others have indicated, I don't think that's going to work very well.
>You've got two different nodes trying to write to the same file constantly.
>I would keep each server's log on a locally mounted file system, or simply
>keep the logs on the OCFS2 filesystem, but have each node write to
>different log files.
>
>Yeah, that makes parsing access_logs slightly more of a problem for
>producing hit reports, etc, but I think you'll notice performance improve.


Yes, parsing logs is just one good reason for having unified log 
files -- one of the motivations for using OCFS2 even.  If our 
statistics program can handle multiple files, then at least having 
them in a shared directory would be useful.

Another major area this would affect is web site issue 
troubleshooting which outputs to log files (not the access logs but 
others).  I can only imagine the complexity of having to deal with 
locating specific logging information for a site user who is having 
trouble by going to 5 different nodes to dig through locally stored 
log files.  Or worse yet, trying to correlate actions of multiple 
users who are each hitting different nodes!

On that note, these other logs are written to by our aplications 
running under Tomcat.  I really am not seeing any similar lags for 
those processes, only from apache.  The only big difference I can see 
between them is the I/O pattern -- apache is usually 1 line per 
request as they are serviced, java web apps are more bursts of 
numerous lines, but not every request.  There is still a non-trivial 
amount of logging happening for these java apps though, so I am 
surprised.  In fact, Tomcat itself is configured to log each request 
with the processing time (used to produce user response time 
statistics), but those shared logs don't seem to be a point of 
contention like the apache access logs.

For informational purposes, here are some line counts for logs on our 
main web site yesterday:
   1577860 access log
      1361 error log
   4887437 web app log
    340164 processing time log
   6806822 total

So only about 20% of the requests are handled by Tomcat.  The web app 
log actually writes 3x as many lines, but overall it's less data 
(373M vs. 428M) and fewer actual write operations.  This could 
explain why it is not/less prone to these write delays.


I will try changing the logging to use separate files for a day or so 
to at least see what effect it has with regards to this issue...stay tuned.