[Ocfs2-users] High load average on Apache Cluster with drbd + ocfs2

[Joel Becker]

On Wed, Mar 03, 2010 at 11:04:48AM +0100, Andreas Kossmann wrote:
> I have an enviroment with 2 Debian 5.0 servers. 
> Kernel is 2.6.26-2-amd64. I have installed drbd-8.0.14 and ocfs2-tools 1.4.1.
> It is an Active/Active WebCluster with Apache.
> The 2 servers write to the same log files.

	It's always better if they can write to separate log files, but
I'm going to proceed as if that won't change, because we should handle
both.

> First I thought the problem may be drbd, but I have read many problemes with ocfs2 and apache load average.

	Doesn't mean it's not drbd ;-)  Let's see what I can find out.

> The curios thing is that the load is often very high at times where request are very small ( eg. 11:00 PM )

	The problem may or may not be related to apache's volume of
requests.

> I've disconnected the second webserver from the network and checked the filesystem. A few bitmap errors occured and i repaired them. Then I changed the drbd config so, that only webserver 1 is primary and the webserver 2 is secondary. So webserver 2 cannot write to the device. 

	Wait, you checked the filesystem while webserver 1 was still
running?  The errors you found are dirty state from webserver 1.  They
would not occur if webserver 1 was cleanly unmounted.

> After I connect the webserver 2 to the network again and the sync from the primary starts. The load on webserver 1 is going > 100.

	What does it look like after the sync is done?  Does performance
get better?

> Is there any solution for the ocfs2 load problem with apache?

	I don't know if this is drbd keeping the volumes in sync or
an ocfs2 issue.  Are you sure it is the log files that cause the
problem?  Can you get me debugfs.ocfs2 output for those files?  For
example, if your filesystem is on /dev/drbd1, is mounted at /web, and
your log file is /web/logs/error_log, then we want the output of
"debugfs.ocfs2 -R 'stat /logs/error_log' /dev/drbd1".  Please do this
for each log file and send it along to me.

Joel

-- 

"Can any of you seriously say the Bill of Rights could get through
 Congress today?  It wouldn't even get out of committee."
	- F. Lee Bailey

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127