[Ocfs2-users] High load average on Apache Cluster with drbd + ocfs2

[Brad Plant]

On Wed, 3 Mar 2010 03:26:04 -0800
Joel Becker <Joel.Becker at oracle.com> wrote:

> On Wed, Mar 03, 2010 at 09:47:08PM +1100, Brad Plant wrote:
> > Turns out I'd hit a free space fragmentation problem. While df reported I had heaps of free space (>50% from memory!), I couldn't write (echo >>) to the log files on the problem web server. Note that you'll find you can still create small files and append to small files, but not the larger apache log files.
> > 
> > The fact that it happens late at night was very confusing, but eventually made sense. As the day goes on, the log files get bigger and bigger pieces of contiguous free space are required to extend the file. Eventually, a contiguous piece of free space cannot be found and your writes will start to fail.
> 
> 	This is my assumption, which is why I asked for the debugfs
> output.  This will give us layout information.
> 	There are actually two fragmentation issues here.  First, free
> space is fragmented.  This prevents us from allocating metadata blocks.
> These metadata blocks are used in the housekeeping of files with many
> data extents.  This fits your description of lots of free space, yet
> files are unable to grow.
> 	The second issue is fragmentation of the actual file.  Log files
> are written in small hunks.  As each node takes turns extending the log
> file, the hunks come from disparate parts of the disk.  This means the
> log files have lots of small extents instead of a few large ones.  This
> is why we recommend separate log files for each node.  Separate log
> files would grow in a much more contiguous fashion, allowing fewer,
> larger extents.  With fewer extents, they would need fewer metadata
> blocks to grow.

I forgot to mention that I was actually using a separate log file per web server.

Cheers,

Brad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://oss.oracle.com/pipermail/ocfs2-users/attachments/20100303/350e1683/attachment-0001.bin