[Ocfs2-users] Too much journaling or not ?

[greg at greg.net.au]

Hi All,

 

I have stumbled across (via Google) a post on this mailing list in relation
to performance issues with OCFS2.

 

A little overview of our setup:

 

3 x Dell Poweredge R200 servers, w/8GB RAM, Dual Gig NIC's running VSphere
4.1 (ESXi w/Enterprise License)

1 x Dell Poweredge MD3000i ISCSI SAN w/15x 1tb SATA drives in RAID6

 

Each ESXi server runs 2 Gentoo Virtual machines running kernel 2.6.34 with
ocfs2-tools - workload consists of lighttpd, Apache & Squid, with caching
from the SAN to the local vm disks & RAM.

 

Our problem lies within performance of the OCFS2 volume (which is ~10TB)
over the disks.

 

The iowait is constantly high (30-40% per server), and even though there are
plenty of inodes and physical disk free, we cannot explain the problem.

 

dnetwww2 ~ # df -h

Filesystem            Size  Used Avail Use% Mounted on

rootfs                 18G  6.5G   11G  39% /

/dev/sda3              18G  6.5G   11G  39% /

rc-svcdir             1.0M   72K  952K   8% /lib64/rc/init.d

udev                   10M  184K  9.9M   2% /dev

shm                   2.0G     0  2.0G   0% /dev/shm

/dev/sdb1             247G   29G  206G  13% /cache

/dev/ram0             190M   13M  177M   7% /home/core

/dev/ram1             190M   60M  130M  32% /home/moddb

/dev/ram2             190M   20M  170M  11% /home/desura

/dev/mapper/360024e8000758ab1000007624c1525dc1

                      9.8T  2.1T  7.7T  22% /home/shared

 

dnetwww2 ~ # cat /etc/fstab

--snip--

/dev/mapper/360024e8000758ab1000007624c1525dc1  /home/shared    ocfs2
commit=15,heartbeat=local,data=writeback,noatime,user_xattr     1 2

 

 

dnetwww2 ~ # multipath -ll

360024e8000758ab1000007624c1525dcdm-0 ,

[size=9.7T][features=1 queue_if_no_path][hwhandler=0]

\_ round-robin 0 [prio=6][active]

\_ #:#:#:# sdc 8:32  [active][ready]

\_ #:#:#:# sde 8:64  [active][ready]

\_ round-robin 0 [prio=0][enabled]

\_ #:#:#:# sdd 8:48  [active][ghost]

\_ #:#:#:# sdf 8:80  [active][ghost]

 

Now if I mount an ext4 formatted lun/partition from the MD3000i (mapped via
iscsi&multipath-tools) I can read/write to it at 125MB/s with no issues. The
ocfs2 mounted volume struggles to sustain 25-30MB/s read/write. :-(

 

We have spent countless hours working (troubleshooting/debugging) this now
without result. We've even replaced both controllers, switches, network
cards and so on in an attempt to rule out a specific hardware cause, but it
seems to be ocfs2 related.

 

I've noted there are a number of new ocfs2 patches in Linux 2.6.35 & the yet
to be released 2.6.36 - would like to know if any of these resolve this
issue before we are forced to ditch ocfs2 and go back to NFS.

 

Cheers,

 

Greg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20100822/2040dbdb/attachment.html