[Ocfs2-users] OCFS2 performance issues

[Andy]

I have a 12 node cluster using ocfs2.  Not all the nodes mount the same 
filesystems, so for this issue I am just focusing on the 8 nodes that 
mount the filesystem that has performance issues.

The 8 nodes have 10 ocfs2 volumes mounted on them, and the volume that 
seems to be causing the most problems is 2TB (data), which is currently 
90% full.

At times, data performs very poorly.  Processes using the volume are 
slow, df will hang for a long time accessing it, and nfs mounts to it 
are slow to return or even give an i/o error.

This is becoming a big problem and I would like some help to resolve the 
performance issue.  I have included the output of both stat and fs_state 
below.  The kernel version on all nodes 3.2.29, and the ocfs2-utilities 
version in 1.6.4.

I tried 1.8.2, but that version seemed to have issues reading 
cluster.conf properly. It would give errors about the local node not 
being in the configuration file and it would only list one node in 
ocfs2console.  I looked to me as if it was only looking at the first 
node in the configuration file, 1.6.4 reads the same configuration file 
fine.

Thanks,

Andy


Here is the stat for the data volume
         Revision: 0.90
         Mount Count: 0   Max Mount Count: 20
         State: 0   Errors: 0
         Check Interval: 0   Last Check: Tue Apr 10 11:00:52 2012
         Creator OS: 0
         Feature Compat: 3 backup-super strict-journal-super
         Feature Incompat: 592 sparse inline-data xattr
         Tunefs Incomplete: 0
         Feature RO compat: 1 unwritten
         Root Blknum: 513   System Dir Blknum: 514
         First Cluster Group Blknum: 256
         Block Size Bits: 12   Cluster Size Bits: 20
         Max Node Slots: 16
         Extended Attributes Inline Size: 256
         Label:
         UUID: B4B05BE929E54DFEB5FDC035E3F6E931
         Hash: 1144689215 (0x443a923f)
         DX Seed[0]: 0x00000000
         DX Seed[1]: 0x00000000
         DX Seed[2]: 0x00000000
         Cluster stack: classic o2cb
         Inode: 2   Mode: 00   Generation: 2584189387 (0x9a0799cb)
         FS Generation: 2584189387 (0x9a0799cb)
         CRC32: 00000000   ECC: 0000
         Type: Unknown   Attr: 0x0   Flags: Valid System Superblock
         Dynamic Features: (0x0)
         User: 0 (root)   Group: 0 (root)   Size: 0
         Links: 0   Clusters: 1280000
         ctime: 0x4f845934 -- Tue Apr 10 11:00:52 2012
         atime: 0x0 -- Wed Dec 31 18:00:00 1969
         mtime: 0x4f845934 -- Tue Apr 10 11:00:52 2012
         dtime: 0x0 -- Wed Dec 31 18:00:00 1969
         ctime_nsec: 0x00000000 -- 0
         atime_nsec: 0x00000000 -- 0
         mtime_nsec: 0x00000000 -- 0
     Refcount Block: 0
         Last Extblk: 0   Orphan Slot: 0
         Sub Alloc Slot: Global   Sub Alloc Bit: 65535

fs_state:

Device => Id: 254,2  Uuid: 45B8B88228BD4DFBBE91BFC85C748FB6  Gen: 
0x364EAC5A  Label:
     Volume => State: 2  Flags: 0x0
      Sizes => Block: 4096  Cluster: 4096
   Features => Compat: 0x3  Incompat: 0x250  ROcompat: 0x1
      Mount => Opts: 0x105  AtimeQuanta: 60
    Cluster => Stack: o2cb  Name: 45B8B88228BD4DFBBE91BFC85C748FB6 
Version: 1.0
   DownCnvt => Pid: 9804  Count: 0  WakeSeq: 42573  WorkSeq: 42573
   Recovery => Pid: -1  Nodes: None
     Commit => Pid: 9809  Interval: 0  Needs: 1
    Journal => State: 1  TxnId: 57  NumTxns: 0
      Stats => GlobalAllocs: 1  LocalAllocs: 393  SubAllocs: 108 
LAWinMoves: 1  SAExtends: 0
LocalAlloc => State: 1  Descriptor: 0  Size: 27136 bits  Default: 27136 bits
      Steal => InodeSlot: -1  StolenInodes: 0, MetaSlot: -1  StolenMeta: 0
OrphanScan => Local: 126  Global: 1036  Last Scan: 851 seconds ago
      Slots => Num     RecoGen
                 0          23
                 1          18
             *   2           5
                 3          16
                 4           3
                 5          12
                 6           6
                 7           1
                 8           3
                 9           0
                10           0
                11           0
                12           0
                13           0
                14           0
                15           0