[Ocfs2-users] OCFS2 filesystem hangs with "dirty" locks on internal files

[Charlie Smurthwaite]

Hi,

I have been trying for some time to get to the bottom of a problem that 
is causing an OCFS2 filesystem to hang (increasing numbers of file 
operations hang until the filesystem becomes unusable) seemingly at 
random, approximately once per day.

I have got as far as dumping the busy locks and dlm lock state on all 
nodes when this occurs.

In summary, it appears that all nodes are waiting on locks for shared 
internal data files, specifically:

debugfs: encode //global_bitmap
M000000000000000000000baa25b2b2
debugfs: encode //aquota.user
M000000000000000000000caa25b2b2
debugfs: encode //aquota.group
M000000000000000000000daa25b2b2

The DLM status of these 3 files are pasted below. It seems that all 
nodes are waiting for access to the global bitmap (the bottom entry in 
the DLM output below) but nobody is able to obtain this lock. Is there 
an obvious cause of this situation?

I'd be happy to provide any further information that may help. Sorry if 
I'm not understanding the situation very well yet.

Thanks!
Charlie



Lockres: M000000000000000000000caa25b2b2 Owner: 3 State: 0x8 Dirty
Last Used: 0 ASTs Reserved: 0 Inflight: 0 Migration Pending: No
Refs: 12 Locks: 9 On Lists: Dirty
Reference Map: 0 1 2 4 5 6 7 8
Lock-Queue Node Level Conv Cookie Refs AST BAST Pending-Action
Granted 0 NL -1 0:5 2 No No None
Converting 1 NL EX 1:10 2 No No None
Converting 5 NL EX 5:8 2 No No None
Converting 6 NL EX 6:4 2 No No None
Converting 2 NL EX 2:7 2 No No None
Converting 7 NL EX 7:9 2 No No None
Converting 8 NL EX 8:11 2 No No None
Converting 4 NL EX 4:6 2 No No None
Converting 3 NL EX 3:27 2 No No None
--
Lockres: M000000000000000000000daa25b2b2 Owner: 3 State: 0x8 Dirty
Last Used: 0 ASTs Reserved: 0 Inflight: 0 Migration Pending: No
Refs: 12 Locks: 9 On Lists: Dirty
Reference Map: 0 1 2 4 5 6 7 8
Lock-Queue Node Level Conv Cookie Refs AST BAST Pending-Action
Granted 0 NL -1 0:8 2 No No None
Converting 1 NL EX 1:13 2 No No None
Converting 5 NL EX 5:11 2 No No None
Converting 6 NL EX 6:7 2 No No None
Converting 2 NL EX 2:10 2 No No None
Converting 7 NL EX 7:12 2 No No None
Converting 8 NL EX 8:14 2 No No None
Converting 4 NL EX 4:9 2 No No None
Converting 3 NL EX 3:30 2 No No None
--

Lockres: M000000000000000000000baa25b2b2 Owner: 3 State: 0x8 Dirty
Last Used: 0 ASTs Reserved: 0 Inflight: 0 Migration Pending: No
Refs: 12 Locks: 9 On Lists: Dirty
Reference Map: 0 1 2 4 5 6 7 8
Lock-Queue Node Level Conv Cookie Refs AST BAST Pending-Action
Converting 4 NL EX 4:39 2 No No None
Converting 8 NL PR 8:39 2 No No None
Converting 0 NL PR 0:30 2 No No None
Converting 6 NL PR 6:39 2 No No None
Converting 1 NL PR 1:39 2 No No None
Converting 3 NL EX 3:33 2 No No None
Converting 7 NL EX 7:39 2 No No None
Converting 2 NL EX 2:39 2 No No None
Converting 5 NL PR 5:39 2 No No None