[Ocfs2-users] OCFS2 Error: Group Descriptor Mismatch

[Joel Becker]

On Tue, Mar 17, 2009 at 11:37:29AM +0000, Jari Takkala wrote:
> We recently ran into an issue with another one of our OCFS2 clusters where OCFS2 detected on-disk corruption. The filesystem in question has a capacity of 641GB, and I had attempted to remove about 500GB of files. The number of files that I was removing would have been about 20,000.

	First and foremost, can you file a bugzilla bug?  This is great
detail, and it should be captured there.  More comments below.

> I started the 'rm -rf' on host2. Shortly after that the filesystem was automatically mounted read-only and the following errors logged:
> 
> Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_commit_truncate:6490 ERROR: status = -5
> Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_delete_inode:974 ERROR: status = -5
> Mar 15 14:31:34 host2 kernel: (2008,1):__ocfs2_flush_truncate_log:5111 ERROR: status = -5
> Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_free_clusters:1842 ERROR: status = -5
> Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_free_suballoc_bits:1755 ERROR: status = -5
> Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_replay_truncate_records:5039 ERROR: status = -5
> Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_truncate_for_delete:562 ERROR: status = -5
> Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_wipe_inode:733 ERROR: status = -5
> Mar 15 14:31:34 host2 kernel: File system is now read-only due to the potential of on-disk corruption. Please run fsck.ocfs2 once the file system is unmounted.
> Mar 15 14:31:34 host2 kernel: OCFS2: ERROR (device xvdb1): ocfs2_check_group_descriptor: Group descriptor # 12257280 has bit count 32256 but claims that 32639 are free
> Mar 15 14:31:36 host2 kernel: (1796,1):__ocfs2_flush_truncate_log:5111 ERROR: status = -5
> Mar 15 14:31:36 host2 kernel: (1796,1):ocfs2_free_clusters:1842 ERROR: status = -5
> Mar 15 14:31:36 host2 kernel: (1796,1):ocfs2_free_suballoc_bits:1755 ERROR: status = -5
> Mar 15 14:31:36 host2 kernel: (1796,1):ocfs2_replay_truncate_records:5039 ERROR: status = -5
> Mar 15 14:31:36 host2 kernel: (1796,1):ocfs2_truncate_log_worker:5150 ERROR: status = -5
> Mar 15 14:31:36 host2 kernel: OCFS2: ERROR (device xvdb1): ocfs2_check_group_descriptor: Group descriptor # 12257280 has bit count 32256 but claims that 32639 are free

	All your errors are -5, or EIO.  They appear to all be coming
from the group descriptor error, but your log is very weird - it's
almost in the reverse order the functions are called.

> I unmounted the filesystem from all three nodes and ran 'ocfs2.fsck -f' on it. I had to run ocfs2.fsck twice before it reported the clean. A snippet of the fsck output:
> 
> [GROUP_FREE_BITS] Group descriptor at block 12257280 claims to have 32639 free bits which is more than 32238 bits indicated by the bitmap. Drop its free bit count down to the total? <y> y
> [CHAIN_BITS] Chain 137 in allocator inode 11 has 249271 bits marked free out of 677376 total bits but the block groups in the chain have 248870 free out of 677376 total.  Fix this by updating the chain record? <y> y
> [CHAIN_GROUP_BITS] Allocator inode 11 has 109425928 bits marked used out of 167772803 total bits but the chains have 109426329 used out of 167772803 total.  Fix this by updating the inode counts <y> y

	These errors are self-consistent.  That is, the higher levels of
the chain agree with the lower levels.  Of course, they all agree on the
bit count at the lowest level that is wrong.  How it came to be wrong is
the $64k question.
	Can you attach the message logs from all nodes to the bugzilla
bug?  Maybe one of the other nodes did something.

> I have a snapshot of the LUN with the original data, and I can run some tests on it if necessary and try to reproduce the problem. Note that we had another OCFS2 issue recently (http://oss.oracle.com/pipermail/ocfs2-users/2009-February/003369.html), and we're still investigating that. However, that problem was on a database cluster, which is on a different storage array, and it does not seem to be the same problem.

	I'm guessing this was a global bitmap cluster group based on the
function call chain, but I'd like to verify.  Is it possible to get an
o2image of the volume for us to look at?  o2image should create an image
without data so that its safe to send to us.

Joel

-- 

"Reality is merely an illusion, albeit a very persistent one."
        - Albert Einstien

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127