[Ocfs2-users] fsck.ocfs2 can't fix an orphaned inode

[Carl J. Benson]

Hello!

I searched through the mailing list back to 07/2008, and didn't see
this question answered before.

I have 7 systems that use an ocfs2 filesystem. After many months of
solid reliable use, they all crashed yesterday.

6 systems run openSUSE 11.1, kernel 2.627.29-0.1-default, with these
RPMs:

ocfs2-tools-1.4.1-6.9
ocfs2console -1.4.1-6.9

1 system has for a week been running openSUSE 11.2, kernel
2.6.31.12-0.1-default, with these RPMs:

ocfs2console-1.4.1-25.6.x86_64
ocfs2-tools-1.4.1-25.6.x86_64
ocfs2-tools-o2cb-1.4.1-25.6.x86_64

I still haven't figured out where the corruption started, but the
problem at the moment is this: After repeated runs of fsck.ocfs2
(with the filesystem unmounted, of course!), it's fallen into a
pattern. Here is the output of fsck.ocfs2:

/root # fsck.ocfs2 /dev/sdc1
Checking OCFS2 filesystem in /dev/sdc1:
  label:              iscsi_ocfs2_cluster
  uuid:               23 48 29 28 4d 71 44 e6 b4 1d 88 75 c9 69 46 d3
  number of blocks:   268438109
  bytes per block:    4096
  number of clusters: 268438109
  bytes per cluster:  4096
  max slots:          20

pass4: Invalid block number while truncating orphan inode 104935559
fsck.ocfs2: Invalid block number while trying to replay the orphan directory
fsck encountered errors while recovering slot information, check forced.
/dev/sdc1 was run with -f, check forced.
Pass 0a: Checking cluster allocation chains
Pass 0b: Checking inode allocation chains
Pass 0c: Checking extent block allocation chains
Pass 1: Checking inodes and blocks.
Pass 2: Checking directory entries.
Pass 3: Checking directory connectivity.
Pass 4a: checking for orphaned inodes
[INODE_ORPHANED] Inode 104935559 was found in the orphan directory.
Delete its contents and unlink it? <y> y
pass4: Invalid block number while truncating orphan inode 104935559

[INODE_ORPHANED] Inode 106959312 was found in the orphan directory.
Delete its contents and unlink it? <y> y
pass4: Invalid block number while truncating orphan inode 106959312

Pass 4b: Checking inodes link counts.

All passes succeeded.

At this point I reboot the server (named "merlot1"), run fsck.ocfs2,
and get exactly the same result.

What can I do now? I looked at the man page for debugfs.ocfs2,
but it doesn't look like that's going to help me.

Any suggestions, please?

--
Carl Benson                  |  cbenson at fhcrc.org
Linux System Administrator   |  Telephone: (206) 667-4862
Fred Hutchinson Cancer Research Center