[Ocfs2-users] Filesystem corruption and OCFS2 errors

Wed May 20 11:30:40 PDT 2009

Christian van Barneveld wrote:
> Our OCFS2 cluster has been stable for approx 8 months, but since this week it went wrong. First we had high load problems. We had this problem because a couple of directories got filled with files, 1 directory over 1,5 milion files (symlinks) and NFS (mounts are exported with NFS) caused high load because of that. Dir listing wasn't posible anymore.
> I cleaned up the directories and after that the load became normal again and everything seemed to be fine.
>
> But within a day our customer reported continuous disappearance of files. Those files where not from directories that I have cleaned, but random at the filesystem. There are also files that are not accesible anymore and a readonly FSCK showed some inode errors.
> We have 3 OCFS2 filesystems mounted and 2 of them had problems. Last night I brought down the cluster, unmount the filesystems and did a filesystem check. The 2 affected filesystems reported several errors like:
> [DIRENT_INODE_FREE] Directory entry 'f5377cd11ee628fe7c76c7f5b47f3bee.jpg' refers to inode number 811823124 which isn't allocated, clear the entry? <y> y
> [INODE_ORPHANED] Inode 800661759 was found in the orphan directory. Delete its contents and unlink it? <y> y

When one issues a rm, we first remove the directory entry and add a 
corresponding
entry in the orphan dir. Then we delete... free all the extents, inode 
bit, etc.
In your case, the dir entry is there but the inode bit is free. Also, 
that inode
is still in the orphan dir. Do you by chance save the fsck output? 
Wanted to see
if all the inodes were co-located... meaning the error stemmed from a 
corrupt inode
alloc bitmap.

> I fixed the 2 filesystems which had problems and decided to check the (thirth) filesystem which had  no problems and after that something went terribly wrong.
> First error was like this:
> [SUPERBLOCK_CLUSTERS] Superblock has clusters set to 40959872 instead of 999936 recorded in global_bitmap, it may be caused by an unsuccessful resize. Trust global_bitmap? <y>
> And I think I have given the wrong answer. After that a lot of Inode errors and when it finished there was no data anymore! 

No, you did not give the wrong answer. More specifically, your answer 
did not cause
the problem. That yes only set the size in the superblock the same as 
what was in the
global bitmap. That's harmless. The qs is why that value in the global 
bitmap was so
wrong. And this is one value we don't touch.... other than during 
resize. And we don't
allow shrinking. So size in gloabl bitmap should never be smaller than 
the one in the
superblock.

> Also after a remount the filesystem is not 2.5 TB, but 500 GB. LVM is used to create a 2,5 TB filesystem of one 2 TB LUN and a 500 GB LUN:
>   VG Size               2.44 TB
> But fdisk says:
> Disk /dev/mapper/vg04-FS1: 485.3 GB, 485322915840 bytes
>
> OCFS2:
>   number of blocks:   118702080
>   bytes per block:    4096
>   number of clusters: 7418880
>   bytes per cluster:  65536

If fdisk is saying the device is 485G, then that's what the other tools 
will see.
And this appears to be the root cause of your problem. LVM. There is a 
reason why
we don't support LVM.

> After that I tried:
>  tunefs.ocfs2 -S /dev/vg04/FS1 tunefs.ocfs2 1.4.1 tunefs.ocfs2: Cannot shrink volume size from 118702080 blocks to 118487040 blocks tunefs.ocfs2: Nothing to do. Exiting.
> But no results
>
> Is there anything I can do to fix this? I have tried a lot of things, but without results.

Best solution is to salvage your data using debugfs.ocfs2. It has 
commands like
dump/rdump that read the files directly off the disk.

> I also tried a new kernel (2.6.29.3), but after booting and mounting it crashed (dm-17 is NOT the corrupted 3rth filesystem, but the second which had no problems anymore):
>
> May 15 23:47:31 fileserver-1 kernel: OCFS2: ERROR (device dm-17): ocfs2_validate_inode_block: Invalid dinode #664384: signature = u^P£è\¼z
> May 15 23:47:31 fileserver-1 kernel:
> May 15 23:47:31 fileserver-1 kernel: File system is now read-only due to the potential of on-disk corruption. Please run fsck.ocfs2 once the file system is unmounted.
> May 15 23:47:31 fileserver-1 kernel: (14610,1):ocfs2_read_locked_inode:466 ERROR: status = -22
> May 15 23:47:31 fileserver-1 kernel: dm-17: rw=0, want=6635799728, limit=419422208

What this is saying is that the disk size has shrunk. It is trying to 
read 6G into
the volume but the block layer is saying that the device is 500M only. 
You have to
look into your block device setup.

> May 15 23:47:31 fileserver-1 kernel: (14613,0):ocfs2_read_locked_inode:466 ERROR: status = -5
> May 15 23:47:31 fileserver-1 kernel: OCFS2: ERROR (device dm-17): ocfs2_validate_inode_block: Invalid dinode #659658: signature = ^Bu^SÔ¤\237\235
> May 15 23:47:31 fileserver-1 kernel:
> May 15 23:47:31 fileserver-1 kernel: (14606,1):ocfs2_read_locked_inode:466 ERROR: status = -22
> May 15 23:47:31 fileserver-1 kernel: attempt to access beyond end of device
> May 15 23:47:32 fileserver-1 kernel: dm-17: rw=0, want=6635799728, limit=419422208
> May 15 23:47:32 fileserver-1 kernel: (14613,0):ocfs2_read_locked_inode:466 ERROR: status = -5
> May 15 23:47:32 fileserver-1 kernel: attempt to access beyond end of device
> May 15 23:47:32 fileserver-1 kernel: dm-17: rw=0, want=6635799728, limit=419422208
> May 15 23:47:32 fileserver-1 kernel: (14612,1):ocfs2_read_locked_inode:466 ERROR: status = -5
> May 15 23:47:32 fileserver-1 kernel: attempt to access beyond end of device
> May 15 23:47:32 fileserver-1 kernel: dm-17: rw=0, want=6635799728, limit=419422208
> May 15 23:47:32 fileserver-1 kernel: (14611,0):ocfs2_read_locked_inode:466 ERROR: status = -5
> May 15 23:47:33 fileserver-1 kernel: attempt to access beyond end of device
> May 15 23:47:33 fileserver-1 kernel: dm-17: rw=0, want=6635799728, limit=419422208
> May 15 23:47:33 fileserver-1 kernel: (14612,0):ocfs2_read_locked_inode:466 ERROR: status = -5
> May 15 23:47:33 fileserver-1 kernel: attempt to access beyond end of device
> May 15 23:47:33 fileserver-1 kernel: dm-17: rw=0, want=6635799728, limit=419422208
> May 15 23:47:34 fileserver-1 kernel: (14611,0):ocfs2_read_locked_inode:466 ERROR: status = -5
> May 15 23:47:34 fileserver-1 kernel: OCFS2: ERROR (device dm-17): ocfs2_validate_inode_block: Invalid dinode #664384: signature = u^P£è\¼z
> May 15 23:47:34 fileserver-1 kernel:
> May 15 23:47:34 fileserver-1 kernel: (14613,1):ocfs2_read_locked_inode:466 ERROR: status = -22
> May 15 23:47:34 fileserver-1 kernel: OCFS2: ERROR (device dm-17): ocfs2_validate_inode_block: Invalid dinode #659658: signature = ^Bu^SÔ¤\237\235
> May 15 23:47:34 fileserver-1 kernel:
> May 15 23:47:34 fileserver-1 kernel: (14610,1):ocfs2_read_locked_inode:466 ERROR: status = -22
> May 15 23:47:34 fileserver-1 kernel: OCFS2: ERROR (device dm-17): ocfs2_validate_inode_block: Invalid dinode #664384: signature = u^P£è\¼z
> May 15 23:47:34 fileserver-1 kernel:
> May 15 23:47:34 fileserver-1 kernel: (14612,1):ocfs2_read_locked_inode:466 ERROR: status = -22
> May 15 23:47:34 fileserver-1 kernel: attempt to access beyond end of device
> May 15 23:47:34 fileserver-1 kernel: dm-17: rw=0, want=6635799728, limit=419422208
> May 15 23:47:34 fileserver-1 kernel: (14611,0):ocfs2_read_locked_inode:466 ERROR: status = -5
> May 15 23:47:34 fileserver-1 kernel: attempt to access beyond end of device
> May 15 23:47:34 fileserver-1 kernel: dm-17: rw=0, want=6635799728, limit=419422208
> May 15 23:47:34 fileserver-1 kernel: (14612,1):ocfs2_read_locked_inode:466 ERROR: status = -5
> May 15 23:59:30 fileserver-1 kernel: (10887,0):ocfs2_orphan_del:1978 ERROR: status = -2
> May 15 23:59:30 fileserver-1 kernel: (10887,0):ocfs2_remove_inode:619 ERROR: status = -2
> May 15 23:59:30 fileserver-1 kernel: (10887,0):ocfs2_wipe_inode:753 ERROR: status = -2
> May 15 23:59:30 fileserver-1 kernel: (10887,0):ocfs2_delete_inode:990 ERROR: status = -2
> May 16 00:28:39 fileserver-1 kernel: ocfs2_dlm: Nodes in domain ("296B7CF537094A9BA5F193A426D92440"): 0
>
> May 16 00:40:19 fileserver-1 kernel: ------------[ cut here ]------------
> May 16 00:40:19 fileserver-1 kernel: kernel BUG at fs/ocfs2/inode.c:244!
> May 16 00:40:19 fileserver-1 kernel: invalid opcode: 0000 [#1] SMP
> May 16 00:40:19 fileserver-1 kernel: last sysfs file: /sys/fs/o2cb/interface_revision
> May 16 00:40:19 fileserver-1 kernel: Modules linked in: ocfs2 jbd2 xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs dm_round_robin scsi_dh_rdac dm_multipath dm_mod scsi_dh qla2xxx
> May 16 00:40:19 fileserver-1 kernel:
> May 16 00:40:19 fileserver-1 kernel: Pid: 14609, comm: nfsd Not tainted (2.6.29.3-amd-mods-qla2xxx-mpath-fw-cluster-hm64 #1) Sun Fire V40z
> May 16 00:40:19 fileserver-1 kernel: EIP: 0060:[<fa8c2580>] EFLAGS: 00010246 CPU: 0
> May 16 00:40:19 fileserver-1 kernel: EIP is at ocfs2_populate_inode+0x550/0x560 [ocfs2]
> May 16 00:40:19 fileserver-1 kernel: EAX: 00000000 EBX: f49ae000 ECX: 00000000 EDX: fa9002aa
> May 16 00:40:19 fileserver-1 kernel: ESI: e44eddfc EDI: f66f1000 EBP: f2821cb8 ESP: f2821c6c
> May 16 00:40:19 fileserver-1 kernel:  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> May 16 00:40:19 fileserver-1 kernel: Process nfsd (pid: 14609, ti=f2820000 task=f6660080 task.ti=f2820000)
> May 16 00:40:19 fileserver-1 kernel: Stack:
> May 16 00:40:19 fileserver-1 kernel:  00000001 00000000 e44eda80 00000000 00000000 e44eddfc 00000001 f2821cac
> May 16 00:40:19 fileserver-1 kernel:  f2821cf4 00000001 f2821cb8 00000000 00000001 f2821cac 00000000 fa8c07f0
> May 16 00:40:19 fileserver-1 kernel:  f66f1000 e44eddfc 00000001 f2821d04 fa8c2b7b 00000000 f2821ce0 f3d0b0c0
> May 16 00:40:19 fileserver-1 kernel: Call Trace:
> May 16 00:40:19 fileserver-1 kernel:  [<fa8c07f0>] ? ocfs2_validate_inode_block+0x0/0x280 [ocfs2]
> May 16 00:40:19 fileserver-1 kernel:  [<fa8c2b7b>] ? ocfs2_iget+0x5eb/0x930 [ocfs2]
> May 16 00:40:19 fileserver-1 kernel:  [<fa8b708a>] ? ocfs2_get_dentry+0x9a/0x1e0 [ocfs2]
> May 16 00:40:19 fileserver-1 kernel:  [<c04d80d2>] ? skb_copy_datagram_iovec+0x132/0x1d0
> May 16 00:40:19 fileserver-1 kernel:  [<fa8b7277>] ? ocfs2_fh_to_dentry+0x47/0x60 [ocfs2]
> May 16 00:40:19 fileserver-1 kernel:  [<c0251cc5>] ? exportfs_decode_fh+0x35/0x1f0
> May 16 00:40:19 fileserver-1 kernel:  [<c02c470f>] ? security_task_setgroups+0xf/0x20
> May 16 00:40:19 fileserver-1 kernel:  [<c0132de6>] ? set_groups+0x16/0x1f0
> May 16 00:40:19 fileserver-1 kernel:  [<c057794d>] ? cache_check+0x2d/0x3e0
> May 16 00:40:19 fileserver-1 kernel:  [<c013305a>] ? groups_alloc+0x3a/0xc0
> May 16 00:40:19 fileserver-1 kernel:  [<c025babc>] ? nfsd_setuser+0x17c/0x360
> May 16 00:40:19 fileserver-1 kernel:  [<c0254bca>] ? nfsd_setuser_and_check_port+0x5a/0x60
> May 16 00:40:19 fileserver-1 kernel:  [<c02599c4>] ? exp_find+0x54/0x80
> May 16 00:40:19 fileserver-1 kernel:  [<c0259a26>] ? rqst_exp_find+0x36/0xd0
> May 16 00:40:19 fileserver-1 kernel:  [<c0254fe4>] ? fh_verify+0x414/0x650
> May 16 00:40:19 fileserver-1 kernel:  [<c02556f0>] ? nfsd_acceptable+0x0/0xe0
> May 16 00:40:19 fileserver-1 kernel:  [<c011fa3b>] ? default_wake_function+0xb/0x10
> May 16 00:40:19 fileserver-1 kernel:  [<c057794d>] ? cache_check+0x2d/0x3e0
> May 16 00:40:19 fileserver-1 kernel:  [<c025d6f9>] ? nfsd3_proc_getattr+0x69/0xe0
> May 16 00:40:19 fileserver-1 kernel:  [<c025fbb0>] ? nfs3svc_decode_fhandle+0x0/0x40
> May 16 00:40:19 fileserver-1 kernel:  [<c025fbb0>] ? nfs3svc_decode_fhandle+0x0/0x40
> May 16 00:40:19 fileserver-1 kernel:  [<c025208a>] ? nfsd_dispatch+0x9a/0x220
> May 16 00:40:19 fileserver-1 kernel:  [<c0251ff0>] ? nfsd_dispatch+0x0/0x220
> May 16 00:40:19 fileserver-1 kernel:  [<c057106b>] ? svc_process+0x3eb/0x6c0
> May 16 00:40:19 fileserver-1 kernel:  [<c0252746>] ? nfsd+0x136/0x240
> May 16 00:40:19 fileserver-1 kernel:  [<c011c5d8>] ? complete+0x48/0x60
> May 16 00:40:19 fileserver-1 kernel:  [<c0252610>] ? nfsd+0x0/0x240
> May 16 00:40:19 fileserver-1 kernel:  [<c0138972>] ? kthread+0x42/0x70
> May 16 00:40:19 fileserver-1 kernel:  [<c0138930>] ? kthread+0x0/0x70
> May 16 00:40:19 fileserver-1 kernel:  [<c010389b>] ? kernel_thread_helper+0x7/0x1c
> May 16 00:40:19 fileserver-1 kernel: Code: 8f fa 85 d2 ba 20 dc 8f fa 0f 44 c2 89 86 9c 00 00 00 e9 39 ff ff ff 83 8e 44 01 00 00 20 e9 a1 fc ff ff 0f 0b eb fe 0f 0b eb fe <0f> 0b eb fe 0f 0b eb fe 90 8d b4 26 00 00 00 00 55 89 e5 57 56
> May 16 00:40:19 fileserver-1 kernel: EIP: [<fa8c2580>] ocfs2_populate_inode+0x550/0x560 [ocfs2] SS:ESP 0068:f2821c6c
> May 16 00:40:19 fileserver-1 kernel: ---[ end trace 3b05f9cfd74396a1 ]---
>
> NFS with OCFS2 problems?
> I went back to my previous kernel 2.6.25.5 and it seemed to be stable. At this moment I have 2 mounted (production) filesystems and 1 umounted  corrupted filesystem. This morning I looked in the logs and again errors!
> Many like this:
> (249,1):ocfs2_orphan_del:1869 ERROR: status = -2
> (249,1):ocfs2_remove_inode:610 ERROR: status = -2
> (249,1):ocfs2_wipe_inode:736 ERROR: status = -2
> (249,1):ocfs2_delete_inode:970 ERROR: status = -2

Finally, a harmless error. ENOENT. It probably is NFS related. I don't 
have the
said kernel handy right now to confirm.

> This came from the 2 filesystems that seemed to be clean last night.
>  
> - What can I do to prevent filesystem corruption on my 2 production OCFS2 filesystems and get rid of the above errors?
> - Is it possible to fix the corrupted thirth filesystem?
> - What is the most stable kernel (or setup) in my case? Now (and the last year) I am using 2.6.25.5. The 2.6.29.3 kernel I've tried crashed after a couple of minutes.
>
> Versions:
> OS: Debian Etch (4.0)
> kernel: custom 2.6.25.5

I would avoid LVM. We are working towards supporting CLVM but we don't do it
today. I mean using CLVM would be better. Actually, if you 
have-to-have-to use
LVM, use sles11 ha ext. It will have proper ocfs2/clvm support.

I am not sure as to why 2.6.29.3 crashed and 2.6.25 worked. The error 
reported
by 2.6.29.3 should have shown up with 2.6.25 too. Just for the record - 
we did
run the full fs regression with 2.6.29-stock kernel.