Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_commit_truncate:6490 ERROR: status = -5 Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_delete_inode:974 ERROR: status = -5 Mar 15 14:31:34 host2 kernel: (2008,1):__ocfs2_flush_truncate_log:5111 ERROR: status = -5 Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_free_clusters:1842 ERROR: status = -5 Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_free_suballoc_bits:1755 ERROR: status = -5 Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_replay_truncate_records:5039 ERROR: status = -5 Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_truncate_for_delete:562 ERROR: status = -5 Mar 15 14:31:34 host2 kernel: (2008,1):ocfs2_wipe_inode:733 ERROR: status = -5 Mar 15 14:31:34 host2 kernel: File system is now read-only due to the potential of on-disk corruption. Please run fsck.ocfs2 once the file system is unmounted. Mar 15 14:31:34 host2 kernel: OCFS2: ERROR (device xvdb1): ocfs2_check_group_descriptor: Group descriptor # 12257280 has bit count 32256 but claims that 32639 are free Mar 15 14:31:36 host2 kernel: (1796,1):__ocfs2_flush_truncate_log:5111 ERROR: status = -5 Mar 15 14:31:36 host2 kernel: (1796,1):ocfs2_free_clusters:1842 ERROR: status = -5 Mar 15 14:31:36 host2 kernel: (1796,1):ocfs2_free_suballoc_bits:1755 ERROR: status = -5 Mar 15 14:31:36 host2 kernel: (1796,1):ocfs2_replay_truncate_records:5039 ERROR: status = -5 Mar 15 14:31:36 host2 kernel: (1796,1):ocfs2_truncate_log_worker:5150 ERROR: status = -5 Mar 15 14:31:36 host2 kernel: OCFS2: ERROR (device xvdb1): ocfs2_check_group_descriptor: Group descriptor # 12257280 has bit count 32256 but claims that 32639 are free Mar 15 14:37:34 host2 kernel: (2393,0):__ocfs2_flush_truncate_log:5111 ERROR: status = -5 Mar 15 14:37:34 host2 kernel: (2393,0):ocfs2_free_clusters:1842 ERROR: status = -5 Mar 15 14:37:34 host2 kernel: (2393,0):ocfs2_free_suballoc_bits:1755 ERROR: status = -5 Mar 15 14:37:34 host2 kernel: (2393,0):ocfs2_replay_truncate_records:5039 ERROR: status = -5 Mar 15 14:37:34 host2 kernel: (2393,0):ocfs2_sync_fs:220 ERROR: status = -5 Mar 15 14:37:34 host2 kernel: (2393,1):__ocfs2_flush_truncate_log:5111 ERROR: status = -5 Mar 15 14:37:34 host2 kernel: (2393,1):ocfs2_free_clusters:1842 ERROR: status = -5 Mar 15 14:37:34 host2 kernel: (2393,1):ocfs2_free_suballoc_bits:1755 ERROR: status = -5 Mar 15 14:37:34 host2 kernel: (2393,1):ocfs2_replay_truncate_records:5039 ERROR: status = -5 Mar 15 14:37:34 host2 kernel: (2393,1):ocfs2_sync_fs:220 ERROR: status = -5 Mar 15 14:37:34 host2 kernel: OCFS2: ERROR (device xvdb1): ocfs2_check_group_descriptor: Group descriptor # 12257280 has bit count 32256 but claims that 32639 are free Mar 15 14:37:34 host2 kernel: OCFS2: ERROR (device xvdb1): ocfs2_check_group_descriptor: Group descriptor # 12257280 has bit count 32256 but claims that 32639 are free Mar 15 14:37:37 host2 kernel: (2393,0):__ocfs2_flush_truncate_log:5111 ERROR: status = -5 Mar 15 14:37:37 host2 kernel: (2393,0):ocfs2_free_clusters:1842 ERROR: status = -5 Mar 15 14:37:37 host2 kernel: (2393,0):ocfs2_free_suballoc_bits:1755 ERROR: status = -5 Mar 15 14:37:37 host2 kernel: (2393,0):ocfs2_replay_truncate_records:5039 ERROR: status = -5 Mar 15 14:37:37 host2 kernel: (2393,0):ocfs2_truncate_log_shutdown:5351 ERROR: status = -5 Mar 15 14:37:37 host2 kernel: OCFS2: ERROR (device xvdb1): ocfs2_check_group_descriptor: Group descriptor # 12257280 has bit count 32256 but claims that 32639 are free Mar 15 14:37:46 host1 kernel: ocfs2_dlm: Node 1 leaves domain 6EAF64F9C61F4421A45B97A4418ADE4F Mar 15 14:37:46 host1 kernel: ocfs2_dlm: Nodes in domain ("6EAF64F9C61F4421A45B97A4418ADE4F"): 0 2 Mar 15 14:37:46 host2 kernel: o2net: no longer connected to node host1 (num 0) at 10.8.4.200:7778 Mar 15 14:37:46 host2 kernel: o2net: no longer connected to node host3 (num 2) at 10.8.4.202:7778 Mar 15 14:37:46 host2 kernel: ocfs2: Unmounting device (202,17) on (node 1) Mar 15 14:37:46 host3 kernel: ocfs2_dlm: Node 1 leaves domain 6EAF64F9C61F4421A45B97A4418ADE4F Mar 15 14:37:46 host3 kernel: ocfs2_dlm: Nodes in domain ("6EAF64F9C61F4421A45B97A4418ADE4F"): 0 2 Mar 15 14:37:47 host1 kernel: o2net: no longer connected to node host2 (num 1) at 10.8.4.201:7778 Mar 15 14:37:47 host3 kernel: o2net: no longer connected to node host2 (num 1) at 10.8.4.201:7778 Mar 15 14:37:59 host3 kernel: (7375,1):__ocfs2_flush_truncate_log:5111 ERROR: status = -5 Mar 15 14:37:59 host3 kernel: (7375,1):ocfs2_free_clusters:1842 ERROR: status = -5 Mar 15 14:37:59 host3 kernel: (7375,1):ocfs2_free_suballoc_bits:1755 ERROR: status = -5 Mar 15 14:37:59 host3 kernel: (7375,1):ocfs2_replay_truncate_records:5039 ERROR: status = -5 Mar 15 14:37:59 host3 kernel: (7375,1):ocfs2_sync_fs:220 ERROR: status = -5 Mar 15 14:37:59 host3 kernel: File system is now read-only due to the potential of on-disk corruption. Please run fsck.ocfs2 once the file system is unmounted. Mar 15 14:37:59 host3 kernel: OCFS2: ERROR (device xvdb1): ocfs2_check_group_descriptor: Group descriptor # 12257280 has bit count 32256 but claims that 32639 are free Mar 15 14:38:17 host1 kernel: ocfs2_dlm: Node 2 leaves domain 6EAF64F9C61F4421A45B97A4418ADE4F Mar 15 14:38:17 host1 kernel: ocfs2_dlm: Nodes in domain ("6EAF64F9C61F4421A45B97A4418ADE4F"): 0 Mar 15 14:38:19 host1 kernel: o2net: no longer connected to node host3 (num 2) at 10.8.4.202:7778 Mar 15 14:38:19 host3 kernel: o2net: no longer connected to node host1 (num 0) at 10.8.4.200:7778 Mar 15 14:38:19 host3 kernel: ocfs2: Unmounting device (202,17) on (node 2) Mar 15 14:38:24 host1 kernel: ocfs2: Unmounting device (202,17) on (node 0) Mar 15 14:39:41 host2 kernel: ocfs2_dlm: Nodes in domain ("6EAF64F9C61F4421A45B97A4418ADE4F"): 1 ----------------------------------------- Checking OCFS2 filesystem in /dev/xvdb1: label: volstor01 uuid: 6e af 64 f9 c6 1f 44 21 a4 5b 97 a4 41 8a de 4f number of blocks: 167772803 bytes per block: 4096 number of clusters: 167772803 bytes per cluster: 4096 max slots: 16 /dev/xvdb1 was run with -f, check forced. Pass 0a: Checking cluster allocation chains [GROUP_FREE_BITS] Group descriptor at block 12257280 claims to have 32639 free bits which is more than 32238 bits indicated by the bitmap. Drop its free bit count down to the total? y [CHAIN_BITS] Chain 137 in allocator inode 11 has 249271 bits marked free out of 677376 total bits but the block groups in the chain have 248870 free out of 677376 total. Fix this by updating the chain record? y [CHAIN_GROUP_BITS] Allocator inode 11 has 109425928 bits marked used out of 167772803 total bits but the chains have 109426329 used out of 167772803 total. Fix this by updating the inode counts? y Pass 0b: Checking inode allocation chains Pass 0c: Checking extent block allocation chains Pass 1: Checking inodes and blocks. [CLUSTER_ALLOC_BIT] Cluster 12268148 is marked in the global cluster bitmap but it isn't in use. Clear its bit in the bitmap? y [CLUSTER_ALLOC_BIT] Cluster 12271814 is marked in the global cluster bitmap but it isn't in use. Clear its bit in the bitmap? y [CLUSTER_ALLOC_BIT] Cluster 12272251 is marked in the global cluster bitmap but it isn't in use. Clear its bit in the bitmap? y [CLUSTER_ALLOC_BIT] Cluster 12273264 is marked in the global cluster bitmap but it isn't in use. Clear its bit in the bitmap? y [CLUSTER_ALLOC_BIT] Cluster 12274133 is marked in the global cluster bitmap but it isn't in use. Clear its bit in the bitmap? y [CLUSTER_ALLOC_BIT] Cluster 12274534 is marked in the global cluster bitmap but it isn't in use. Clear its bit in the bitmap? y Pass 4a: checking for orphaned inodes [INODE_ORPHANED] Inode 15975682 was found in the orphan directory. Delete its contents and unlink it? y [INODE_ORPHANED] Inode 2313105 was found in the orphan directory. Delete its contents and unlink it? y [INODE_ORPHANED] Inode 3712900 was found in the orphan directory. Delete its contents and unlink it? y [INODE_ORPHANED] Inode 3712901 was found in the orphan directory. Delete its contents and unlink it? y [INODE_ORPHANED] Inode 3712902 was found in the orphan directory. Delete its contents and unlink it? y [INODE_ORPHANED] Inode 3712903 was found in the orphan directory. Delete its contents and unlink it? y [INODE_ORPHANED] Inode 3712904 was found in the orphan directory. Delete its contents and unlink it?