[Ocfs2-users] OCFS2 Crash

[Martin Lund]

Hello,

We have a 3 node OCFS2 cluster, using:

Kernel: 3.16.0-0.bpo.4-amd64 
ii  ocfs2-tools                           1.6.4-1+deb7u1                amd64        tools for managing OCFS2 cluster filesystems

Today two of the nodes out of the 3 had some partial OCFS2 related kernel panic (see at the end). After rebooting the first node the FS become available again. Anyone run into error like this?

[Fri Aug 21 11:17:27 2015] INFO: task df:16329 blocked for more than 120 seconds.
[Fri Aug 21 11:17:27 2015]       Not tainted 3.16.0-0.bpo.4-amd64 #1
[Fri Aug 21 11:17:27 2015] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Fri Aug 21 11:17:27 2015] df              D ffff88003fc12ec0     0 16329  16324 0x00000000
[Fri Aug 21 11:17:27 2015]  ffffffff81818480 0000000000000086 0000000000000000 ffff88003a96a210
[Fri Aug 21 11:17:27 2015]  0000000000012ec0 ffff88000b43ffd8 0000000000012ec0 ffff88003a96a210
[Fri Aug 21 11:17:27 2015]  0000000000000002 ffff88000b43fd08 7fffffffffffffff ffff88000b43fd00
[Fri Aug 21 11:17:27 2015] Call Trace:
[Fri Aug 21 11:17:27 2015]  [<ffffffff815478cd>] ? schedule_timeout+0x1dd/0x240
[Fri Aug 21 11:17:27 2015]  [<ffffffff81156f65>] ? __alloc_pages_nodemask+0x165/0xbb0
[Fri Aug 21 11:17:27 2015]  [<ffffffff8119c86c>] ? alloc_pages_vma+0xac/0x180
[Fri Aug 21 11:17:27 2015]  [<ffffffff815496ac>] ? wait_for_completion+0xac/0x120
[Fri Aug 21 11:17:27 2015]  [<ffffffff810a0650>] ? try_to_wake_up+0x310/0x310
[Fri Aug 21 11:17:27 2015]  [<ffffffffa04871ea>] ? __ocfs2_cluster_lock.isra.36+0x1ba/0x7c0 [ocfs2]
[Fri Aug 21 11:17:27 2015]  [<ffffffff811c85de>] ? __inode_permission+0x2e/0xd0
[Fri Aug 21 11:17:27 2015]  [<ffffffff811c8b4b>] ? link_path_walk+0x5b/0x880
[Fri Aug 21 11:17:27 2015]  [<ffffffffa0488929>] ? ocfs2_inode_lock_full_nested+0x149/0xbb0 [ocfs2]
[Fri Aug 21 11:17:27 2015]  [<ffffffff811c9b34>] ? filename_lookup+0x34/0xd0
[Fri Aug 21 11:17:27 2015]  [<ffffffff811c83d4>] ? getname_flags+0xc4/0x1b0
[Fri Aug 21 11:17:27 2015]  [<ffffffffa04ce2f9>] ? ocfs2_statfs+0x79/0x350 [ocfs2]
[Fri Aug 21 11:17:27 2015]  [<ffffffff811edbb0>] ? statfs_by_dentry+0xa0/0x140
[Fri Aug 21 11:17:27 2015]  [<ffffffff811edc6a>] ? vfs_statfs+0x1a/0xa0
[Fri Aug 21 11:17:27 2015]  [<ffffffff811edd2e>] ? user_statfs+0x3e/0x70
[Fri Aug 21 11:17:27 2015]  [<ffffffff811eddb2>] ? SYSC_statfs+0x12/0x30
[Fri Aug 21 11:17:27 2015]  [<ffffffff8154bdcd>] ? system_call_fast_compare_end+0x10/0x15
[Fri Aug 21 11:17:27 2015] INFO: task df:16336 blocked for more than 120 seconds.
[Fri Aug 21 11:17:27 2015]       Not tainted 3.16.0-0.bpo.4-amd64 #1
[Fri Aug 21 11:17:27 2015] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Fri Aug 21 11:17:27 2015] df              D ffff88003fc12ec0     0 16336  16331 0x00000000
[Fri Aug 21 11:17:27 2015]  ffffffff81818480 0000000000000082 0000000000000000 ffff8800370dcce0
[Fri Aug 21 11:17:27 2015]  0000000000012ec0 ffff8800239e7fd8 0000000000012ec0 ffff8800370dcce0
[Fri Aug 21 11:17:27 2015]  0000000000000001 ffff8800239e7d08 7fffffffffffffff ffff8800239e7d00
[Fri Aug 21 11:17:27 2015] Call Trace:
[Fri Aug 21 11:17:27 2015]  [<ffffffff815478cd>] ? schedule_timeout+0x1dd/0x240
[Fri Aug 21 11:17:27 2015]  [<ffffffff81156f65>] ? __alloc_pages_nodemask+0x165/0xbb0
[Fri Aug 21 11:17:27 2015]  [<ffffffff8119c86c>] ? alloc_pages_vma+0xac/0x180
[Fri Aug 21 11:17:27 2015]  [<ffffffff815496ac>] ? wait_for_completion+0xac/0x120
[Fri Aug 21 11:17:27 2015]  [<ffffffff810a0650>] ? try_to_wake_up+0x310/0x310
[Fri Aug 21 11:17:27 2015]  [<ffffffffa04871ea>] ? __ocfs2_cluster_lock.isra.36+0x1ba/0x7c0 [ocfs2]
[Fri Aug 21 11:17:27 2015]  [<ffffffff811c85de>] ? __inode_permission+0x2e/0xd0
[Fri Aug 21 11:17:27 2015]  [<ffffffff811c8b4b>] ? link_path_walk+0x5b/0x880
[Fri Aug 21 11:17:27 2015]  [<ffffffffa0488929>] ? ocfs2_inode_lock_full_nested+0x149/0xbb0 [ocfs2]
[Fri Aug 21 11:17:27 2015]  [<ffffffff811c9b34>] ? filename_lookup+0x34/0xd0
[Fri Aug 21 11:17:27 2015]  [<ffffffff811c83d4>] ? getname_flags+0xc4/0x1b0
[Fri Aug 21 11:17:27 2015]  [<ffffffffa04ce2f9>] ? ocfs2_statfs+0x79/0x350 [ocfs2]
[Fri Aug 21 11:17:27 2015]  [<ffffffff811edbb0>] ? statfs_by_dentry+0xa0/0x140
[Fri Aug 21 11:17:27 2015]  [<ffffffff811edc6a>] ? vfs_statfs+0x1a/0xa0
[Fri Aug 21 11:17:27 2015]  [<ffffffff811edd2e>] ? user_statfs+0x3e/0x70
[Fri Aug 21 11:17:27 2015]  [<ffffffff811eddb2>] ? SYSC_statfs+0x12/0x30
[Fri Aug 21 11:17:27 2015]  [<ffffffff8154bdcd>] ? system_call_fast_compare_end+0x10/0x15


Thank you