[Ocfs2-users] Unstable Cluster

[Tony Rios]

I managed to get ahold of the kernel panic message because it's happening on any new machines I try to introduce to the cluster:

[   66.276054] OCFS2 1.5.0
[   66.337531] o2dlm: Nodes in domain A3AA504BE42E4D3D8A15248D8FCD49BB: 3 5 
[   66.380092] ocfs2: Mounting device (8,16) on (node 5, slot 2) with ordered data mode.
[   66.401719] (ocfs2rec,1382,0):ocfs2_replay_journal:1601 Recovering node 1 from slot 1 on device (8,16)
[  118.890439] o2net: connected to node pedge33 (num 1) at 10.88.0.33:7777
[  118.911765] o2net: connected to node pedge38 (num 4) at 10.88.0.38:7777
[  240.440024] INFO: task kworker/u:3:46 blocked for more than 120 seconds.
[  240.460495] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  240.481125] kworker/u:3     D 0000000000000001     0    46      2 0x00000000
[  240.501761]  ffff88020338dbd0 0000000000000046 ffff88020338dfd8 ffff88020338c000
[  240.522547]  0000000000013d00 ffff88020333c858 ffff88020338dfd8 0000000000013d00
[  240.543180]  ffff8802034c96e0 ffff88020333c4a0 ffff880205105b80 0000000000000001
[  240.563579] Call Trace:
[  240.583607]  [<ffffffffa03c01dd>] ocfs2_wait_for_recovery+0x7d/0xd0 [ocfs2]
[  240.604038]  [<ffffffff81087fb0>] ? autoremove_wake_function+0x0/0x40
[  240.624265]  [<ffffffffa03a8088>] ocfs2_inode_lock_full_nested+0x268/0x6a0 [ocfs2]
[  240.645091]  [<ffffffffa03b6026>] ? ocfs2_node_map_set_bit+0x46/0x60 [ocfs2]
[  240.666163]  [<ffffffffa03bc018>] ocfs2_queue_orphans+0x68/0x260 [ocfs2]
[  240.687431]  [<ffffffff81038c79>] ? default_spin_lock_flags+0x9/0x10
[  240.708689]  [<ffffffffa03bd3d4>] ocfs2_recover_orphans+0x54/0x230 [ocfs2]
[  240.729977]  [<ffffffffa03bbe1c>] ? __ocfs2_wait_on_mount+0xcc/0x140 [ocfs2]
[  240.751404]  [<ffffffff81087fb0>] ? autoremove_wake_function+0x0/0x40
[  240.772669]  [<ffffffffa03c0417>] ocfs2_complete_recovery+0x1e7/0x690 [ocfs2]
[  240.793894]  [<ffffffffa03c0230>] ? ocfs2_complete_recovery+0x0/0x690 [ocfs2]
[  240.814874]  [<ffffffff8108284d>] process_one_work+0x11d/0x420
[  240.835518]  [<ffffffff810832e9>] worker_thread+0x169/0x360
[  240.855910]  [<ffffffff81083180>] ? worker_thread+0x0/0x360
[  240.876296]  [<ffffffff81087866>] kthread+0x96/0xa0
[  240.896305]  [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
[  240.915969]  [<ffffffff810877d0>] ? kthread+0x0/0xa0
[  240.935412]  [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
[  240.954898] INFO: task ureadahead:1384 blocked for more than 120 seconds.
[  240.974688] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  240.994902] ureadahead      D 0000000000000001     0  1384      1 0x00000000
[  241.015064]  ffff880206991d08 0000000000000086 ffff880206991fd8 ffff880206990000
[  241.035271]  0000000000013d00 ffff8802051ac858 ffff880206991fd8 0000000000013d00
[  241.055316]  ffff8802068216e0 ffff8802051ac4a0 ffff880206991ce8 0000000000000001
[  241.075007] Call Trace:
[  241.094061]  [<ffffffffa03c01dd>] ocfs2_wait_for_recovery+0x7d/0xd0 [ocfs2]
[  241.113431]  [<ffffffff81087fb0>] ? autoremove_wake_function+0x0/0x40
[  241.132382]  [<ffffffffa03a8088>] ocfs2_inode_lock_full_nested+0x268/0x6a0 [ocfs2]
[  241.151399]  [<ffffffffa03b8812>] ocfs2_inode_revalidate+0x72/0x2c0 [ocfs2]
[  241.170682]  [<ffffffffa03b0839>] ocfs2_getattr+0x59/0x1d0 [ocfs2]
[  241.189648]  [<ffffffff81169521>] vfs_getattr+0x51/0x120
[  241.208176]  [<ffffffff81169648>] vfs_fstatat+0x58/0x70
[  241.226132]  [<ffffffff8116969b>] vfs_stat+0x1b/0x20
[  241.243766]  [<ffffffff811698da>] sys_newstat+0x1a/0x40
[  241.261254]  [<ffffffff815c3955>] ? page_fault+0x25/0x30
[  241.278749]  [<ffffffff8100c002>] system_call_fastpath+0x16/0x1b