<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:10pt"><div><br>Hi, <br></div><div><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">We saw the following errors this morning on 2 of the 3 node OCFS2 (1.4.7-1) cluster running on Redhat5.5. Does anyone know what it means?</div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">Sep 12 07:02:54 ncom kernel: [<ffffffff888144b7>] :ocfs2:ocfs2_wait_for_mask+0xd/0x19<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888158d8>]
:ocfs2:ocfs2_cluster_lock+0x9ae/0x9d3<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8881429c>] :ocfs2:ocfs2_init_mask_waiter+0x24/0x3d<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888160b8>] :ocfs2:ocfs2_inode_lock_full+0x24e/0xfd9<br>Sep 12 07:02:54 ncom kernel: [<ffffffff88834379>] :ocfs2:ocfs2_mknod+0xf4/0xa07<br>Sep 12 07:02:54 ncom kernel: [<ffffffff88834e1b>] :ocfs2:ocfs2_create+0x91/0xff<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888144b7>] :ocfs2:ocfs2_wait_for_mask+0xd/0x19<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888158d8>] :ocfs2:ocfs2_cluster_lock+0x9ae/0x9d3<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888160b8>] :ocfs2:ocfs2_inode_lock_full+0x24e/0xfd9<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888128b9>] :ocfs2:ocfs2_cluster_unlock+0x290/0x30d<br>Sep 12 07:02:54 ncom kernel: [<ffffffff88824176>] :ocfs2:ocfs2_permission+0x77/0x1a4<br>Sep 12
07:02:54 ncom kernel: [<ffffffff888157d1>] :ocfs2:ocfs2_cluster_lock+0x8a7/0x9d3<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888144b7>] :ocfs2:ocfs2_wait_for_mask+0xd/0x19<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888158d8>] :ocfs2:ocfs2_cluster_lock+0x9ae/0x9d3<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888160b8>] :ocfs2:ocfs2_inode_lock_full+0x24e/0xfd9<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888128b9>] :ocfs2:ocfs2_cluster_unlock+0x290/0x30d<br>Sep 12 07:02:54 ncom kernel: [<ffffffff88824176>] :ocfs2:ocfs2_permission+0x77/0x1a4<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8881429c>] :ocfs2:ocfs2_init_mask_waiter+0x24/0x3d<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888157d1>] :ocfs2:ocfs2_cluster_lock+0x8a7/0x9d3<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888144b7>] :ocfs2:ocfs2_wait_for_mask+0xd/0x19<br>Sep 12 07:02:54 ncom kernel:
[<ffffffff888158d8>] :ocfs2:ocfs2_cluster_lock+0x9ae/0x9d3<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888160b8>] :ocfs2:ocfs2_inode_lock_full+0x24e/0xfd9<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888128b9>] :ocfs2:ocfs2_cluster_unlock+0x290/0x30d<br>Sep 12 07:02:54 ncom kernel: [<ffffffff88824176>] :ocfs2:ocfs2_permission+0x77/0x1a4<br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">We also saw the application got stuck, but not sure which one was the cause:</div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br></div><div
style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">Sep 12 07:02:54 ncom kernel: INFO: task ReplayServer:19626 blocked for more than 120 seconds.<br>Sep 12 07:02:54 ncom kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>Sep 12 07:02:54 ncom kernel: ReplayServer D ffff810b12997860 0 19626 9648 19627 19625 (NOTLB)<br>Sep 12 07:02:54 ncom kernel: ffff810a7e513dc8 0000000000200086 ffff810c257feba0 ffffffff800099ae<br>Sep 12 07:02:54 ncom kernel: 0000000000002440 000000000000000a ffff810aafcbe820 ffff810b12997860<br>Sep 12 07:02:54 ncom kernel: 00018bd3e41e07b5 000000000000e79e ffff810aafcbea08 0000000127eda9c0<br>Sep 12 07:02:54 ncom kernel: Call Trace:<br>Sep 12 07:02:54 ncom kernel:
[<ffffffff800099ae>] __link_path_walk+0x173/0xf42<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8002cd2c>] mntput_no_expire+0x19/0x89<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8000ea75>] link_path_walk+0xa6/0xb2<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80064c6f>] __mutex_lock_slowpath+0x60/0x9b<br>Sep 12 07:02:54 ncom kernel: [<ffffffff800236d7>] __path_lookup_intent_open+0x56/0x97<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80064cb9>] .text.lock.mutex+0xf/0x14<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8001afe7>] open_namei+0xea/0x6d5<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80067b88>] do_page_fault+0x4fe/0x874<br>Sep 12 07:02:54 ncom kernel: [<ffffffff800274fb>] do_filp_open+0x1c/0x38<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80019e1e>] do_sys_open+0x44/0xbe<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8005e116>]
system_call+0x7e/0x83<br>Sep 12 07:02:54 ncom kernel: <br>Sep 12 07:02:54 ncom kernel: INFO: task ReplayServer:19634 blocked for more than 120 seconds.<br>Sep 12 07:02:54 ncom kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>Sep 12 07:02:54 ncom kernel: ReplayServer D ffff810b3ce150c0 0 19634 9648 19635 19633 (NOTLB)<br>Sep 12 07:02:54 ncom kernel: ffff810aafa3bc58 0000000000200086 0000000052319eb3 00000000178c2c15<br>Sep 12 07:02:54 ncom kernel: ffff810aafa3bba8 000000000000000a ffff810a914e3080 ffff810b3ce150c0<br>Sep 12 07:02:54 ncom kernel: 00018bd3ebab498a 0000000000018f23 ffff810a914e3268 00000001888140b3<br>Sep 12 07:02:54 ncom kernel: Call Trace:<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80064c6f>] __mutex_lock_slowpath+0x60/0x9b<br>Sep 12 07:02:54 ncom kernel:
[<ffffffff80064cb9>] .text.lock.mutex+0xf/0x14<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8000cef1>] do_lookup+0x90/0x1e6<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8000a23c>] __link_path_walk+0xa01/0xf42<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8000ea11>] link_path_walk+0x42/0xb2<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8000cce1>] do_path_lookup+0x275/0x2f1<br>Sep 12 07:02:54 ncom kernel: [<ffffffff800236d7>] __path_lookup_intent_open+0x56/0x97<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8001af70>] open_namei+0x73/0x6d5<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80067b88>] do_page_fault+0x4fe/0x874<br>Sep 12 07:02:54 ncom kernel: [<ffffffff800274fb>] do_filp_open+0x1c/0x38<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80019e1e>] do_sys_open+0x44/0xbe<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8005e116>] system_call+0x7e/0x83<br>Sep 12
07:02:54 ncom kernel: <br>Sep 12 07:02:54 ncom kernel: INFO: task ReplayServer:19629 blocked for more than 120 seconds.<br>Sep 12 07:02:54 ncom kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>Sep 12 07:02:54 ncom kernel: ReplayServer D ffff81063d0e1420 0 19629 9648 19630 19628 (NOTLB)<br>Sep 12 07:02:54 ncom kernel: ffff810b13007dc8 0000000000200082 ffff810c257feba0 ffffffff800099ae<br>Sep 12 07:02:54 ncom kernel: 0000000000002a97 000000000000000a ffff810b7723b7e0 ffff810628051080<br>Sep 12 07:02:54 ncom kernel: 00018bd41261c4c8 00000000000090fa ffff810b7723b9c8 0000000127eda9c0<br>Sep 12 07:02:54 ncom kernel: Call Trace:<br>Sep 12 07:02:54 ncom kernel: [<ffffffff800099ae>] __link_path_walk+0x173/0xf42<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8002cd2c>]
mntput_no_expire+0x19/0x89<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80064c6f>] __mutex_lock_slowpath+0x60/0x9b<br>Sep 12 07:02:54 ncom kernel: [<ffffffff800236d7>] __path_lookup_intent_open+0x56/0x97<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80064cb9>] .text.lock.mutex+0xf/0x14<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8001afe7>] open_namei+0xea/0x6d5<br>Sep 12 07:02:54 ncom kernel: [<ffffffff88277b9f>] :dm_mod:dm_put+0x2a/0x15d<br>Sep 12 07:02:54 ncom kernel: [<ffffffff800274fb>] do_filp_open+0x1c/0x38<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80019e1e>] do_sys_open+0x44/0xbe<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8005e116>] system_call+0x7e/0x83<br>Sep 12 07:02:54 ncom kernel: <br>Sep 12 07:02:54 ncom kernel: INFO: task ReplayServer:19639 blocked for more than 120 seconds.<br>Sep 12 07:02:54 ncom kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>Sep 12 07:02:54 ncom kernel: ReplayServer D 0000000000000000 0 19639 9648 19640 19638 (NOTLB)<br>Sep 12 07:02:54 ncom kernel: ffff810a916f3b98 0000000000200082 00000000000024f0 0000000014234000<br>Sep 12 07:02:54 ncom kernel: ffff810c2427bba0 0000000000000009 ffff810b1768e0c0 ffff810a914e37e0<br>Sep 12 07:02:54 ncom kernel: 00018bd422e0df8e 00000000000006cc ffff810b1768e2a8 0000000180019bc9<br>Sep 12 07:02:54 ncom kernel: Call Trace:<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80064167>] wait_for_completion+0x79/0xa2<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8008e16d>] default_wake_function+0x0/0xe<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888144b7>] :ocfs2:ocfs2_wait_for_mask+0xd/0x19<br>Sep 12 07:02:54 ncom kernel:
[<ffffffff888158d8>] :ocfs2:ocfs2_cluster_lock+0x9ae/0x9d3<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8881429c>] :ocfs2:ocfs2_init_mask_waiter+0x24/0x3d<br>Sep 12 07:02:54 ncom kernel: [<ffffffff888160b8>] :ocfs2:ocfs2_inode_lock_full+0x24e/0xfd9<br>Sep 12 07:02:54 ncom kernel: [<ffffffff88834379>] :ocfs2:ocfs2_mknod+0xf4/0xa07<br>Sep 12 07:02:54 ncom kernel: [<ffffffff88834e1b>] :ocfs2:ocfs2_create+0x91/0xff<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8003a8a2>] vfs_create+0xe6/0x158<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8001b09a>] open_namei+0x19d/0x6d5<br>Sep 12 07:02:54 ncom kernel: [<ffffffff800274fb>] do_filp_open+0x1c/0x38<br>Sep 12 07:02:54 ncom kernel: [<ffffffff80019e1e>] do_sys_open+0x44/0xbe<br>Sep 12 07:02:54 ncom kernel: [<ffffffff8005e116>] system_call+0x7e/0x83<br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px;
font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">Thanks,</div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">William</div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br></div><div
style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br></div><div class="yui_3_7_2_50_1378987032718_60" style="font-family: times new roman, new york, times, serif; font-size: 10pt;"><div class="yui_3_7_2_50_1378987032718_120" style="font-family: times new roman, new york, times, serif; font-size: 12pt;"><br> </div> </div> <span></span><div></div></div></body></html>