<div dir="ltr"><div><div><div>
[ 1481.620253] o2hb: Unable to stabilize heartbeart on region
1352E2692E704EEB8040E5B8FF560997 (vdb) <br><br></div>What this means is that the device is suspect. o2hb writes are not hitting the disk. vdb is accepting and<br>acknowledging the write but spitting out something else during the next read. Heartbeat detects this and<br>
aborts, as it should.<br></div><br></div>Then we hit a race during socket close that triggers the oops. Yes, that needs to be fixed. But you also<br>need to "fix" vdb... what appears to be a virtual device.<br></div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Mar 1, 2013 at 1:25 PM, richard -rw- weinberger <span dir="ltr"><<a href="mailto:richard.weinberger@gmail.com" target="_blank">richard.weinberger@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi!<br>
<br>
Using 3.8.1 OCFS2 crashes while joining nodes to the cluster.<br>
The cluster consists of 10 nodes, while node3 joins the kernel on node3 crashes.<br>
(Somtimes later...)<br>
See dmesg below.<br>
Is this a known issue? I didn't test older kernels so far.<br>
<br>
node1:<br>
[ 1471.881922] o2dlm: Joining domain 1352E2692E704EEB8040E5B8FF560997<br>
( 0 ) 1 nodes<br>
[ 1471.919522] JBD2: Ignoring recovery information on journal<br>
[ 1471.947027] ocfs2: Mounting device (253,16) on (node 0, slot 0)<br>
with ordered data mode.<br>
[ 1475.802497] o2net: Accepted connection from node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a><br>
[ 1481.814048] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 8<br>
[ 1481.814955] o2net: No longer connected to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a><br>
[ 1482.468827] o2net: Accepted connection from node node3 (num 2) at<br>
<a href="http://192.168.66.3:7777" target="_blank">192.168.66.3:7777</a><br>
[ 1511.904100] o2net: No connection established with node 1 after 30.0<br>
seconds, giving up.<br>
[ 1514.472995] o2net: Connection to node node3 (num 2) at<br>
<a href="http://192.168.66.3:7777" target="_blank">192.168.66.3:7777</a> shutdown, state 8<br>
[ 1514.473960] o2net: No longer connected to node node3 (num 2) at<br>
<a href="http://192.168.66.3:7777" target="_blank">192.168.66.3:7777</a><br>
[ 1516.076044] o2net: Accepted connection from node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a><br>
[ 1520.181430] o2dlm: Node 1 joins domain<br>
1352E2692E704EEB8040E5B8FF560997 ( 0 1 ) 2 nodes<br>
[ 1544.544030] o2net: No connection established with node 2 after 30.0<br>
seconds, giving up.<br>
[ 1574.624029] o2net: No connection established with node 2 after 30.0<br>
seconds, giving up.<br>
<br>
node2:<br>
[ 1475.613170] o2net: Connected to node node1 (num 0) at <a href="http://192.168.66.1:7777" target="_blank">192.168.66.1:7777</a><br>
[ 1481.620253] o2hb: Unable to stabilize heartbeart on region<br>
1352E2692E704EEB8040E5B8FF560997 (vdb)<br>
[ 1481.622489] o2net: No longer connected to node node1 (num 0) at<br>
<a href="http://192.168.66.1:7777" target="_blank">192.168.66.1:7777</a><br>
[ 1515.886605] o2net: Connected to node node1 (num 0) at <a href="http://192.168.66.1:7777" target="_blank">192.168.66.1:7777</a><br>
[ 1519.992766] o2dlm: Joining domain 1352E2692E704EEB8040E5B8FF560997<br>
( 0 1 ) 2 nodes<br>
[ 1520.017054] JBD2: Ignoring recovery information on journal<br>
[ 1520.078888] ocfs2: Mounting device (253,16) on (node 1, slot 1)<br>
with ordered data mode.<br>
[ 1520.159590] mount.ocfs2 (2186) used greatest stack depth: 2568 bytes left<br>
<br>
node3:<br>
[ 1482.836865] o2net: Connected to node node1 (num 0) at <a href="http://192.168.66.1:7777" target="_blank">192.168.66.1:7777</a><br>
[ 1482.837542] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1484.840952] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1486.844994] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1488.848952] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1490.853052] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1492.857046] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1494.861042] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1496.865024] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1498.869021] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1500.873016] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1502.877056] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1504.881042] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1506.885040] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1508.888991] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1510.893077] o2net: Connection to node node2 (num 1) at<br>
<a href="http://192.168.66.2:7777" target="_blank">192.168.66.2:7777</a> shutdown, state 7<br>
[ 1512.843172] (mount.ocfs2,2179,0):dlm_request_join:1477 ERROR: Error<br>
-107 when sending message 510 (key 0x666c6172) to node 1<br>
[ 1512.845580] (mount.ocfs2,2179,0):dlm_try_to_join_domain:1653 ERROR:<br>
status = -107<br>
[ 1512.847778] (mount.ocfs2,2179,0):dlm_join_domain:1955 ERROR: status = -107<br>
[ 1512.849334] (mount.ocfs2,2179,0):dlm_register_domain:2214 ERROR:<br>
status = -107<br>
[ 1512.850921] (mount.ocfs2,2179,0):o2cb_cluster_connect:368 ERROR:<br>
status = -107<br>
[ 1512.852511] (mount.ocfs2,2179,0):ocfs2_dlm_init:3004 ERROR: status = -107<br>
[ 1512.854090] (mount.ocfs2,2179,0):ocfs2_mount_volume:1881 ERROR: status = -107<br>
[ 1512.855476] ocfs2: Unmounting device (253,16) on (node 0)<br>
[ 1512.855915] (mount.ocfs2,2179,0):ocfs2_fill_super:1230 ERROR: status = -107<br>
[ 1514.839138] o2net: No longer connected to node node1 (num 0) at<br>
<a href="http://192.168.66.1:7777" target="_blank">192.168.66.1:7777</a><br>
[ 1514.840690] BUG: unable to handle kernel NULL pointer dereference<br>
at 0000000000000028<br>
[ 1514.841627] IP: [<ffffffff816ce840>] kernel_sock_ioctl+0x50/0x50<br>
[ 1514.841627] PGD 1d980067 PUD 1db0c067 PMD 0<br>
[ 1514.841627] Oops: 0000 [#1] PREEMPT SMP<br>
[ 1514.841627] Modules linked in:<br>
[ 1514.841627] CPU 0<br>
[ 1514.841627] Pid: 6, comm: kworker/u:0 Not tainted 3.8.1+ #8 Bochs Bochs<br>
[ 1514.841627] RIP: 0010:[<ffffffff816ce840>] [<ffffffff816ce840>]<br>
kernel_sock_ioctl+0x50/0x50<br>
[ 1514.841627] RSP: 0018:ffff88001f949d40 EFLAGS: 00010292<br>
[ 1514.841627] RAX: ffff88001e6e9c01 RBX: ffff88001e6e9000 RCX: 0000000180080006<br>
[ 1514.841627] RDX: 0000000180080007 RSI: 0000000000000002 RDI: 0000000000000000<br>
[ 1514.841627] RBP: ffff88001f949dc8 R08: 0000000000016090 R09: 0000000000000001<br>
[ 1514.841627] R10: ffffea000079ba40 R11: ffffffff81321207 R12: ffff88001db69440<br>
[ 1514.841627] R13: ffffffff8204c760 R14: ffff88001db695a8 R15: ffff88001e6e9058<br>
[ 1514.841627] FS: 0000000000000000(0000) GS:ffff88001fc00000(0000)<br>
knlGS:0000000000000000<br>
[ 1514.841627] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br>
[ 1514.841627] CR2: 0000000000000028 CR3: 000000001c009000 CR4: 00000000000006f0<br>
[ 1514.841627] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br>
[ 1514.841627] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br>
[ 1514.841627] Process kworker/u:0 (pid: 6, threadinfo<br>
ffff88001f948000, task ffff88001f90a670)<br>
[ 1514.841627] Stack:<br>
[ 1514.841627] ffffffff81323776 ffff88001fad2e20 0000000000010500<br>
ffff88001f949d98<br>
[ 1514.841627] ffffffff810013fa ffff88001f949d78 ffffffff818a5c22<br>
ffff88001f949da8<br>
[ 1514.841627] ffffffff81069d26 0000000000012900 0000000000012900<br>
0000000000000000<br>
[ 1514.841627] Call Trace:<br>
[ 1514.841627] [<ffffffff81323776>] ? o2net_shutdown_sc+0x106/0x1e0<br>
[ 1514.841627] [<ffffffff810013fa>] ? __switch_to+0x2a/0x4a0<br>
[ 1514.841627] [<ffffffff818a5c22>] ? _raw_spin_unlock_irq+0x12/0x40<br>
[ 1514.841627] [<ffffffff81069d26>] ? finish_task_switch+0x56/0xc0<br>
[ 1514.841627] [<ffffffff81056eb3>] process_one_work+0x133/0x510<br>
[ 1514.841627] [<ffffffff81323670>] ? o2net_sc_connect_completed+0xf0/0xf0<br>
[ 1514.841627] [<ffffffff810585ed>] worker_thread+0x15d/0x450<br>
[ 1514.841627] [<ffffffff81058490>] ? busy_worker_rebind_fn+0x100/0x100<br>
[ 1514.841627] [<ffffffff8105e10b>] kthread+0xbb/0xc0<br>
[ 1514.841627] [<ffffffff818a0000>] ? e1000_regdump+0x262/0x3be<br>
[ 1514.841627] [<ffffffff8105e050>] ? kthread_create_on_node+0x130/0x130<br>
[ 1514.841627] [<ffffffff818accac>] ret_from_fork+0x7c/0xb0<br>
[ 1514.841627] [<ffffffff8105e050>] ? kthread_create_on_node+0x130/0x130<br>
[ 1514.841627] Code: ff ff ff ff ff ff 48 8b 47 28 ff 50 48 4c 89 a3<br>
48 e0 ff ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 66 66 66 2e 0f 1f 84 00<br>
00 00 00 00 <48> 8b 47 28 55 48 89 e5 ff 50 60 5d c3 0f 1f 00 55 41 b8<br>
4b 43<br>
[ 1514.841627] RIP [<ffffffff816ce840>] kernel_sock_ioctl+0x50/0x50<br>
[ 1514.841627] RSP <ffff88001f949d40><br>
[ 1514.841627] CR2: 0000000000000028<br>
[ 1514.867242] ---[ end trace 36ffe9378168cdc2 ]---<br>
[ 1514.867619] BUG: unable to handle kernel paging request at ffffffffffffffd8<br>
[ 1514.868007] IP: [<ffffffff8105e3cb>] kthread_data+0xb/0x20<br>
[ 1514.868007] PGD 1e0d067 PUD 1e0e067 PMD 0<br>
[ 1514.868007] Oops: 0000 [#2] PREEMPT SMP<br>
[ 1514.868007] Modules linked in:<br>
[ 1514.868007] CPU 0<br>
[ 1514.868007] Pid: 6, comm: kworker/u:0 Tainted: G D 3.8.1+<br>
#8 Bochs Bochs<br>
[ 1514.868007] RIP: 0010:[<ffffffff8105e3cb>] [<ffffffff8105e3cb>]<br>
kthread_data+0xb/0x20<br>
[ 1514.868007] RSP: 0018:ffff88001f949928 EFLAGS: 00010092<br>
[ 1514.868007] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006<br>
[ 1514.868007] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88001f90a670<br>
[ 1514.868007] RBP: ffff88001f949928 R08: 0000000000000001 R09: 0000000000000196<br>
[ 1514.868007] R10: ffff88001f9116a0 R11: 0000000000000004 R12: ffff88001f90aa40<br>
[ 1514.868007] R13: 0000000000000000 R14: ffff88001f90a660 R15: ffff88001f90a670<br>
[ 1514.868007] FS: 0000000000000000(0000) GS:ffff88001fc00000(0000)<br>
knlGS:0000000000000000<br>
[ 1514.868007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br>
[ 1514.868007] CR2: ffffffffffffffd8 CR3: 000000001d83d000 CR4: 00000000000006f0<br>
[ 1514.868007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br>
[ 1514.868007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br>
[ 1514.868007] Process kworker/u:0 (pid: 6, threadinfo<br>
ffff88001f948000, task ffff88001f90a670)<br>
[ 1514.868007] Stack:<br>
[ 1514.868007] ffff88001f949948 ffffffff81059980 ffff88001f949948<br>
ffff88001fc12900<br>
[ 1514.868007] ffff88001f9499c8 ffffffff818a4c48 ffff88001f949978<br>
0000000000000000<br>
[ 1514.868007] ffff88001f949fd8 ffff88001f90a670 ffff88001f949fd8<br>
ffff88001f949fd8<br>
[ 1514.868007] Call Trace:<br>
[ 1514.868007] [<ffffffff81059980>] wq_worker_sleeping+0x10/0xc0<br>
[ 1514.868007] [<ffffffff818a4c48>] __schedule+0x528/0x770<br>
[ 1514.868007] [<ffffffff818a4fa4>] schedule+0x24/0x70<br>
[ 1514.868007] [<ffffffff81040c72>] do_exit+0x702/0xad0<br>
[ 1514.868007] [<ffffffff818a7071>] oops_end+0x91/0xe0<br>
[ 1514.868007] [<ffffffff8189a00d>] no_context+0x24e/0x279<br>
[ 1514.868007] [<ffffffff817436f8>] ? tcp_rearm_rto+0xa8/0xd0<br>
[ 1514.868007] [<ffffffff8189a1ed>] __bad_area_nosemaphore+0x1b5/0x1d4<br>
[ 1514.868007] [<ffffffff8189a21a>] bad_area_nosemaphore+0xe/0x10<br>
[ 1514.868007] [<ffffffff818a90b6>] __do_page_fault+0x296/0x490<br>
[ 1514.868007] [<ffffffff816ce8ce>] ? sock_destroy_inode+0x2e/0x40<br>
[ 1514.868007] [<ffffffff8114e787>] ? destroy_inode+0x37/0x60<br>
[ 1514.868007] [<ffffffff8114e8ca>] ? evict+0x11a/0x1b0<br>
[ 1514.868007] [<ffffffff81321207>] ? sc_kref_release+0x77/0x160<br>
[ 1514.868007] [<ffffffff8112d421>] ? kfree+0x121/0x160<br>
[ 1514.868007] [<ffffffff81321207>] ? sc_kref_release+0x77/0x160<br>
[ 1514.868007] [<ffffffff818a92b9>] do_page_fault+0x9/0x10<br>
[ 1514.868007] [<ffffffff818a65e2>] page_fault+0x22/0x30<br>
[ 1514.868007] [<ffffffff81321207>] ? sc_kref_release+0x77/0x160<br>
[ 1514.868007] [<ffffffff816ce840>] ? kernel_sock_ioctl+0x50/0x50<br>
[ 1514.868007] [<ffffffff81323776>] ? o2net_shutdown_sc+0x106/0x1e0<br>
[ 1514.868007] [<ffffffff810013fa>] ? __switch_to+0x2a/0x4a0<br>
[ 1514.868007] [<ffffffff818a5c22>] ? _raw_spin_unlock_irq+0x12/0x40<br>
[ 1514.868007] [<ffffffff81069d26>] ? finish_task_switch+0x56/0xc0<br>
[ 1514.868007] [<ffffffff81056eb3>] process_one_work+0x133/0x510<br>
[ 1514.868007] [<ffffffff81323670>] ? o2net_sc_connect_completed+0xf0/0xf0<br>
[ 1514.868007] [<ffffffff810585ed>] worker_thread+0x15d/0x450<br>
[ 1514.868007] [<ffffffff81058490>] ? busy_worker_rebind_fn+0x100/0x100<br>
[ 1514.868007] [<ffffffff8105e10b>] kthread+0xbb/0xc0<br>
[ 1514.868007] [<ffffffff818a0000>] ? e1000_regdump+0x262/0x3be<br>
[ 1514.868007] [<ffffffff8105e050>] ? kthread_create_on_node+0x130/0x130<br>
[ 1514.868007] [<ffffffff818accac>] ret_from_fork+0x7c/0xb0<br>
[ 1514.868007] [<ffffffff8105e050>] ? kthread_create_on_node+0x130/0x130<br>
[ 1514.868007] Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01<br>
c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 8b 87 78 03 00 00<br>
55 48 89 e5 <48> 8b 40 d8 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00<br>
00 00<br>
[ 1514.868007] RIP [<ffffffff8105e3cb>] kthread_data+0xb/0x20<br>
[ 1514.868007] RSP <ffff88001f949928><br>
[ 1514.868007] CR2: ffffffffffffffd8<br>
[ 1514.868007] ---[ end trace 36ffe9378168cdc3 ]---<br>
[ 1514.868007] Fixing recursive fault but reboot is needed!<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Thanks,<br>
//richard<br>
--<br>
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in<br>
the body of a message to <a href="mailto:majordomo@vger.kernel.org">majordomo@vger.kernel.org</a><br>
More majordomo info at <a href="http://vger.kernel.org/majordomo-info.html" target="_blank">http://vger.kernel.org/majordomo-info.html</a><br>
</font></span></blockquote></div><br></div>