<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style>
<!--
@font-face
{font-family:SimSun}
@font-face
{font-family:SimSun}
@font-face
{font-family:Calibri}
@font-face
{font-family:SimSun}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri","sans-serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
.MsoChpDefault
{}
@page WordSection1
{margin:72.0pt 90.0pt 72.0pt 90.0pt}
div.WordSection1
{}
-->
</style>
</head>
<body lang="ZH-CN" link="blue" vlink="purple" style="">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi, </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">One node A shutdown, another node B recovery It, but the node B panic and should re-powered to avoid it.</span></p>
<p class="MsoNormal"><span lang="EN-US">There are many soft lockup msg in the syslog.</span></p>
<p class="MsoNormal"><span lang="EN-US">The test environment is Ubuntu 12.04 with kernel 3.2.50.</span></p>
<p class="MsoNormal"><span lang="EN-US">The log digested is as below. </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:20 cvknode5 kernel: [ 5121.328969] o2net: No connection established with node 8 after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:20 cvknode5 kernel: [ 5121.328978] o2net_set_nn_state 578: Local and remote node is heartbeating, and try connect</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:20 cvknode5 kernel: [ 5121.994075] o2net_set_nn_state 581: Local or remote node is not heartbeating, not try connect</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:20 cvknode5 kernel: [ 5122.101757] o2cb: o2dlm has evicted node 8 from domain F803D818039043EBA0D0590A05E8076B</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:21 cvknode5 kernel: [ 5123.104931] o2dlm: Waiting on the recovery of node 8 in domain F803D818039043EBA0D0590A05E8076B</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:22 cvknode5 kernel: [ 5123.724936] o2dlm: Begin recovery on domain F803D818039043EBA0D0590A05E8076B for node 8</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:22 cvknode5 kernel: [ 5123.726360] o2dlm: Node 1 (me) is the Recovery Master for the dead node 8 in domain F803D818039043EBA0D0590A05E8076B</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:27 cvknode5 kernel: [ 5128.726079] o2dlm: End recovery on domain F803D818039043EBA0D0590A05E8076B</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.721118] BUG: soft lockup - CPU#11 stuck for 22s! [dlm_thread:17918]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760610] Modules linked in: ocfs2(O) quota_tree ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs joydev usbhid hid ip6table_filter
ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables drbd lru_cache 8021q garp stp vhost_net macvtap macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi openvswitch_mod(O)
nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath psmouse serio_raw sb_edac edac_core hpilo ioatdma dca video acpi_power_meter mac_hid lp parport hpsa bnx2x libcrc32c mdio [last unloaded: configfs]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760661] CPU 11
</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760662] Modules linked in: ocfs2(O) quota_tree ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs joydev usbhid hid ip6table_filter
ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables drbd lru_cache 8021q garp stp vhost_net macvtap macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi openvswitch_mod(O)
nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath psmouse serio_raw sb_edac edac_core hpilo ioatdma dca video acpi_power_meter mac_hid lp parport hpsa bnx2x libcrc32c mdio [last unloaded: configfs]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760691]
</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760693] Pid: 17918, comm: dlm_thread Tainted: G W O 3.2.50 #1 H3C FlexServer B390</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760697] RIP: 0010:[<ffffffff8103db15>] [<ffffffff8103db15>] __ticket_spin_lock+0x25/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760707] RSP: 0018:ffff8806055efd00 EFLAGS: 00000292</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760709] RAX: 000000000000f9f8 RBX: 0000000100126f06 RCX: 00000000000000ff</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760711] RDX: 000000000000056f RSI: ffff8806056ff9b8 RDI: ffff8806056ffa00</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760712] RBP: ffff8806055efd00 R08: ffff880615d66028 R09: ffff880615d65bc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760714] R10: ffff8806056ff980 R11: ffff8806056ffa30 R12: ffff8806055efc90</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760716] R13: 0000000000000282 R14: ffff880617dd44d0 R15: ffffffff810773d2</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760718] FS: 0000000000000000(0000) GS:ffff880c3f6a0000(0000) knlGS:0000000000000000</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760720] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760722] CR2: 00007f12275d83d0 CR3: 0000000001c05000 CR4: 00000000000426e0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760724] DR0: 0000000000000003 DR1: 00000000000000b0 DR2: 0000000000000001</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760726] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760728] Process dlm_thread (pid: 17918, threadinfo ffff8806055ee000, task ffff880615d65bc0)</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.760729] Stack:</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.798675] ffff8806055efd10 ffffffff81642fee ffff8806055efd70 ffffffffa0414476</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.798679] ffff8806055efd40 ffff8806055efd40 0000000000000286 ffff880608fc0a70</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.798683] ffff8806055efd50 ffffffff81642fee ffff880605fa8810 ffff880605fa8800</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.798687] Call Trace:</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835695] [<ffffffff81642fee>] _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835704] [<ffffffffa0414476>] __dlm_queue_bast+0x86/0x150 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835708] [<ffffffff81642fee>] ? _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835712] [<ffffffffa03ff6f7>] dlm_thread+0x897/0x1740 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835717] [<ffffffff8101b8a9>] ? sched_clock+0x9/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835722] [<ffffffff810908ad>] ? sched_clock_cpu+0xbd/0x110</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835728] [<ffffffff8108a6b0>] ? add_wait_queue+0x60/0x60</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835732] [<ffffffffa03fee60>] ? __dlm_dirty_lockres+0x130/0x130 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835736] [<ffffffff81089c0c>] kthread+0x8c/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835741] [<ffffffff8164d2f4>] kernel_thread_helper+0x4/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835744] [<ffffffff81089b80>] ? flush_kthread_worker+0xa0/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835747] [<ffffffff8164d2f0>] ? gs_change+0x13/0x13</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.835749] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00
8b 17 55 31 c0 48 89 e5 89 d1 </span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949781] Call Trace:</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949790] [<ffffffff81642fee>] _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949795] [<ffffffffa0414476>] __dlm_queue_bast+0x86/0x150 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949798] [<ffffffff81642fee>] ? _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949802] [<ffffffffa03ff6f7>] dlm_thread+0x897/0x1740 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949804] [<ffffffff8101b8a9>] ? sched_clock+0x9/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949807] [<ffffffff810908ad>] ? sched_clock_cpu+0xbd/0x110</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949810] [<ffffffff8108a6b0>] ? add_wait_queue+0x60/0x60</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949814] [<ffffffffa03fee60>] ? __dlm_dirty_lockres+0x130/0x130 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949817] [<ffffffff81089c0c>] kthread+0x8c/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949820] [<ffffffff8164d2f4>] kernel_thread_helper+0x4/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949823] [<ffffffff81089b80>] ? flush_kthread_worker+0xa0/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:54:52 cvknode5 kernel: [ 5153.949826] [<ffffffff8164d2f0>] ? gs_change+0x13/0x13</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.157117] BUG: soft lockup - CPU#6 stuck for 22s! [ocfs2rec:36720]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194491] Modules linked in: ocfs2(O) quota_tree ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs joydev usbhid hid ip6table_filter
ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables drbd lru_cache 8021q garp stp vhost_net macvtap macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi openvswitch_mod(O)
nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath psmouse serio_raw sb_edac edac_core hpilo ioatdma dca video acpi_power_meter mac_hid lp parport hpsa bnx2x libcrc32c mdio [last unloaded: configfs]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194527] CPU 6
</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194528] Modules linked in: ocfs2(O) quota_tree ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs joydev usbhid hid ip6table_filter
ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables drbd lru_cache 8021q garp stp vhost_net macvtap macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi openvswitch_mod(O)
nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath psmouse serio_raw sb_edac edac_core hpilo ioatdma dca video acpi_power_meter mac_hid lp parport hpsa bnx2x libcrc32c mdio [last unloaded: configfs]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194557]
</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194559] Pid: 36720, comm: ocfs2rec Tainted: G W O 3.2.50 #1 H3C FlexServer B390</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194562] RIP: 0010:[<ffffffff8103db15>] [<ffffffff8103db15>] __ticket_spin_lock+0x25/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194568] RSP: 0018:ffff880606fff9e0 EFLAGS: 00000297</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194569] RAX: 00000000000003c1 RBX: ffff880606fff980 RCX: 0000000000000000</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194571] RDX: 00000000000003c2 RSI: ffff880c16442f80 RDI: ffff880608fc0870</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194573] RBP: ffff880606fff9e0 R08: 0000000000000000 R09: ffff880c0b146440</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194575] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880606fff960</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194576] R13: ffff880608fc0800 R14: ffff880c0b146440 R15: ffff880c15004000</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194578] FS: 0000000000000000(0000) GS:ffff880c3f600000(0000) knlGS:0000000000000000</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194580] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194582] CR2: 00007fffba310dc8 CR3: 0000000001c05000 CR4: 00000000000426e0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194584] DR0: 0000000000000001 DR1: 0000000000000002 DR2: 0000000000000001</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194586] DR3: 000000000000000a DR6: 00000000ffff0ff0 DR7: 0000000000000400</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194588] Process ocfs2rec (pid: 36720, threadinfo ffff880606ffe000, task ffff880609b616f0)</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.194589] Stack:</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.230701] ffff880606fff9f0 ffffffff81642fee ffff880606fffa20 ffffffffa04143c5</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.230705] 0000000000000001 ffff880c0b146440 0000000000000001 ffff880c16442f80</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.230709] ffff880606fffa80 ffffffffa04174a4 ffff880606fffa80 ffffffffa0417a24</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.230713] Call Trace:</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266276] [<ffffffff81642fee>] _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266283] [<ffffffffa04143c5>] dlm_queue_ast+0x35/0x60 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266289] [<ffffffffa04174a4>] dlmlock_master+0x194/0x530 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266294] [<ffffffffa0417a24>] ? dlm_new_lock+0xd4/0x170 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266298] [<ffffffffa0418305>] dlmlock+0x845/0x1500 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266302] [<ffffffff810136e5>] ? __switch_to+0xf5/0x360</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266306] [<ffffffffa0118020>] ? o2dlm_lock_ast_wrapper+0x20/0x20 [ocfs2_stack_o2cb]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266309] [<ffffffff8101b319>] ? read_tsc+0x9/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266313] [<ffffffff81093e0d>] ? ktime_get_ts+0xad/0xe0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266316] [<ffffffff8103dba9>] ? default_spin_lock_flags+0x9/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266319] [<ffffffffa0118635>] o2cb_dlm_lock+0x65/0x7c [ocfs2_stack_o2cb]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266324] [<ffffffffa0118000>] ? 0xffffffffa0117fff</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266327] [<ffffffffa0118020>] ? o2dlm_lock_ast_wrapper+0x20/0x20 [ocfs2_stack_o2cb]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266332] [<ffffffffa01124d5>] ocfs2_dlm_lock+0x25/0x50 [ocfs2_stackglue]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266359] [<ffffffffa05ac01e>] __ocfs2_cluster_lock.isra.34+0x3de/0x810 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266379] [<ffffffffa060bc70>] ? ocfs2_set_buffer_uptodate+0x30/0x5b0 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266391] [<ffffffffa05bab57>] ? ocfs2_validate_inode_block+0x57/0x220 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266406] [<ffffffffa060b969>] ? ocfs2_metadata_cache_io_unlock+0x19/0x20 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266419] [<ffffffffa05ad64e>] ocfs2_inode_lock_full_nested+0x13e/0xa90 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266432] [<ffffffffa05bdc7b>] ? ocfs2_read_inode_block_full+0x3b/0x60 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266444] [<ffffffffa05c4d3d>] __ocfs2_recovery_thread+0x66d/0x1230 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266450] [<ffffffff8105472a>] ? finish_task_switch+0x4a/0xf0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266462] [<ffffffffa05c46d0>] ? ocfs2_check_journals_nolocks+0x150/0x150 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266465] [<ffffffff81089c0c>] kthread+0x8c/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266469] [<ffffffff8164d2f4>] kernel_thread_helper+0x4/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266472] [<ffffffff81089b80>] ? flush_kthread_worker+0xa0/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266475] [<ffffffff8164d2f0>] ? gs_change+0x13/0x13</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.266477] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00
8b 17 55 31 c0 48 89 e5 89 d1 </span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374744] Call Trace:</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374748] [<ffffffff81642fee>] _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374753] [<ffffffffa04143c5>] dlm_queue_ast+0x35/0x60 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374758] [<ffffffffa04174a4>] dlmlock_master+0x194/0x530 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374762] [<ffffffffa0417a24>] ? dlm_new_lock+0xd4/0x170 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374766] [<ffffffffa0418305>] dlmlock+0x845/0x1500 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374769] [<ffffffff810136e5>] ? __switch_to+0xf5/0x360</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374772] [<ffffffffa0118020>] ? o2dlm_lock_ast_wrapper+0x20/0x20 [ocfs2_stack_o2cb]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374774] [<ffffffff8101b319>] ? read_tsc+0x9/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374777] [<ffffffff81093e0d>] ? ktime_get_ts+0xad/0xe0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374780] [<ffffffff8103dba9>] ? default_spin_lock_flags+0x9/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374783] [<ffffffffa0118635>] o2cb_dlm_lock+0x65/0x7c [ocfs2_stack_o2cb]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374788] [<ffffffffa0118000>] ? 0xffffffffa0117fff</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374791] [<ffffffffa0118020>] ? o2dlm_lock_ast_wrapper+0x20/0x20 [ocfs2_stack_o2cb]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374794] [<ffffffffa01124d5>] ocfs2_dlm_lock+0x25/0x50 [ocfs2_stackglue]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374806] [<ffffffffa05ac01e>] __ocfs2_cluster_lock.isra.34+0x3de/0x810 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374822] [<ffffffffa060bc70>] ? ocfs2_set_buffer_uptodate+0x30/0x5b0 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374834] [<ffffffffa05bab57>] ? ocfs2_validate_inode_block+0x57/0x220 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374848] [<ffffffffa060b969>] ? ocfs2_metadata_cache_io_unlock+0x19/0x20 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374860] [<ffffffffa05ad64e>] ocfs2_inode_lock_full_nested+0x13e/0xa90 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374872] [<ffffffffa05bdc7b>] ? ocfs2_read_inode_block_full+0x3b/0x60 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374885] [<ffffffffa05c4d3d>] __ocfs2_recovery_thread+0x66d/0x1230 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374888] [<ffffffff8105472a>] ? finish_task_switch+0x4a/0xf0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374900] [<ffffffffa05c46d0>] ? ocfs2_check_journals_nolocks+0x150/0x150 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374903] [<ffffffff81089c0c>] kthread+0x8c/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374906] [<ffffffff8164d2f4>] kernel_thread_helper+0x4/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374909] [<ffffffff81089b80>] ? flush_kthread_worker+0xa0/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:00 cvknode5 kernel: [ 5161.374912] [<ffffffff8164d2f0>] ? gs_change+0x13/0x13</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:01 cvknode5 CRON[36939]: (root) CMD ( /opt/bin/libvirtd_check.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:01 cvknode5 CRON[36940]: (root) CMD ( /opt/bin/ocfs2_iscsi_conf_chg_timer.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:01 cvknode5 CRON[36941]: (root) CMD ( /opt/bin/ha_check_resource.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:01 cvknode5 CRON[36942]: (root) CMD ( /opt/bin/ha_cleanup.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Mar 27 00:55:01 cvknode5 CRON[36943]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)</span></p>
</div>
<span style="font-size:7.5pt; font-family:华文细黑; color:gray"><span lang="EN-US">-------------------------------------------------------------------------------------------------------------------------------------<br>
</span>本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出<span lang="EN-US"><br>
</span>的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、<span lang="EN-US"><br>
</span>或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本<span lang="EN-US"><br>
</span>邮件！<span lang="EN-US"><br>
</span></span><span lang="EN-US" style="font-size:7.5pt; font-family:"Arial","sans-serif"; color:gray">This e-mail and its attachments contain confidential information from H3C, which is
<br>
intended only for the person or entity whose address is listed above. Any use of the
<br>
information contained herein in any way (including, but not limited to, total or partial
<br>
disclosure, reproduction, or dissemination) by persons other than the intended <br>
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
<br>
by phone or email immediately and delete it!</span>
</body>
</html>