<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style>
<!--
@font-face
{font-family:SimSun}
@font-face
{font-family:SimSun}
@font-face
{font-family:Calibri}
@font-face
{font-family:SimSun}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri","sans-serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
.MsoChpDefault
{}
@page WordSection1
{margin:72.0pt 90.0pt 72.0pt 90.0pt}
div.WordSection1
{}
-->
</style>
</head>
<body lang="ZH-CN" link="blue" vlink="purple" style="">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">As we test the performance of ocfs2, and found that one node is always blocked.
</span></p>
<p class="MsoNormal"><span lang="EN-US">The ocfs2 module code we merged from 3.11 to 3.2.50.</span></p>
<p class="MsoNormal"><span lang="EN-US">My co-worker analysis the log and said that it may be the hard ware which lead to the resut.</span></p>
<p class="MsoNormal"><span lang="EN-US">Is there any other advices?</span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks a lot.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">root@cvknode7:~# numactl --hardware</span></p>
<p class="MsoNormal"><span lang="EN-US">available: 2 nodes (0-1)</span></p>
<p class="MsoNormal"><span lang="EN-US">node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31</span></p>
<p class="MsoNormal"><span lang="EN-US">node 0 size: 8157 MB</span></p>
<p class="MsoNormal"><span lang="EN-US">node 0 free: 6937 MB</span></p>
<p class="MsoNormal"><span lang="EN-US">node 1 cpus: 8 9 10 11 12 13 14 15</span></p>
<p class="MsoNormal"><span lang="EN-US">node 1 size: 8191 MB</span></p>
<p class="MsoNormal"><span lang="EN-US">node 1 free: 7436 MB</span></p>
<p class="MsoNormal"><span lang="EN-US">node distances:</span></p>
<p class="MsoNormal"><span lang="EN-US">node 0 1 </span></p>
<p class="MsoNormal"><span lang="EN-US"> 0: 10 21 </span></p>
<p class="MsoNormal"><span lang="EN-US"> 1: 21 10</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:01 cvknode7 CRON[35625]: (CRON) info (No MTA installed, discarding output)</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021452] ------------[ cut here ]------------</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021465] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9a/0xc0()</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021467] Hardware name: FlexServer B590</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021468] Watchdog detected hard LOCKUP on cpu 0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021470] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables ocfs2(O) quota_tree drbd lru_cache 8021q garp stp vhost_net macvtap
macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs openvswitch_mod(O) nfsd nfs
lockd fscache auth_rpcgss nfs_acl sunrpc psmouse dm_multipath serio_raw joydev sb_edac hpilo edac_core mac_hid ioatdma lp acpi_power_meter dca parport usbhid hid bnx2x libcrc32c mdio hpsa</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021527] Pid: 4880, comm: ocfs2dc Tainted: G W O 3.2.50 #1</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021529] Call Trace:</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021530] <NMI> [<ffffffff81066daf>] warn_slowpath_common+0x7f/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021543] [<ffffffff81066ea6>] warn_slowpath_fmt+0x46/0x50</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021547] [<ffffffff8101b833>] ? native_sched_clock+0x13/0x80</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021551] [<ffffffff810d6b1a>] watchdog_overflow_allback+0x9a/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021556] [<ffffffff8110eb76>] __perf_event_overflow+0x96/0x1f0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021559] [<ffffffff8110c491>] ? perf_event_update_userpage+0x11/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021564] [<ffffffff8102468a>] ? x86_perf_event_set_period+0xda/0x150</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021567] [<ffffffff8110f534>] perf_event_overflow+0x14/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021571] [<ffffffff81028c93>] intel_pmu_handle_irq+0x163/0x2e0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021581] [<ffffffff81644b01>] perf_event_nmi_handler+0x21/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021584] [<ffffffff816443d1>] do_nmi+0x101/0x350</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021587] [<ffffffff81643a30>] nmi+0x20/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021594] [<ffffffff8103db15>] ? __ticket_spin_lock+0x25/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021595] <<EOE>> <IRQ> [<ffffffff81642fee>] _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021620] [<ffffffffa0450487>] ocfs2_wake_downconvert_thread+0x27/0x60 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021632] [<ffffffffa0450554>] __ocfs2_cluster_unlock.isra.32+0x94/0xf0 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021643] [<ffffffffa0450b2b>] ocfs2_rw_unlock+0x6b/0xe0 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021648] [<ffffffff811aa22f>] ? bio_free+0x5f/0x70</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021658] [<ffffffffa043aa2a>] ocfs2_dio_end_io+0x6a/0x110 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021661] [<ffffffff811ad806>] dio_complete+0xe6/0xf0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021664] [<ffffffff811ad87d>] dio_bio_end_aio+0x6d/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021667] [<ffffffff816432d5>] ? _raw_spin_lock_irq+0x15/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021670] [<ffffffff811a8ecd>] bio_endio+0x1d/0x40</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021675] [<ffffffff812ebaf3>] req_bio_endio.isra.45+0xa3/0xe0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021678] [<ffffffff812ec23d>] blk_update_request+0xfd/0x480</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021681] [<ffffffff812ec5f1>] blk_update_bidi_request+0x31/0x90</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021684] [<ffffffff812ed8ec>] blk_end_bidi_request+0x2c/0x80</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021687] [<ffffffff812ed980>] blk_end_request+0x10/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021692] [<ffffffff814229bf>] scsi_io_completion+0xaf/0x630</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021698] [<ffffffff81418ebc>] scsi_finish_command+0xcc/0x130</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021701] [<ffffffff8142281e>] scsi_softirq_done+0x13e/0x150</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021705] [<ffffffff812f38b3>] blk_done_softirq+0x83/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021710] [<ffffffff8104f835>] ? check_preempt_curr+0x75/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021714] [<ffffffff8106e438>] __do_softirq+0xa8/0x210</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021716] [<ffffffff8104f89d>] ? ttwu_do_wakeup+0x3d/0x120</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021722] [<ffffffff8164d3ec>] call_softirq+0x1c/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021728] [<ffffffff81016205>] do_softirq+0x65/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021730] [<ffffffff8106e81e>] irq_exit+0x8e/0xb0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021734] [<ffffffff810313c5>] smp_call_function_single_interrupt+0x35/0x40</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021737] [<ffffffff8164ce5e>] call_function_single_interrupt+0x6e/0x80</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021739] <EOI> [<ffffffffa0282a72>] ? o2net_send_message_vec+0x142/0x9f0 [ocfs2_nodemanager]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021752] [<ffffffff8103dafd>] ? __ticket_spin_lock+0xd/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021755] [<ffffffff81642fee>] _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021766] [<ffffffffa045362f>] ocfs2_downconvert_thread+0x1af/0xc50 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021769] [<ffffffff810136e5>] ? __switch_to+0xf5/0x360</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021775] [<ffffffff8108a6b0>] ? add_wait_queue+0x60/0x60</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021786] [<ffffffffa0453480>] ? ocfs2_downconvert_lock+0x250/0x250 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021789] [<ffffffff81089c0c>] kthread+0x8c/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021792] [<ffffffff8164d2f4>] kernel_thread_helper+0x4/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021795] [<ffffffff81089b80>] ? flush_kthread_worker+0xa0/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021798] [<ffffffff8164d2f0>] ? gs_change+0x13/0x13</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13795.021800] ---[ end trace 54aca07eb6804380 ]---</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327684] ------------[ cut here ]------------</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327695] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9a/0xc0()</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327697] Hardware name: FlexServer B590</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327698] Watchdog detected hard LOCKUP on cpu 5</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327700] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables ocfs2(O) quota_tree drbd lru_cache 8021q garp stp vhost_net macvtap
macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs openvswitch_mod(O) nfsd nfs
lockd fscache auth_rpcgss nfs_acl sunrpc psmouse dm_multipath serio_raw joydev sb_edac hpilo edac_core mac_hid ioatdma lp acpi_power_meter dca parport usbhid hid bnx2x libcrc32c mdio hpsa</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327752] Pid: 7842, comm: kworker/u:2 Tainted: G W O 3.2.50 #1</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327753] Call Trace:</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327755] <NMI> [<ffffffff81066daf>] warn_slowpath_common+0x7f/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327764] [<ffffffff81066ea6>] warn_slowpath_fmt+0x46/0x50</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327768] [<ffffffff8101b833>] ? native_sched_clock+0x13/0x80</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327771] [<ffffffff810d6b1a>] watchdog_overflow_callback+0x9a/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327775] [<ffffffff8110eb76>] __perf_event_overflow+0x96/0x1f0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327778] [<ffffffff8110c491>] ? perf_event_update_userpage+0x11/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327782] [<ffffffff8102468a>] ? x86_perf_event_set_period+0xda/0x150</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327785] [<ffffffff8110f534>] perf_event_overflow+0x14/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327789] [<ffffffff81028c93>] intel_pmu_handle_irq+0x163/0x2e0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327795] [<ffffffff81644b01>] perf_event_nmi_handler+0x21/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327798] [<ffffffff816443d1>] do_nmi+0x101/0x350</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327801] [<ffffffff81643a30>] nmi+0x20/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327805] [<ffffffff8103db12>] ? __ticket_spin_lock+0x22/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327807] <<EOE>> [<ffffffff81642fee>] _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327834] [<ffffffffa044e034>] ocfs2_schedule_blocked_lock+0x84/0x130 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327847] [<ffffffffa045590b>] ocfs2_blocking_ast+0x24b/0x2b0 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327856] [<ffffffffa02b4fea>] ? __dlm_lookup_lockres_full+0xba/0x130 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327861] [<ffffffffa00e5020>] ? o2dlm_lock_ast_wrapper+0x20/0x20 [ocfs2_stack_o2cb]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327864] [<ffffffffa00e5034>] o2dlm_blocking_ast_wrapper+0x14/0x20 [ocfs2_stack_o2cb]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327872] [<ffffffffa02ce6eb>] dlm_do_local_bast+0x4b/0xe0 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327877] [<ffffffffa02b51f8>] ? dlm_lookup_lockres+0x88/0xa0 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327881] [<ffffffffa02cef86>] dlm_proxy_ast_handler+0x806/0xa10 [ocfs2_dlm]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327887] [<ffffffff81077e5c>] ? mod_timer+0x24c/0x2f0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327892] [<ffffffff810823fe>] ? queue_delayed_work_on+0xbe/0x1a0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327909] [<ffffffffa0280002>] ? o2net_handler_tree_lookup+0x22/0xc0 [ocfs2_nodemanager]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327915] [<ffffffffa0282036>] o2net_rx_until_empty+0x506/0xe00 [ocfs2_nodemanager]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327920] [<ffffffff810136e5>] ? __switch_to+0xf5/0x360</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327924] [<ffffffff810843c7>] process_one_work+0x127/0x470</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327927] [<ffffffff810854a4>] worker_thread+0x164/0x370</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327930] [<ffffffff81085340>] ? manage_workers.isra.31+0x230/0x230</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327934] [<ffffffff81089c0c>] kthread+0x8c/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327938] [<ffffffff8164d2f4>] kernel_thread_helper+0x4/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327941] [<ffffffff81089b80>] ? flush_kthread_worker+0xa0/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327944] [<ffffffff8164d2f0>] ? gs_change+0x13/0x13</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13804.327946] ---[ end trace 54aca07eb6804381 ]---</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13815.194830] o2net: Connection to node cvknode9 (num 5) at 192.168.3.9:7100 shutdown, state 8</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13815.199118] o2net: Connection to node cvknode9 (num 5) at 192.168.3.9:7100 has been idle for 30.76 secs, shutting it down.</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13815.199123] o2net_idle_timer 1598: Local and remote node is heartbeating, and try connect</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13815.234687] o2net: Connection to node cvknode4 (num 3) at 192.168.3.4:7100 shutdown, state 8</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13815.263054] o2net: Connection to node cvknode4 (num 3) at 192.168.3.4:7100 has been idle for 30.60 secs, shutting it down.</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:27 cvknode7 kernel: [13815.263061] o2net_idle_timer 1598: Local and remote node is heartbeating, and try connect</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:28 cvknode7 kernel: [13816.349993] o2net: Connection to node cvknode13 (num 2) at 192.168.3.13:7100 has been idle for 30.81 secs, shutting it down.</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:28 cvknode7 kernel: [13816.349998] o2net_idle_timer 1598: Local and remote node is heartbeating, and try connect</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419584] ------------[ cut here ]------------</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419599] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9a/0xc0()</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419601] Hardware name: FlexServer B590</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419603] Watchdog detected hard LOCKUP on cpu 27</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419604] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables ocfs2(O) quota_tree drbd lru_cache 8021q garp stp vhost_net macvtap
macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs openvswitch_mod(O) nfsd nfs
lockd fscache auth_rpcgss nfs_acl sunrpc psmouse dm_multipath serio_raw joydev sb_edac hpilo edac_core mac_hid ioatdma lp acpi_power_meter dca parport usbhid hid bnx2x libcrc32c mdio hpsa</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419670] Pid: 4308, comm: lichd Tainted: G W O 3.2.50 #1</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419672] Call Trace:</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419674] <NMI> [<ffffffff81066daf>] warn_slowpath_common+0x7f/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419688] [<ffffffff81066ea6>] warn_slowpath_fmt+0x46/0x50</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419693] [<ffffffff8101b833>] ? native_sched_clock+0x13/0x80</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419696] [<ffffffff810d6b1a>] watchdog_overflow_callback+0x9a/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419702] [<ffffffff8110eb76>] __perf_event_overflow+0x96/0x1f0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419706] [<ffffffff8110c491>] ? perf_event_update_userpage+0x11/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419712] [<ffffffff8102468a>] ? x86_perf_event_set_period+0xda/0x150</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419715] [<ffffffff8110f534>] perf_event_overflow+0x14/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419720] [<ffffffff81028c93>] intel_pmu_handle_irq+0x163/0x2e0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419731] [<ffffffff81644b01>] perf_event_nmi_handler+0x21/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419734] [<ffffffff816443d1>] do_nmi+0x101/0x350</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419738] [<ffffffff81643a30>] nmi+0x20/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419745] [<ffffffff8103db15>] ? __ticket_spin_lock+0x25/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419747] <<EOE>> <IRQ> [<ffffffff81642fee>] _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419809] [<ffffffffa0450487>] ocfs2_wake_downconvert_thread+0x27/0x60 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419820] [<ffffffffa0450554>] __ocfs2_cluster_unlock.isra.32+0x94/0xf0 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419832] [<ffffffffa0450b2b>] ocfs2_rw_unlock+0x6b/0xe0 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419838] [<ffffffff811aa22f>] ? bio_free+0x5f/0x70</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419848] [<ffffffffa043aa2a>] ocfs2_dio_end_io+0x6a/0x110 [ocfs2]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419852] [<ffffffff811ad806>] dio_complete+0xe6/0xf0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419855] [<ffffffff811ad87d>] dio_bio_end_aio+0x6d/0xc0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419859] [<ffffffff811a8ecd>] bio_endio+0x1d/0x40</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419866] [<ffffffff812ebaf3>] req_bio_endio.isra.45+0xa3/0xe0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419869] [<ffffffff812ec23d>] blk_update_request+0xfd/0x480</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419871] [<ffffffff812ec5f1>] blk_update_bidi_request+0x31/0x90</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419875] [<ffffffff812ed8ec>] blk_end_bidi_request+0x2c/0x80</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419878] [<ffffffff812ed980>] blk_end_request+0x10/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419886] [<ffffffff814229bf>] scsi_io_completion+0xaf/0x630</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419893] [<ffffffff81418ebc>] scsi_finish_command+0xcc/0x130</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419896] [<ffffffff8142281e>] scsi_softirq_done+0x13e/0x150</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419901] [<ffffffff812f38b3>] blk_done_softirq+0x83/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419906] [<ffffffff8106e438>] __do_softirq+0xa8/0x210</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419911] [<ffffffff8164d3ec>] call_softirq+0x1c/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419912] <EOI> [<ffffffff81016205>] do_softirq+0x65/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419922] [<ffffffff8106daa4>] local_bh_enable+0x94/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419930] [<ffffffff815393d1>] dev_queue_xmit+0x1c1/0x650</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419938] [<ffffffff815702ab>] ip_finish_output+0x16b/0x2f0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419941] [<ffffffff81570df8>] ip_output+0x98/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419944] [<ffffffff815704d4>] ? __ip_local_out+0xa4/0xb0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419946] [<ffffffff81570509>] ip_local_out+0x29/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419949] [<ffffffff81570661>] ip_queue_xmit+0x151/0x3f0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419953] [<ffffffff81587af0>] tcp_transmit_skb+0x400/0x8d0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419956] [<ffffffff8158860f>] tcp_write_xmit+0xff/0xa70</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419960] [<ffffffff8158b260>] ? tcp_v4_md5_lookup+0x10/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419962] [<ffffffff81588fe6>] __tcp_push_pending_frames+0x26/0x90</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419967] [<ffffffff8157b866>] do_tcp_setsockopt.isra.31+0x726/0x7b0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419974] [<ffffffff810908ad>] ? sched_clock_cpu+0xbd/0x110</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419977] [<ffffffff8152d986>] ? verify_iovec+0x56/0xd0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419983] [<ffffffff81520f56>] ? ___sys_sendmsg+0x396/0x3b0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419987] [<ffffffff8151ecca>] ? sock_poll+0x1a/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419994] [<ffffffff811b76d5>] ? ep_send_events_proc+0x65/0x110</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.419997] [<ffffffff8103dba9>] ? default_spin_lock_flags+0x9/0x10</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.420000] [<ffffffff811b8063>] ? ep_scan_ready_list.isra.7+0x183/0x190</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.420004] [<ffffffff811b8c73>] ? sys_epoll_wait+0x93/0x440</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.420007] [<ffffffff8157b953>] tcp_setsockopt+0x33/0x40</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.420010] [<ffffffff815241c4>] sock_common_setsockopt+0x14/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.420013] [<ffffffff8152319f>] sys_setsockopt+0x7f/0xe0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.420017] [<ffffffff8164b182>] system_call_fastpath+0x16/0x1b</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13816.420019] ---[ end trace 54aca07eb6804382 ]---</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221256] BUG: soft lockup - CPU#19 stuck for 23s! [lichd:4335]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221264] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables ocfs2(O) quota_tree drbd lru_cache 8021q garp stp vhost_net macvtap
macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs openvswitch_mod(O) nfsd nfs
lockd fscache auth_rpcgss nfs_acl sunrpc psmouse dm_multipath serio_raw joydev sb_edac hpilo edac_core mac_hid ioatdma lp acpi_power_meter dca parport usbhid hid bnx2x libcrc32c mdio hpsa</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221342] CPU 19
</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221343] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables ocfs2(O) quota_tree drbd lru_cache 8021q garp stp vhost_net macvtap
macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs openvswitch_mod(O) nfsd nfs
lockd fscache auth_rpcgss nfs_acl sunrpc psmouse dm_multipath serio_raw joydev sb_edac hpilo edac_core mac_hid ioatdma lp acpi_power_meter dca parport usbhid hid bnx2x libcrc32c mdio hpsa</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221370]
</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221373] Pid: 4335, comm: lichd Tainted: G W O 3.2.50 #1 H3C FlexServer B590</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221376] RIP: 0010:[<ffffffff810458a2>] [<ffffffff810458a2>] native_flush_tlb_others+0x122/0x140</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221389] RSP: 0018:ffff880222ef3d68 EFLAGS: 00000246</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221391] RAX: 0000000000000000 RBX: dead000000100100 RCX: 0000000000000004</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221393] RDX: 0000000000000019 RSI: 0000000000000100 RDI: ffffffff81e08d18</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221394] RBP: ffff880222ef3da8 R08: ffff88023756db60 R09: 0000000000000100</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221396] R10: dead000000100100 R11: dead000000200200 R12: ffff88023760db60</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221398] R13: ffff880222ef3da8 R14: ffffffff813067a6 R15: ffff880222ef3cd8</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221400] FS: 00007fa82ffff700(0000) GS:ffff880237560000(0000) knlGS:0000000000000000</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221401] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221403] CR2: 00007fa88863b000 CR3: 0000000430bf8000 CR4: 00000000000406e0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221407] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221409] Process lichd (pid: 4335, threadinfo ffff880222ef2000, task ffff88022eacade0)</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221410] Stack:</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221413] ffffffffffffffff ffff88022f739f80 00007fa888800000 ffff88022f739f80</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221420] ffff88022f73a240 ffff88043115a160 ffff8802308a6210 00007fa88863c000</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221423] ffff880222ef3dc8 ffffffff8104598b ffff88043115a0b0 ffff880222ef3e40</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221428] Call Trace:</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221434] [<ffffffff8104598b>] flush_tlb_mm+0x5b/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221439] [<ffffffff811380d6>] tlb_flush_mmu+0x46/0x90</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221442] [<ffffffff81138134>] tlb_finish_mmu+0x14/0x40</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221447] [<ffffffff8113f6ad>] unmap_region+0x10d/0x130</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221454] [<ffffffff8118a6fc>] ? d_free+0x4c/0x70</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221466] [<ffffffff81642fee>] ? _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221471] [<ffffffff8119214e>] ? vfsmount_lock_local_unlock+0x1e/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221475] [<ffffffff81193b80>] ? mntput_no_expire+0x30/0xf0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221478] [<ffffffff81140968>] do_munmap+0x318/0x3a0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221481] [<ffffffff81141cd3>] sys_munmap+0x53/0x80</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221488] [<ffffffff8164b182>] system_call_fastpath+0x16/0x1b</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221490] Code: 2e 0f 1f 84 00 00 00 00 00 48 8b 05 11 89 c9 00 41 8d b6 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 <be> 00 01 00 00 4c 89 e7 e8 b1 fc
2c 00 85 c0 74 ed e9 59 ff ff </span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221521] Call Trace:</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221523] [<ffffffff8104598b>] flush_tlb_mm+0x5b/0xa0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221525] [<ffffffff811380d6>] tlb_flush_mmu+0x46/0x90</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221528] [<ffffffff81138134>] tlb_finish_mmu+0x14/0x40</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221530] [<ffffffff8113f6ad>] unmap_region+0x10d/0x130</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221533] [<ffffffff8118a6fc>] ? d_free+0x4c/0x70</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221536] [<ffffffff81642fee>] ? _raw_spin_lock+0xe/0x20</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221539] [<ffffffff8119214e>] ? vfsmount_lock_local_unlock+0x1e/0x30</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221542] [<ffffffff81193b80>] ? mntput_no_expire+0x30/0xf0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221545] [<ffffffff81140968>] do_munmap+0x318/0x3a0</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221547] [<ffffffff81141cd3>] sys_munmap+0x53/0x80</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:29 cvknode7 kernel: [13817.221550] [<ffffffff8164b182>] system_call_fastpath+0x16/0x1b</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193836] BUG: soft lockup - CPU#19 stuck for 22s! [lichd:4335]</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193852] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables ocfs2(O) quota_tree drbd lru_cache 8021q garp stp vhost_net macvtap
macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs openvswitch_mod(O) nfsd nfs
lockd fscache auth_rpcgss nfs_acl sunrpc psmouse dm_multipath serio_raw joydev sb_edac hpilo edac_core mac_hid ioatdma lp acpi_power_meter dca parport usbhid hid bnx2x libcrc32c mdio hpsa</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193895] CPU 19
</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193896] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables ocfs2(O) quota_tree drbd lru_cache 8021q garp stp vhost_net macvtap
macvlan kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ocfs2_dlmfs(O) ocfs2_stack_o2cb(O) ocfs2_dlm(O) ocfs2_nodemanager(O) ocfs2_stackglue(O) configfs openvswitch_mod(O) nfsd nfs
lockd fscache auth_rpcgss nfs_acl sunrpc psmouse dm_multipath serio_raw joydev sb_edac hpilo edac_core mac_hid ioatdma lp acpi_power_meter dca parport usbhid hid bnx2x libcrc32c mdio hpsa</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193934]
</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193936] Pid: 4335, comm: lichd Tainted: G W O 3.2.50 #1 H3C FlexServer B590</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193939] RIP: 0010:[<ffffffff81315560>] [<ffffffff81315560>] kasprintf+0x40/0x40</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193947] RSP: 0018:ffff880222ef3d60 EFLAGS: 00000246</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193948] RAX: 0000000000000000 RBX: dead000000100100 RCX: 0000000000000004</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193950] RDX: 0000000000000019 RSI: 0000000000000100 RDI: ffffffff81e08d18</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193951] RBP: ffff880222ef3da8 R08: ffff88023756db60 R09: 0000000000000100</span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 26 19:16:57 cvknode7 kernel: [13845.193953] R10: dead000000100100 R11: dead000000200200 R12: ffff88023760db60</span></p>
</div>
<span style="font-size:7.5pt; font-family:华文细黑; color:gray"><span lang="EN-US">-------------------------------------------------------------------------------------------------------------------------------------<br>
</span>本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出<span lang="EN-US"><br>
</span>的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、<span lang="EN-US"><br>
</span>或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本<span lang="EN-US"><br>
</span>邮件！<span lang="EN-US"><br>
</span></span><span lang="EN-US" style="font-size:7.5pt; font-family:"Arial","sans-serif"; color:gray">This e-mail and its attachments contain confidential information from H3C, which is
<br>
intended only for the person or entity whose address is listed above. Any use of the
<br>
information contained herein in any way (including, but not limited to, total or partial
<br>
disclosure, reproduction, or dissemination) by persons other than the intended <br>
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
<br>
by phone or email immediately and delete it!</span>
</body>
</html>