<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style>
<!--
@font-face
{font-family:\5B8B\4F53}
@font-face
{font-family:\5B8B\4F53}
@font-face
{font-family:Calibri}
@font-face
{font-family:"\@\5B8B\4F53"}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri","sans-serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
.MsoChpDefault
{}
@page WordSection1
{margin:72.0pt 90.0pt 72.0pt 90.0pt}
div.WordSection1
{}
-->
</style>
</head>
<body lang="ZH-CN" link="blue" vlink="purple" style="">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">The above error is not appeared again after 18:10:01.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">At the time about 18:58:00, the node 4, server14, is rebooted.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">On the Server16, the syslog is as below, after the 19:00 pm there are not any errors in syslog:
</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:21 Server16 kernel: [11656.038213] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 has been idle for 30.42 secs, shutting it down.</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:21 Server16 kernel: [11656.038253] o2net: No longer connected to node server14 (num 4) at 192.168.20.14:7100</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:39 Server16 kernel: [11674.053498] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:42 Server16 kernel: [11677.052047] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:45 Server16 kernel: [11680.050594] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:48 Server16 kernel: [11683.049138] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:51 Server16 kernel: [11686.047637] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:51 Server16 kernel: [11686.103617] o2net: No connection established with node 4 after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:54 Server16 kernel: [11688.791530] o2cb: o2dlm has evicted node 4 from domain 3B55DD8C17CA4A368C1195E040324DCE</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:54 Server16 kernel: [11689.382018] o2dlm: Begin recovery on domain 3B55DD8C17CA4A368C1195E040324DCE for node 4</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:54 Server16 kernel: [11689.388464] o2dlm: Node 9 (me) is the Recovery Master for the dead node 4 in domain 3B55DD8C17CA4A368C1195E040324DCE</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:01 Server16 CRON[19809]: (root) CMD ( /opt/bin/libvirtd_check.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:01 Server16 CRON[19808]: (root) CMD ( /opt/bin/ocfs2_iscsi_conf_chg_timer.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:25 Server16 kernel: [11720.007116] o2net: Connection to node Server19 (num 6) at 192.168.20.19:7100 has been idle for 30.107 secs, shutting it down.</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:25 Server16 kernel: [11720.007152] o2net: No longer connected to node Server19 (num 6) at 192.168.20.19:7100</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:25 Server16 kernel: [11720.007238] (libvirtd,3622,5):dlm_do_master_request:1332 ERROR: link to 6 went down!</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:25 Server16 kernel: [11720.024181] (libvirtd,3622,5):dlm_get_lock_resource:917 ERROR: status = -112</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:25 Server16 kernel: [11720.032794] (libvirtd,3622,5):dlm_restart_lock_mastery:1221 ERROR: node down! 4</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:25 Server16 kernel: [11720.049430] (libvirtd,3622,5):dlm_wait_for_lock_mastery:1038 ERROR: status = -11</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:43 Server16 kernel: [11738.022388] o2net: Connection to node Server19 (num 6) at 192.168.20.19:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:46 Server16 kernel: [11741.020966] o2net: Connection to node Server19 (num 6) at 192.168.20.19:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:49 Server16 kernel: [11744.019521] o2net: Connection to node Server19 (num 6) at 192.168.20.19:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:52 Server16 kernel: [11747.018067] o2net: Connection to node Server19 (num 6) at 192.168.20.19:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:55 Server16 kernel: [11750.016610] o2net: Connection to node Server19 (num 6) at 192.168.20.19:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:55 Server16 kernel: [11750.072553] o2net: No connection established with node 6 after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:59:56 Server16 kernel: [11750.885081] o2cb: o2dlm has evicted node 6 from domain 3B55DD8C17CA4A368C1195E040324DCE</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:00:01 Server16 CRON[19971]: (root) CMD ( /opt/bin/libvirtd_check.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:00:01 Server16 CRON[19972]: (root) CMD ( /opt/bin/ocfs2_iscsi_conf_chg_timer.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:00:01 Server16 CRON[19973]: (root) CMD ( /opt/bin/ocfs2_sync_task.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:00:01 Server16 CRON[19974]: (root) CMD ( /opt/bin/ha_cleanup.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:00:01 Server16 CRON[19975]: (root) CMD ( /opt/bin/ha_check_resource.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:01:01 Server16 CRON[21342]: (root) CMD ( /opt/bin/ocfs2_iscsi_conf_chg_timer.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:01:01 Server16 CRON[21341]: (root) CMD ( /opt/bin/libvirtd_check.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:02:01 Server16 CRON[21501]: (root) CMD ( /opt/bin/ocfs2_iscsi_conf_chg_timer.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:02:01 Server16 CRON[21503]: (root) CMD ( /opt/bin/libvirtd_check.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:02:01 Server16 CRON[21502]: (root) CMD ( /opt/bin/ha_check_resource.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:03:01 Server16 CRON[21841]: (root) CMD ( /opt/bin/libvirtd_check.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:03:01 Server16 CRON[21842]: (root) CMD ( /opt/bin/ocfs2_iscsi_conf_chg_timer.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:04:01 Server16 CRON[22162]: (root) CMD ( /opt/bin/ocfs2_iscsi_conf_chg_timer.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:04:01 Server16 CRON[22163]: (root) CMD ( /opt/bin/ha_check_resource.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:04:01 Server16 CRON[22164]: (root) CMD ( /opt/bin/libvirtd_check.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:04:44 Server16 kernel: [12039.321932] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:04:46 Server16 kernel: [12041.323643] o2net: Connected to node server14 (num 4) at 192.168.20.14:7100</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:05:01 Server16 CRON[22541]: (root) CMD ( /opt/bin/ocfs2_iscsi_conf_chg_timer.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:05:01 Server16 CRON[22544]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:05:01 Server16 CRON[22542]: (root) CMD ( /opt/bin/libvirtd_check.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:05:01 Server16 CRON[22543]: (root) CMD ( /opt/bin/ha_cleanup.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:05:13 Server16 kernel: [12067.530385] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 shutdown, state 8</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 19:05:13 Server16 kernel: [12067.530452] o2net: No longer connected to node server14 (num 4) at 192.168.20.14:7100</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">It is strange that there is “Begin recovery”, and Server16 does not has any “End recovery” log still.</span></p>
<p class="MsoNormal"><span lang="EN-US">In the OCFS2 code, there would be “End recovery… ” logs.</span></p>
<p class="MsoNormal"><span style="font-family:宋体">…</span><span lang="EN-US">. ……….. ….</span></p>
<p class="MsoNormal"><span lang="EN-US">master_here:</span></p>
<p class="MsoNormal"><span lang="EN-US"> dlm_print_recovery_master(dlm); // There log the message: “Node 9 (me) is the Recovery Master for the dead node 4 in domain</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> status = dlm_remaster_locks(dlm, dlm->reco.dead_node);</span></p>
<p class="MsoNormal"><span lang="EN-US"> if (status < 0) {</span></p>
<p class="MsoNormal"><span lang="EN-US"> /* we should never hit this anymore */</span></p>
<p class="MsoNormal"><span lang="EN-US"> mlog(ML_ERROR, "%s: Error %d remastering locks for node %u, "</span></p>
<p class="MsoNormal"><span lang="EN-US"> "retrying.\n", dlm->name, status, dlm->reco.dead_node);</span></p>
<p class="MsoNormal"><span lang="EN-US"> /* yield a bit to allow any final network messages</span></p>
<p class="MsoNormal"><span lang="EN-US"> * to get handled on remaining nodes */</span></p>
<p class="MsoNormal"><span lang="EN-US"> msleep(100);</span></p>
<p class="MsoNormal"><span lang="EN-US"> } else {</span></p>
<p class="MsoNormal"><span lang="EN-US"> /* success! see if any other nodes need recovery */</span></p>
<p class="MsoNormal"><span lang="EN-US"> mlog(0, "DONE mastering recovery of %s:%u here(this=%u)!\n",</span></p>
<p class="MsoNormal"><span lang="EN-US"> dlm->name, dlm->reco.dead_node, dlm->node_num);</span></p>
<p class="MsoNormal"><span lang="EN-US"> dlm_reset_recovery(dlm);</span></p>
<p class="MsoNormal"><span lang="EN-US"> }</span></p>
<p class="MsoNormal"><span lang="EN-US"> dlm_end_recovery(dlm); </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> /* continue and look for another dead node */</span></p>
<p class="MsoNormal"><span lang="EN-US"> return -EAGAIN;</span></p>
<p class="MsoNormal"><span lang="EN-US">} </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">It about 18:58:39, the node 4, server14 is rebooted.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">On the other nodes, such as Server13, Server15, Server17, and so on, the log message is repeated until all the OCFS2 umounted and the o2cb service stopped.</span></p>
<p class="MsoNormal"><span lang="EN-US">The error log information is as below: </span>
</p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:01 Server18 CRON[11925]: (root) CMD ( /opt/bin/libvirtd_check.sh)</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:04 Server18 kernel: [15320.745062] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:07 Server18 kernel: [15323.738643] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:10 Server18 kernel: [15326.732267] o2net: Connection to node server14 (num 4) at 192.168.20.14:7100 shutdown, state 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:10 Server18 kernel: [15326.788131] o2net: No connection established with node 4 after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:12 Server18 kernel: [15328.502721] (dlm_thread,4375,10):dlm_send_proxy_ast_msg:484 ERROR: 3B55DD8C17CA4A368C1195E040324DCE: res S000000000000000000000200000000, error -107 send AST to node 4</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:12 Server18 kernel: [15328.503552] (dlm_thread,4375,10):dlm_flush_asts:605 ERROR: status = -107</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:12 Server18 kernel: [15328.852857] o2cb: o2dlm has evicted node 4 from domain 3B55DD8C17CA4A368C1195E040324DCE</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:13 Server18 kernel: [15330.010501] o2cb: o2dlm has evicted node 4 from domain 3B55DD8C17CA4A368C1195E040324DCE</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:14 Server18 kernel: [15330.619912] o2dlm: Begin recovery on domain 3B55DD8C17CA4A368C1195E040324DCE for node 4</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:14 Server18 kernel: [15330.619918] o2dlm: Node 9 (he) is the Recovery Master for the dead node 4 in domain 3B55DD8C17CA4A368C1195E040324DCE</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:14 Server18 kernel: [15330.619923] o2dlm: End recovery on domain 3B55DD8C17CA4A368C1195E040324DCE</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:19 Server18 kernel: [15335.609250] o2dlm: Begin recovery on domain 3B55DD8C17CA4A368C1195E040324DCE for node 4</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:19 Server18 kernel: [15335.609258] o2dlm: Node 9 (he) is the Recovery Master for the dead node 4 in domain 3B55DD8C17CA4A368C1195E040324DCE</span></p>
<p class="MsoNormal"><span lang="EN-US">Dec 25 18:58:19 Server18 kernel: [15335.609262] o2dlm: End recovery on domain 3B55DD8C17CA4A368C1195E040324DCE</span></p>
<p class="MsoNormal"><span style="font-family:宋体">…</span><span lang="EN-US"> … … … …</span></p>
<p class="MsoNormal"><span style="font-family:宋体">………</span><span lang="EN-US">. Same logs repeated, until the server reboot….</span></p>
<p class="MsoNormal"><span style="font-family:宋体">…</span><span lang="EN-US"> … … … …</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">The mounted point on server13, server14, server15 cannot be list the files; so the ocfs2 cluster may be blocked.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">I delete all the cluster.conf from all the hosts, and restart them, remount the storage target again, and the ocfs2 cluster is OK, all the node can access the same target storage.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">The Server16 may be blocked at 18:02:01, and other nodes wait for it to take over all the resources had mastered by the reboot node 4, Server14.</span></p>
<p class="MsoNormal"><span lang="EN-US">So the ocfs2 cluster may be in a dead lock status after 19:00:00, so as to rebuild the ocfs2 cluster again to avoid it.
</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">I have more log message to help you to found the under reason.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Any good ideas?</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks, Have A Good holiday.</span></p>
</div>
<span style="font-size:7.5pt; font-family:华文细黑; color:gray"><span lang="EN-US">-------------------------------------------------------------------------------------------------------------------------------------<br>
</span>本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出<span lang="EN-US"><br>
</span>的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、<span lang="EN-US"><br>
</span>或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本<span lang="EN-US"><br>
</span>邮件！<span lang="EN-US"><br>
</span></span><span lang="EN-US" style="font-size:7.5pt; font-family:"Arial","sans-serif"; color:gray">This e-mail and its attachments contain confidential information from H3C, which is
<br>
intended only for the person or entity whose address is listed above. Any use of the
<br>
information contained herein in any way (including, but not limited to, total or partial
<br>
disclosure, reproduction, or dissemination) by persons other than the intended <br>
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
<br>
by phone or email immediately and delete it!</span>
</body>
</html>