<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style>
<!--
@font-face
{font-family:SimSun}
@font-face
{font-family:SimSun}
@font-face
{font-family:Calibri}
@font-face
{font-family:SimSun}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
font-size:10.5pt;
font-family:"Calibri","sans-serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
.MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:72.0pt 90.0pt 72.0pt 90.0pt}
div.WordSection1
{}
-->
</style>
</head>
<body lang="ZH-CN" link="blue" vlink="purple" style="">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi, everyone</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">We setup 8 nodes to run ocfs2 as the storage pool providing storage service.</span></p>
<p class="MsoNormal"><span lang="EN-US">The test scenario is on the OS of Ubuntu with kernel version 3.13.6.</span></p>
<p class="MsoNormal"><span lang="EN-US">As the node 7 rebooted, all the other nodes blocked.
</span></p>
<p class="MsoNormal"><span lang="EN-US">The other nodes is racing to be the master of DLM_RECOVERY_LOCK_NAME($RECOVERY), but there is not any one will be successfully, and all of them still loop and wait.</span></p>
<p class="MsoNormal"><span lang="EN-US">So all the other node running unstopped loop, and print the log info as below.</span></p>
<p class="MsoNormal"><span lang="EN-US">The debug level is changed about on Apr 2 18:00:00, so the debug info is begin about 18:02:16.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:21 node-01 kernel: [64409.487556] o2cb: o2dlm has evicted node 7 from domain 2D2B1913CA08467896AC80B2F1AA80DB</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:26 node-01 kernel: [64414.350643] o2dlm: Begin recovery on domain 2D2B1913CA08467896AC80B2F1AA80DB for node 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:02:16 node-01 kernel: [73879.177060] (dlm_reco_thread,7871,3):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:02:16 node-01 kernel: [73879.177068] (dlm_reco_thread,7871,3):dlm_wait_for_lock_mastery:1046 map not changed and voting not done for 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:02:21 node-01 kernel: [73884.174312] (dlm_reco_thread,7871,4):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">…… same as above.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:09 node-02 kernel: [330500.807738] o2cb: o2dlm has evicted node 7 from domain 2D2B1913CA08467896AC80B2F1AA80DB</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:13 node-02 kernel: [330504.679296] o2dlm: Begin recovery on domain 2D2B1913CA08467896AC80B2F1AA80DB for node 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:57:38 node-02 kernel: [343302.676048] (dlm_reco_thread,39426,9):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:57:38 node-02 kernel: [343302.676055] (dlm_reco_thread,39426,9):dlm_wait_for_lock_mastery:1046 map not changed and voting not done for 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:57:43 node-02 kernel: [343307.673271] (dlm_reco_thread,39426,10):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">…… same as above.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:47 node-03 kernel: [105816.250867] o2cb: o2dlm has evicted node 7 from domain 2D2B1913CA08467896AC80B2F1AA80DB</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:49 node-03 kernel: [105818.291396] (dlm_thread,6493,8):dlm_send_proxy_ast_msg:482 ERROR: 2D2B1913CA08467896AC80B2F1AA80DB: res M00000000000000000002084cc0d288, error -107 send AST to node 4</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:50 node-03 kernel: [105819.679081] o2dlm: Begin recovery on domain 2D2B1913CA08467896AC80B2F1AA80DB for node 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:46 node-03 kernel: [118649.084903] (dlm_reco_thread,6494,9):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:46 node-03 kernel: [118649.084911] (dlm_reco_thread,6494,9):dlm_wait_for_lock_mastery:1046 map not changed and voting not done for 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:51 node-03 kernel: [118654.082626] (dlm_reco_thread,6494,9):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">…… same as above.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:50 node-04 kernel: [330501.154090] o2cb: o2dlm has evicted node 7 from domain 2D2B1913CA08467896AC80B2F1AA80DB</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:51 node-04 kernel: [330502.229376] o2dlm: Begin recovery on domain 2D2B1913CA08467896AC80B2F1AA80DB for node 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:59:09 node-04 kernel: [343353.348762] (dlm_reco_thread,38197,2):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:59:14 node-04 kernel: [343358.345980] (dlm_reco_thread,38197,2):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:59:19 node-04 kernel: [343363.343207] (dlm_reco_thread,38197,2):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:59:24 node-04 kernel: [343368.340485] (dlm_reco_thread,38197,3):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">…… same as above.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:18 node-05 kernel: [330489.928160] o2cb: o2dlm has evicted node 7 from domain 2D2B1913CA08467896AC80B2F1AA80DB</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:21 node-05 kernel: [330492.969521] o2dlm: Waiting on the recovery of node 7 in domain 2D2B1913CA08467896AC80B2F1AA80DB</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:24:22 node-05 kernel: [330494.168849] o2dlm: Begin recovery on domain 2D2B1913CA08467896AC80B2F1AA80DB for node 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:52 node-05 kernel: [343357.161638] (dlm_reco_thread,24064,6):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:52 node-05 kernel: [343357.161644] (dlm_reco_thread,24064,6):dlm_wait_for_lock_mastery:1046 map not changed and voting not done for 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:57 node-05 kernel: [343362.158919] (dlm_reco_thread,24064,7):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">…… same as above.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:23:24 node-06 kernel: [330529.804363] o2cb: o2dlm has evicted node 7 from domain 2D2B1913CA08467896AC80B2F1AA80DB</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:23:28 node-06 kernel: [330533.733262] o2dlm: Begin recovery on domain 2D2B1913CA08467896AC80B2F1AA80DB for node 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:08 node-06 kernel: [343408.025502] (dlm_reco_thread,28213,5):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:08 node-06 kernel: [343408.025509] (dlm_reco_thread,28213,5):dlm_wait_for_lock_mastery:1046 map not changed and voting not done for 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:13 node-06 kernel: [343413.023209] (dlm_reco_thread,28213,0):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again
</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:24:26 node-07 kernel: [ 4010.492579] (mount.ocfs2,11993,2):dlm_join_domain:1932 Timed out joining dlm domain 2D2B1913CA08467896AC80B2F1AA80DB after 91200 msecs</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:33:04 node-07 kernel: [ 4528.414250] (mount.ocfs2,31116,11):dlm_register_domain:2138 register called for domain "2D2B1913CA08467896AC80B2F1AA80DB"</span></p>
<p class="MsoNormal"><span lang="EN-US">... ... ... ... </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:33:04 node-07 kernel: [ 4528.415465] (mount.ocfs2,31116,11):sc_put:417 [sc ffff8817f8f98000 refs 4 sock ffff8817fd3b8f00 node 1 page ffffea005fc67240 pg_off 0] put</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:33:04 node-07 kernel: [ 4528.415469] (mount.ocfs2,31116,11):dlm_request_join:1522 status 0, node 1 response is 0 Here the node 1 disallow the node 7 to join the domain.</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:33:04 node-07 kernel: [ 4528.415471] (mount.ocfs2,31116,11):dlm_should_restart_join:1598 Latest response of disallow -- should restart</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:33:04 node-07 kernel: [ 4528.415474] (mount.ocfs2,31116,11):dlm_try_to_join_domain:1724 returning -11</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:33:04 node-07 kernel: [ 4528.415476] (mount.ocfs2,31116,11):dlm_join_domain:1946 backoff 600</span></p>
<p class="MsoNormal"><span lang="EN-US">... ... ... ... </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:33:28 node-07 kernel: [ 4551.869493] (mount.ocfs2,31116,2):dlm_join_domain:1932 Timed out joining dlm domain 2D2B1913CA08467896AC80B2F1AA80DB after 91200 msecs</span></p>
<p class="MsoNormal"><span lang="EN-US">... ... ... ... </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 19:00:26 node-07 kernel: [ 6168.900707] (mount.ocfs2,19177,6):dlm_ctxt_release:338 freeing memory from domain 2D2B1913CA08467896AC80B2F1AA80DB</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:23:42 node-08 kernel: [330502.270181] o2cb: o2dlm has evicted node 7 from domain 2D2B1913CA08467896AC80B2F1AA80DB</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 15:23:43 node-08 kernel: [330503.054257] o2dlm: Begin recovery on domain 2D2B1913CA08467896AC80B2F1AA80DB for node 7</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:48 node-08 kernel: [343401.119756] (dlm_reco_thread,61931,10):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:48 node-08 kernel: [343401.119763] (dlm_reco_thread,61931,10):dlm_wait_for_lock_mastery:1046 map not changed and voting not done for 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY</span></p>
<p class="MsoNormal"><span lang="EN-US">Apr 2 18:58:53 node-08 kernel: [343406.116991] (dlm_reco_thread,61931,11):dlm_wait_for_lock_mastery:1092 2D2B1913CA08467896AC80B2F1AA80DB:$RECOVERY: waiting again</span></p>
<p class="MsoNormal"><span lang="EN-US">…… same as above.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">As the code reviewed times, I think there is may be one bug in the function:
</span></p>
<p class="MsoNormal"><span lang="EN-US">static int dlm_wait_for_lock_mastery(struct dlm_ctxt *dlm,</span></p>
<p class="MsoNormal"><span lang="EN-US"> struct dlm_lock_resource *res,</span></p>
<p class="MsoNormal"><span lang="EN-US"> struct dlm_master_list_entry *mle,</span></p>
<p class="MsoNormal"><span lang="EN-US"> int *blocked)</span></p>
<p class="MsoNormal"><span lang="EN-US">{</span></p>
<p class="MsoNormal"><span lang="EN-US"> … … …</span></p>
<p class="MsoNormal" style="margin-left:21.0pt; text-indent:21.0pt"><span lang="EN-US">if (res->owner == O2NM_MAX_NODES) {</span></p>
<p class="MsoNormal"><span lang="EN-US"> mlog(0, "%s:%.*s: waiting again\n", dlm->name,</span></p>
<p class="MsoNormal"><span lang="EN-US"> res->lockname.len, res->lockname.name);</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> // if ther network failure and not any tcp msg received(tcp package lost), map not changed, and the master racing should be retriggered again.
</span></p>
<p class="MsoNormal"><span lang="EN-US">+ ret = -EAGAIN; </span></p>
<p class="MsoNormal"><span lang="EN-US">+ goto leave;</span></p>
<p class="MsoNormal"><span lang="EN-US">- goto recheck; </span>
</p>
<p class="MsoNormal"><span lang="EN-US"> }</span></p>
<p class="MsoNormal"><span lang="EN-US"> … … …</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">}</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
</div>
<span style="font-size:7.5pt; font-family:华文细黑; color:gray"><span lang="EN-US">-------------------------------------------------------------------------------------------------------------------------------------<br>
</span>本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出<span lang="EN-US"><br>
</span>的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、<span lang="EN-US"><br>
</span>或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本<span lang="EN-US"><br>
</span>邮件！<span lang="EN-US"><br>
</span></span><span lang="EN-US" style="font-size:7.5pt; font-family:"Arial","sans-serif"; color:gray">This e-mail and its attachments contain confidential information from H3C, which is
<br>
intended only for the person or entity whose address is listed above. Any use of the
<br>
information contained herein in any way (including, but not limited to, total or partial
<br>
disclosure, reproduction, or dissemination) by persons other than the intended <br>
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
<br>
by phone or email immediately and delete it!</span>
</body>
</html>