<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 05/09/2013 11:59 PM, Guozhonghua
wrote:<br>
</div>
<blockquote
cite="mid:71604351584F6A4EBAE558C676F37CA417BD79B2@H3CMLB02-EX.srv.huawei-3com.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style>
<!--
@font-face
{font-family:宋体}
@font-face
{font-family:宋体}
@font-face
{font-family:Calibri}
@font-face
{font-family:"\@宋体"}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:#1F497D}
.MsoChpDefault
{}
@page WordSection1
{margin:72.0pt 90.0pt 72.0pt 90.0pt}
div.WordSection1
{}
-->
</style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">Thank you, but I have some
questions about it.
</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US"> </span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">The IP address of network used
by o2net is different with that used by o2hb, such as the
o2net use 192.168.0.7, but the storage network is
192.168.10.7.</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">So the tcp of o2net disconnected
while the o2hb is still living writing heartbeat on the
iSCSI LUNS.</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">Another condition may be that it
is the "deadline" io scheduler used, but the TCP message
may use CFS, is it may cause o2hb is OK, but the o2net
sometimes lose packets?</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US"> </span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">There is one scenario as below:</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">The Node 2013-SRV09(num 2) long
time without messages from ZHJD-VM6 (num 6), so it
disconnects the TCP with ZHJD-VM6 (num 6).</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">At the same time, the node
ZHJD-VM6 detected TCP disconnection with 2013-SRV09, but
ZHJD-VM6 does not reconnect 2013-SRV09, and so the ZHJD-VM6
is hanged.</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">At the same time the o2hb is
still OK, they did not evict each other, and there are other
six nodes in the ocfs2 cluster is still accessing the
storage.</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">The two node is hangs up does
not communicate with echo other but they can still access
the storage disk, and the issue continued more than about 1
hour, we reboot all the nodes in the cluster to solve the
issue.</span><br>
</p>
</div>
</blockquote>
You don't need to reboot all nodes. Just rebooting one node of the
broken connection should resolve the hang.<br>
<br>
Sunil has written the following scripts that will help you list
which nodes lost connection. You can find that from reviewing
messages files as well, but scripts are more convenient.<br>
<br>
<a class="moz-txt-link-freetext" href="https://oss.oracle.com/~smushran/debug/scripts/hb/">https://oss.oracle.com/~smushran/debug/scripts/hb/</a><br>
<br>
This piece of network code needs a little rework. First we need to
know what is causing nodes to loose connection. Can you please run
tcpdump on all nodes and forward me the logs when the problem
happens again.<br>
<br>
tcpdump -Z root -i $DEVICE -C 50 -W 10 -s 2500 -Sw /tmp/`hostname
-s`_tcpdump.log -ttt 'port 7777' &<br>
<br>
logs gets recycled, so you may have to stop the scripts immediately
after the problem.<br>
<blockquote
cite="mid:71604351584F6A4EBAE558C676F37CA417BD79B2@H3CMLB02-EX.srv.huawei-3com.com"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US"> </span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">The syslog digested of
2013-SRV09 is as below:
</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:08:34 2013-SRV09
kernel: [310434.984511] o2net: No longer connected to node
ZHJD-VM6 (num 6) at 185.200.1.17:7100</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:08:34 2013-SRV09
kernel: [310434.984558]
(libvirtd,3314,7):dlm_send_remote_convert_request:395 ERROR:
Error -112 when sending message 504 (key 0x77c0b1d1) to node
6</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">……………………………………………………………………………………………………….</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:08:34 2013-SRV09
kernel: [310434.984653]
(kvm,58972,29):dlm_send_remote_convert_request:395 ERROR:
Error -112 when sending message 504 (key 0x77c0b1d1) to node
6</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:08:34 2013-SRV09
kernel: [310434.984663] o2dlm: Waiting on the death of node
6 in domain AE16636E1B83497A88D6A50178172ECA</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">……………………………………………………………………………………………………….</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:08:39 2013-SRV09
kernel: [310440.077475]
(libvirtd,3314,2):dlm_send_remote_convert_request:395 ERROR:
Error -107 when sending message 504 (key 0x77c0b1d1) to node
6</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">……………………………………………………………………………………………………….</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 10:11:05 2013-SRV09
kernel: [314178.586741]
(kvm,58484,10):dlm_send_remote_convert_request:395 ERROR:
Error -107 when sending message 504 (key 0x77c0b1d1) to node
6</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 10:11:05 2013-SRV09
kernel: [314178.586768] o2dlm: Waiting on the death of node
6 in domain AE16636E1B83497A88D6A50178172ECA</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 10:11:05 2013-SRV09
kernel: [314178.638607]
(kvm,58972,11):dlm_send_remote_convert_request:395 ERROR:
Error -107 when sending message 504 (key 0x77c0b1d1) to node
6</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 10:11:05 2013-SRV09
kernel: [314178.638622] o2dlm: Waiting on the death of node
6 in domain AE16636E1B83497A88D6A50178172ECA</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US"> </span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">The syslog on node ZHJD-VM6:
</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:09:19 ZHJD-VM6 kernel:
[348569.574247] o2net: Connection to node 2013-SRV09 (num 2)
at 185.200.1.14:7100 shutdown, state 8</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:09:19 ZHJD-VM6 kernel:
[348569.574317] o2net: No longer connected to node
2013-SRV09 (num 2) at 185.200.1.14:7100</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:09:19 ZHJD-VM6 kernel:
[348569.574371]
(dlm_thread,4818,7):dlm_send_proxy_ast_msg:484 ERROR:
AE16636E1B83497A88D6A50178172ECA: res
M000000000000000d4a010600000000, error -112 send AST to node
2</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:09:19 ZHJD-VM6 kernel:
[348569.574388] (dlm_thread,4818,7):dlm_flush_asts:553
ERROR: status = -112</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:09:20 ZHJD-VM6 kernel:
[348569.605818]
(dlm_thread,4818,4):dlm_send_proxy_ast_msg:484 ERROR:
AE16636E1B83497A88D6A50178172ECA: res
M00000000000000246c010400000000, error -107 send AST to node
2</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 09:09:20 ZHJD-VM6 kernel:
[348569.605839] (dlm_thread,4818,4):dlm_flush_asts:553
ERROR: status = -107</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">………………………………………………………………………………………………………………….</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 10:12:30 ZHJD-VM6 kernel:
[352357.836983] o2net: No connection established with node 2
after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">May 4 10:13:00 ZHJD-VM6 kernel:
[352387.902370] o2net: No connection established with node 2
after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US"> </span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">If the condition satisfied, is
there some ways to avoid the hangs issues?
</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US"> </span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US">Thanks a lot.</span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US"> </span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-US"> </span></p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt;
padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt">发件人<span
lang="EN-US">:</span></span></b><span
style="font-size:10.0pt" lang="EN-US"> Sunil Mushran
[<a class="moz-txt-link-freetext" href="mailto:sunil.mushran@gmail.com">mailto:sunil.mushran@gmail.com</a>]
<br>
</span><b><span style="font-size:10.0pt">发送时间<span
lang="EN-US">:</span></span></b><span
style="font-size:10.0pt" lang="EN-US"> 2013</span><span
style="font-size:10.0pt">年<span lang="EN-US">5</span>月<span
lang="EN-US">10</span>日<span lang="EN-US"> 1:02<br>
</span><b>收件人<span lang="EN-US">:</span></b><span
lang="EN-US"> guozhonghua 02084<br>
</span><b>抄送<span lang="EN-US">:</span></b><span
lang="EN-US"> <a class="moz-txt-link-abbreviated" href="mailto:ocfs2-devel@oss.oracle.com">ocfs2-devel@oss.oracle.com</a>;
<a class="moz-txt-link-abbreviated" href="mailto:ocfs2-devel-request@oss.oracle.com">ocfs2-devel-request@oss.oracle.com</a>; changlimin 00148<br>
</span><b>主题<span lang="EN-US">:</span></b><span
lang="EN-US"> Re: [Ocfs2-devel] Patch request reviews,
for node reconnecting with other nodes whose node number
is little than local, thanks a lot.</span></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
lang="EN-US">A better fix is to _not_ disconnect on
o2net timeout once a connection has been<br>
cleanly established. Only disconnect on o2hb timeout.</span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
lang="EN-US">The reconnects are a problem as we could
lose packets and not be aware of it<br>
leading to o2dlm hangs.</span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">IOW, this patch
looks to be papering over one specific problem and does
not fix the<br>
underlying issue.</span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
lang="EN-US"> </span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, May 7, 2013
at 7:43 PM, Guozhonghua <<a moz-do-not-send="true"
href="mailto:guozhonghua@h3c.com" target="_blank">guozhonghua@h3c.com</a>>
wrote:</span></p>
<div>
<div>
<p class="MsoNormal" style=""><span lang="EN-US"> </span></p>
<p class="MsoNormal" style=""><span lang="EN-US">Hi,
everyone,</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">I had
have a test with eight nodes and find one issue.</span></p>
<p class="MsoNormal" style=""><span lang="EN-US"><br>
The Linux kernel version is 3.2.40.</span></p>
<p class="MsoNormal" style=""><span lang="EN-US"> </span></p>
<p class="MsoNormal" style=""><span lang="EN-US">As I
migrate processes from one node to another, those
processes is open the files on the OCFS2 storage.
Sometime one node shutdown TCP connection with that
node whose node number is larger because long time
without any message from it. </span></p>
<p class="MsoNormal" style=""><span lang="EN-US">As the
TCP connection shutdown, the node whose number
larger did not restart connection to the node, whose
number is little and shutdown the TCP connection.</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">So I
review the code of the cluster and find it may be a
bug.
</span></p>
<p class="MsoNormal" style=""><span lang="EN-US"> </span></p>
<p class="MsoNormal" style=""><span lang="EN-US">I
changed it and have a test. </span>
</p>
<p class="MsoNormal" style=""><span lang="EN-US"> </span></p>
<p class="MsoNormal" style=""><span lang="EN-US">Is
there anybody having time to view and make sure that
those changes is correct?</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">Thanks
a lot.</span></p>
<p class="MsoNormal" style=""><span lang="EN-US"> </span></p>
<p class="MsoNormal" style=""><span lang="EN-US">The
diff file is as below, of the file is
/cluster/tcp.c:
</span></p>
<p class="MsoNormal" style=""><span lang="EN-US"> </span></p>
<p class="MsoNormal" style=""><span lang="EN-US">root@gzh-dev:/home/dev/test_replace/ocfs2_ko#
diff -pu ocfs2-ko-3.2-compare/cluster/tcp.c
ocfs2-ko-3.2/cluster/tcp.c
</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">---
ocfs2-ko-3.2-compare/cluster/tcp.c 2012-10-29
19:33:19.534200000 +0800</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">+++
ocfs2-ko-3.2/cluster/tcp.c 2013-05-08
09:33:16.386277310 +0800</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">@@
-1699,6 +1698,10 @@ static void
o2net_start_connect(struct w</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">
if (ret == -EINPROGRESS)</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">
ret = 0;</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">+
/** Reset the timeout with 0 to avoid connection
again */</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">+
if (ret == 0) { </span>
</p>
<p class="MsoNormal" style=""><span lang="EN-US">+
atomic_set(&nn->nn_timeout, 0);
</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">+
}</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">out:</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">
if (ret) {</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">
printk(KERN_NOTICE "o2net: Connect attempt to "
SC_NODEF_FMT</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">@@
-1725,6 +1728,11 @@ static void
o2net_connect_expired(struct</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">
spin_lock(&nn->nn_lock);</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">
if (!nn->nn_sc_valid) {</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">+
/** trigger reconnect with other nodes whose node
number is little than local</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">+
* while they are still able to access the storage</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">+
*/</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">+
atomic_set(&nn->nn_timeout, 1);</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">+
</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">
printk(KERN_NOTICE "o2net: No connection established
with "</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">
"node %u after %u.%u seconds, giving up.\n",</span></p>
<p class="MsoNormal" style=""><span lang="EN-US">
o2net_num_from_nn(nn),</span></p>
</div>
</div>
</div>
</div>
</div>
<span style="font-size:7.5pt; font-family:华文细黑; color:gray"><span
lang="EN-US">-------------------------------------------------------------------------------------------------------------------------------------<br>
</span>本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出<span
lang="EN-US"><br>
</span>的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、<span
lang="EN-US"><br>
</span>或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本<span
lang="EN-US"><br>
</span>邮件！<span lang="EN-US"><br>
</span></span><span style="font-size:7.5pt;
font-family:"Arial","sans-serif";
color:gray" lang="EN-US">This e-mail and its attachments contain
confidential information from H3C, which is
<br>
intended only for the person or entity whose address is listed
above. Any use of the
<br>
information contained herein in any way (including, but not
limited to, total or partial
<br>
disclosure, reproduction, or dissemination) by persons other
than the intended <br>
recipient(s) is prohibited. If you receive this e-mail in error,
please notify the sender
<br>
by phone or email immediately and delete it!</span>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Ocfs2-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a>
<a class="moz-txt-link-freetext" href="https://oss.oracle.com/mailman/listinfo/ocfs2-users">https://oss.oracle.com/mailman/listinfo/ocfs2-users</a></pre>
</blockquote>
<br>
</body>
</html>