<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style>
<!--
@font-face
{font-family:SimSun}
@font-face
{font-family:SimSun}
@font-face
{font-family:Calibri}
@font-face
{font-family:SimSun}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri","sans-serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
.MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:72.0pt 90.0pt 72.0pt 90.0pt}
div.WordSection1
{}
-->
</style>
</head>
<body lang="ZH-CN" link="blue" vlink="purple" style="">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi, everyone, is that an issue?</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">The Server version is Linux 3.2.0-23, Ubuntu 1204.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">There are 4 nodes in the OCFS2 Cluster, using three iSCSI LUNS, and every LUN is one OCFS2 domain mounted by thread node.
</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">As the network used buy node has one down/up, the tcp connection between node shutdown and reconnected with each other.</span></p>
<p class="MsoNormal"><span lang="EN-US">But there is one scenario that the node whose node number is little, shut down the tcp with node whose number is large, the node with large node number will not reconnect the node with little node number.
</span></p>
<p class="MsoNormal"><span lang="EN-US">The otherwise is that if the node with large node number shut down the tcp with node with little number, the node with large node number will reconnect the node with little node number OK.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Such as below: </span></p>
<p class="MsoNormal"><span lang="EN-US">The server1 syslog is as below: </span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:10 server1 kernel: [5199872.576027] o2net: Connection to node server2 (num 2) at 192.168.70.20:7100 shutdown, state 8</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:10 server1 kernel: [5199872.576111] o2net: No longer connected to node server2 (num 2) at 192.168.70.20:7100</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:10 server1 kernel: [5199872.576149] (ocfs2dc,14358,1):dlm_send_remote_convert_request:395 ERROR: Error -107 when sending message 504 (key 0x3671059b) to node 2</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:10 server1 kernel: [5199872.576162] o2dlm: Waiting on the death of node 2 in domain 3656D53908DC4149983BDB1DBBDF1291</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:10 server1 kernel: [5199872.576428] o2net: Accepted connection from node server2 (num 2) at 192.168.70.20:7100</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:11 server1 kernel: [5199872.995898] o2net: Connection to node server3 (num 3) at 192.168.70.30:7100 has been idle for 30.100 secs, shutting it down.</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:11 server1 kernel: [5199872.995987] o2net: No longer connected to node server3 (num 3) at 192.168.70.30:7100</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:11 server1 kernel: [5199873.069666] o2net: Connection to node server4 (num 4) at 192.168.70.40:7100 shutdown, state 8</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:11 server1 kernel: [5199873.069700] o2net: No longer connected to node server4 (num 4) at 192.168.70.40:7100</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:11 server1 kernel: [5199873.070385] o2net: Accepted connection from node server4 (num 4) at 192.168.70.40:7100</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">The server1 shutdown the tcp connection with server3, but server3 never reconnect with server1.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">The server3 syslog is as below: </span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:44:12 server3 kernel: [3971907.332698] o2net: Connection to node server1 (num 1) at 192.168.70.10:7100 shutdown, state 8</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:44:12 server3 kernel: [3971907.332748] o2net: No longer connected to node server1 (num 1) at 192.168.70.10:7100</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:44:42 server3 kernel: [3971937.355419] o2net: No connection established with node 1 after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:45:01 server3 CRON[52349]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:45:12 server3 kernel: [3971967.421656] o2net: No connection established with node 1 after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:45:42 server3 kernel: [3971997.487949] o2net: No connection established with node 1 after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:12 server3 kernel: [3972027.554258] o2net: No connection established with node 1 after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span lang="EN-US">Jul 9 17:46:42 server3 kernel: [3972057.620496] o2net: No connection established with node 1 after 30.0 seconds, giving up.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">The node of server2 and server4 shut down the connection with server1, and reconnect them ok.
</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">I review the code of the ocfs2 kernel and found this may be an issue or bug.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">As node of server1 did not receive msg from server3, he shut the connection with server3 and set the timeout with 1.</span></p>
<p class="MsoNormal"><span lang="EN-US">The server1’s node number is little than server3, so he wait the connect request from server3.</span></p>
<p class="MsoNormal"><span lang="EN-US">static void o2net_idle_timer(unsigned long data)</span></p>
<p class="MsoNormal"><span lang="EN-US">{</span></p>
<p class="MsoNormal"><span lang="EN-US"> … …</span></p>
<p class="MsoNormal"><span lang="EN-US"> printk(KERN_NOTICE "o2net: Connection to " SC_NODEF_FMT " has been "</span></p>
<p class="MsoNormal"><span lang="EN-US"> "idle for %lu.%lu secs, shutting it down.\n", SC_NODEF_ARGS(sc),</span></p>
<p class="MsoNormal"><span lang="EN-US"> msecs / 1000, msecs % 1000);</span></p>
<p class="MsoNormal"><span lang="EN-US"> ….. … </span></p>
<p class="MsoNormal"><span lang="EN-US"> atomic_set(&nn->nn_timeout, 1);</span></p>
<p class="MsoNormal"><span lang="EN-US"> o2net_sc_queue_work(sc, &sc->sc_shutdown_work);</span></p>
<p class="MsoNormal"><span lang="EN-US">}</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">But the server3 monitoring the TCP connection state changed and shutdown connect again and it will never reconnect with server1 because the nn->nn_timeout is 0.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">static void o2net_state_change(struct sock *sk)</span></p>
<p class="MsoNormal"><span lang="EN-US">{</span></p>
<p class="MsoNormal" style="margin-left:10.5pt"><span lang="EN-US">……</span></p>
<p class="MsoNormal"><span lang="EN-US"> switch(sk->sk_state) {</span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US"> ……</span></p>
<p class="MsoNormal"><span lang="EN-US"> default:</span></p>
<p class="MsoNormal"><span lang="EN-US"> printk(KERN_INFO "AAAAA o2net: Connection to " SC_NODEF_FMT</span></p>
<p class="MsoNormal"><span lang="EN-US"> " shutdown, state %d\n",</span></p>
<p class="MsoNormal"><span lang="EN-US"> SC_NODEF_ARGS(sc), sk->sk_state);</span></p>
<p class="MsoNormal"><span lang="EN-US"> o2net_sc_queue_work(sc, &sc->sc_shutdown_work);</span></p>
<p class="MsoNormal"><span lang="EN-US"> break;</span></p>
<p class="MsoNormal"><span lang="EN-US"> }</span></p>
<p class="MsoNormal" style="text-indent:21.0pt"><span lang="EN-US">… …</span></p>
<p class="MsoNormal"><span lang="EN-US">}</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">I had test the TCP connect without any shutdown between nodes, but send message will failed because the connection state is error.
</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">I change the code for the connect triggers in function o2net_set_nn_state and o2net_start_connect, and the reconnect rebuild up OK.</span></p>
<p class="MsoNormal"><span lang="EN-US">Is anyone review the code correct? Thanks a lots.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">root@gzh-dev:~/ocfs2# diff -p -C 10 ./ocfs2_org/cluster/tcp.c ocfs2_rep/cluster/tcp.c
</span></p>
<p class="MsoNormal"><span lang="EN-US">*** ./ocfs2_org/cluster/tcp.c 2012-10-29 19:33:19.534200000 +0800</span></p>
<p class="MsoNormal"><span lang="EN-US">--- ocfs2_rep/cluster/tcp.c 2013-07-16 16:58:31.380452531 +0800</span></p>
<p class="MsoNormal"><span lang="EN-US">*************** static void o2net_set_nn_state(struct o2</span></p>
<p class="MsoNormal"><span lang="EN-US">*** 567,586 ****</span></p>
<p class="MsoNormal"><span lang="EN-US">--- 567,590 ----</span></p>
<p class="MsoNormal"><span lang="EN-US"> if (!valid && o2net_wq) {</span></p>
<p class="MsoNormal"><span lang="EN-US"> unsigned long delay;</span></p>
<p class="MsoNormal"><span lang="EN-US"> /* delay if we're within a RECONNECT_DELAY of the</span></p>
<p class="MsoNormal"><span lang="EN-US"> * last attempt */</span></p>
<p class="MsoNormal"><span lang="EN-US"> delay = (nn->nn_last_connect_attempt +</span></p>
<p class="MsoNormal"><span lang="EN-US"> msecs_to_jiffies(o2net_reconnect_delay()))</span></p>
<p class="MsoNormal"><span lang="EN-US"> - jiffies;</span></p>
<p class="MsoNormal"><span lang="EN-US"> if (delay > msecs_to_jiffies(o2net_reconnect_delay()))</span></p>
<p class="MsoNormal"><span lang="EN-US"> delay = 0;</span></p>
<p class="MsoNormal"><span lang="EN-US"> mlog(ML_CONN, "queueing conn attempt in %lu jiffies\n", delay);</span></p>
<p class="MsoNormal"><span lang="EN-US">+ </span></p>
<p class="MsoNormal"><span lang="EN-US">+ /** Trigger the reconnection */</span></p>
<p class="MsoNormal"><span lang="EN-US">+ atomic_set(&nn->nn_timeout, 1);
</span></p>
<p class="MsoNormal"><span lang="EN-US">+ </span></p>
<p class="MsoNormal"><span lang="EN-US"> queue_delayed_work(o2net_wq, &nn->nn_connect_work, delay);</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> /*</span></p>
<p class="MsoNormal"><span lang="EN-US"> * Delay the expired work after idle timeout.</span></p>
<p class="MsoNormal"><span lang="EN-US"> *</span></p>
<p class="MsoNormal"><span lang="EN-US"> * We might have lots of failed connection attempts that run</span></p>
<p class="MsoNormal"><span lang="EN-US"> * through here but we only cancel the connect_expired work when</span></p>
<p class="MsoNormal"><span lang="EN-US"> * a connection attempt succeeds. So only the first enqueue of</span></p>
<p class="MsoNormal"><span lang="EN-US"> * the connect_expired work will do anything. The rest will see</span></p>
<p class="MsoNormal"><span lang="EN-US"> * that it's already queued and do nothing.</span></p>
<p class="MsoNormal"><span lang="EN-US">*************** static void o2net_start_connect(struct w</span></p>
<p class="MsoNormal"><span lang="EN-US">*** 1691,1710 ****</span></p>
<p class="MsoNormal"><span lang="EN-US">--- 1695,1719 ----</span></p>
<p class="MsoNormal"><span lang="EN-US"> remoteaddr.sin_family = AF_INET;</span></p>
<p class="MsoNormal"><span lang="EN-US"> remoteaddr.sin_addr.s_addr = node->nd_ipv4_address;</span></p>
<p class="MsoNormal"><span lang="EN-US"> remoteaddr.sin_port = node->nd_ipv4_port;</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> ret = sc->sc_sock->ops->connect(sc->sc_sock,</span></p>
<p class="MsoNormal"><span lang="EN-US"> (struct sockaddr *)&remoteaddr,</span></p>
<p class="MsoNormal"><span lang="EN-US"> sizeof(remoteaddr),</span></p>
<p class="MsoNormal"><span lang="EN-US"> O_NONBLOCK);</span></p>
<p class="MsoNormal"><span lang="EN-US"> if (ret == -EINPROGRESS)</span></p>
<p class="MsoNormal"><span lang="EN-US"> ret = 0;</span></p>
<p class="MsoNormal"><span lang="EN-US">+ </span></p>
<p class="MsoNormal"><span lang="EN-US">+ /** Reset the timeout with 0 to avoid connection again, Just for test the tcp connection */</span></p>
<p class="MsoNormal"><span lang="EN-US">+ if (ret == 0) {</span></p>
<p class="MsoNormal"><span lang="EN-US">+ atomic_set(&nn->nn_timeout, 0);
</span></p>
<p class="MsoNormal"><span lang="EN-US">+ }</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> out:</span></p>
<p class="MsoNormal"><span lang="EN-US"> if (ret) {</span></p>
<p class="MsoNormal"><span lang="EN-US"> printk(KERN_NOTICE "o2net: Connect attempt to " SC_NODEF_FMT</span></p>
<p class="MsoNormal"><span lang="EN-US"> " failed with errno %d\n", SC_NODEF_ARGS(sc), ret);</span></p>
<p class="MsoNormal"><span lang="EN-US"> /* 0 err so that another will be queued and attempted</span></p>
<p class="MsoNormal"><span lang="EN-US"> * from set_nn_state */</span></p>
<p class="MsoNormal"><span lang="EN-US"> if (sc)</span></p>
<p class="MsoNormal"><span lang="EN-US"> o2net_ensure_shutdown(nn, sc, 0);</span></p>
<p class="MsoNormal"><span lang="EN-US"> }</span></p>
</div>
<span style="font-size:7.5pt; font-family:华文细黑; color:gray"><span lang="EN-US">-------------------------------------------------------------------------------------------------------------------------------------<br>
</span>本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出<span lang="EN-US"><br>
</span>的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、<span lang="EN-US"><br>
</span>或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本<span lang="EN-US"><br>
</span>邮件！<span lang="EN-US"><br>
</span></span><span lang="EN-US" style="font-size:7.5pt; font-family:"Arial","sans-serif"; color:gray">This e-mail and its attachments contain confidential information from H3C, which is
<br>
intended only for the person or entity whose address is listed above. Any use of the
<br>
information contained herein in any way (including, but not limited to, total or partial
<br>
disclosure, reproduction, or dissemination) by persons other than the intended <br>
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
<br>
by phone or email immediately and delete it!</span>
</body>
</html>