[Ocfs2-devel] Is it an issue and whether the code changed correct? Thanks a lot
Guozhonghua
guozhonghua at h3c.com
Tue Jul 16 02:06:10 PDT 2013
Hi, everyone, is that an issue?
The Server version is Linux 3.2.0-23, Ubuntu 1204.
There are 4 nodes in the OCFS2 Cluster, using three iSCSI LUNS, and every LUN is one OCFS2 domain mounted by thread node.
As the network used buy node has one down/up, the tcp connection between node shutdown and reconnected with each other.
But there is one scenario that the node whose node number is little, shut down the tcp with node whose number is large, the node with large node number will not reconnect the node with little node number.
The otherwise is that if the node with large node number shut down the tcp with node with little number, the node with large node number will reconnect the node with little node number OK.
Such as below:
The server1 syslog is as below:
Jul 9 17:46:10 server1 kernel: [5199872.576027] o2net: Connection to node server2 (num 2) at 192.168.70.20:7100 shutdown, state 8
Jul 9 17:46:10 server1 kernel: [5199872.576111] o2net: No longer connected to node server2 (num 2) at 192.168.70.20:7100
Jul 9 17:46:10 server1 kernel: [5199872.576149] (ocfs2dc,14358,1):dlm_send_remote_convert_request:395 ERROR: Error -107 when sending message 504 (key 0x3671059b) to node 2
Jul 9 17:46:10 server1 kernel: [5199872.576162] o2dlm: Waiting on the death of node 2 in domain 3656D53908DC4149983BDB1DBBDF1291
Jul 9 17:46:10 server1 kernel: [5199872.576428] o2net: Accepted connection from node server2 (num 2) at 192.168.70.20:7100
Jul 9 17:46:11 server1 kernel: [5199872.995898] o2net: Connection to node server3 (num 3) at 192.168.70.30:7100 has been idle for 30.100 secs, shutting it down.
Jul 9 17:46:11 server1 kernel: [5199872.995987] o2net: No longer connected to node server3 (num 3) at 192.168.70.30:7100
Jul 9 17:46:11 server1 kernel: [5199873.069666] o2net: Connection to node server4 (num 4) at 192.168.70.40:7100 shutdown, state 8
Jul 9 17:46:11 server1 kernel: [5199873.069700] o2net: No longer connected to node server4 (num 4) at 192.168.70.40:7100
Jul 9 17:46:11 server1 kernel: [5199873.070385] o2net: Accepted connection from node server4 (num 4) at 192.168.70.40:7100
The server1 shutdown the tcp connection with server3, but server3 never reconnect with server1.
The server3 syslog is as below:
Jul 9 17:44:12 server3 kernel: [3971907.332698] o2net: Connection to node server1 (num 1) at 192.168.70.10:7100 shutdown, state 8
Jul 9 17:44:12 server3 kernel: [3971907.332748] o2net: No longer connected to node server1 (num 1) at 192.168.70.10:7100
Jul 9 17:44:42 server3 kernel: [3971937.355419] o2net: No connection established with node 1 after 30.0 seconds, giving up.
Jul 9 17:45:01 server3 CRON[52349]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 9 17:45:12 server3 kernel: [3971967.421656] o2net: No connection established with node 1 after 30.0 seconds, giving up.
Jul 9 17:45:42 server3 kernel: [3971997.487949] o2net: No connection established with node 1 after 30.0 seconds, giving up.
Jul 9 17:46:12 server3 kernel: [3972027.554258] o2net: No connection established with node 1 after 30.0 seconds, giving up.
Jul 9 17:46:42 server3 kernel: [3972057.620496] o2net: No connection established with node 1 after 30.0 seconds, giving up.
The node of server2 and server4 shut down the connection with server1, and reconnect them ok.
I review the code of the ocfs2 kernel and found this may be an issue or bug.
As node of server1 did not receive msg from server3, he shut the connection with server3 and set the timeout with 1.
The server1's node number is little than server3, so he wait the connect request from server3.
static void o2net_idle_timer(unsigned long data)
{
... ...
printk(KERN_NOTICE "o2net: Connection to " SC_NODEF_FMT " has been "
"idle for %lu.%lu secs, shutting it down.\n", SC_NODEF_ARGS(sc),
msecs / 1000, msecs % 1000);
..... ...
atomic_set(&nn->nn_timeout, 1);
o2net_sc_queue_work(sc, &sc->sc_shutdown_work);
}
But the server3 monitoring the TCP connection state changed and shutdown connect again and it will never reconnect with server1 because the nn->nn_timeout is 0.
static void o2net_state_change(struct sock *sk)
{
......
switch(sk->sk_state) {
......
default:
printk(KERN_INFO "AAAAA o2net: Connection to " SC_NODEF_FMT
" shutdown, state %d\n",
SC_NODEF_ARGS(sc), sk->sk_state);
o2net_sc_queue_work(sc, &sc->sc_shutdown_work);
break;
}
... ...
}
I had test the TCP connect without any shutdown between nodes, but send message will failed because the connection state is error.
I change the code for the connect triggers in function o2net_set_nn_state and o2net_start_connect, and the reconnect rebuild up OK.
Is anyone review the code correct? Thanks a lots.
root at gzh-dev:~/ocfs2# diff -p -C 10 ./ocfs2_org/cluster/tcp.c ocfs2_rep/cluster/tcp.c
*** ./ocfs2_org/cluster/tcp.c 2012-10-29 19:33:19.534200000 +0800
--- ocfs2_rep/cluster/tcp.c 2013-07-16 16:58:31.380452531 +0800
*************** static void o2net_set_nn_state(struct o2
*** 567,586 ****
--- 567,590 ----
if (!valid && o2net_wq) {
unsigned long delay;
/* delay if we're within a RECONNECT_DELAY of the
* last attempt */
delay = (nn->nn_last_connect_attempt +
msecs_to_jiffies(o2net_reconnect_delay()))
- jiffies;
if (delay > msecs_to_jiffies(o2net_reconnect_delay()))
delay = 0;
mlog(ML_CONN, "queueing conn attempt in %lu jiffies\n", delay);
+
+ /** Trigger the reconnection */
+ atomic_set(&nn->nn_timeout, 1);
+
queue_delayed_work(o2net_wq, &nn->nn_connect_work, delay);
/*
* Delay the expired work after idle timeout.
*
* We might have lots of failed connection attempts that run
* through here but we only cancel the connect_expired work when
* a connection attempt succeeds. So only the first enqueue of
* the connect_expired work will do anything. The rest will see
* that it's already queued and do nothing.
*************** static void o2net_start_connect(struct w
*** 1691,1710 ****
--- 1695,1719 ----
remoteaddr.sin_family = AF_INET;
remoteaddr.sin_addr.s_addr = node->nd_ipv4_address;
remoteaddr.sin_port = node->nd_ipv4_port;
ret = sc->sc_sock->ops->connect(sc->sc_sock,
(struct sockaddr *)&remoteaddr,
sizeof(remoteaddr),
O_NONBLOCK);
if (ret == -EINPROGRESS)
ret = 0;
+
+ /** Reset the timeout with 0 to avoid connection again, Just for test the tcp connection */
+ if (ret == 0) {
+ atomic_set(&nn->nn_timeout, 0);
+ }
out:
if (ret) {
printk(KERN_NOTICE "o2net: Connect attempt to " SC_NODEF_FMT
" failed with errno %d\n", SC_NODEF_ARGS(sc), ret);
/* 0 err so that another will be queued and attempted
* from set_nn_state */
if (sc)
o2net_ensure_shutdown(nn, sc, 0);
}
-------------------------------------------------------------------------------------------------------------------------------------
????????????????????????????????????????
????????????????????????????????????????
????????????????????????????????????????
???
This e-mail and its attachments contain confidential information from H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20130716/cae7604c/attachment-0001.html
More information about the Ocfs2-devel
mailing list