[rds-devel] [PATCH future/20090207 2/2] RDS/IB+IW: Allow recv handling while in CONNECTING state.

[Steve Wise]

From: None <None>

It is possible to get a recv completion with the QP in RTS _before_
RDS has a change to process the RDMA-CM ESTABLISHED event.  The result is
that the recv completion handler can be called with the RDS connection
state set to CONNECTING.

The current logic only processes recv completions if the RDS connection
state is UP.  Further, it _skips_ completions in this state which results
in effectively missing incoming data.

I've reproduced this problem by running simultaneous rds-stress runs
on 2 nodes over iwarp (one in each direction effectively generating
bidir traffic).  The failure mode in this case is a connection drop due
to an unexpected fragment.

The solution is to allow recv processing while in CONNECTING mode.
This resolves the failure mode I'm seeing.

Signed-off-by: Steve Wise <swise at opengridcomputing.com>
---

 net/rds/ib_recv.c |    7 ++++++-
 net/rds/iw_recv.c |    7 ++++++-
 net/rds/rds.h     |    6 ++++++
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/net/rds/ib_recv.c b/net/rds/ib_recv.c
index 45b569d..e45638a 100644
--- a/net/rds/ib_recv.c
+++ b/net/rds/ib_recv.c
@@ -803,7 +803,12 @@ void rds_ib_recv_cq_comp_handler(struct ib_cq *cq, void *context)
 
 		rds_ib_recv_unmap_page(ic, recv);
 
-		if (rds_conn_up(conn)) {
+		/*
+	 	 * Also process recvs in connecting state because it is possible
+		 * to get a recv completion _before_ the rdmacm ESTABLISHED
+		 * event is processed.
+		 */
+		if (rds_conn_up(conn) || rds_conn_connecting(conn)) {
 			/* We expect errors as the qp is drained during shutdown */
 			if (wc.status == IB_WC_SUCCESS) {
 				rds_ib_process_recv(conn, recv, wc.byte_len, &state);
diff --git a/net/rds/iw_recv.c b/net/rds/iw_recv.c
index 05bccc5..5b89344 100644
--- a/net/rds/iw_recv.c
+++ b/net/rds/iw_recv.c
@@ -803,7 +803,12 @@ void rds_iw_recv_cq_comp_handler(struct ib_cq *cq, void *context)
 
 		rds_iw_recv_unmap_page(ic, recv);
 
-		if (rds_conn_up(conn)) {
+		/*
+	 	 * Also process recvs in connecting state because it is possible
+		 * to get a recv completion _before_ the rdmacm ESTABLISHED
+		 * event is processed.
+		 */
+		if (rds_conn_up(conn) || rds_conn_connecting(conn)) {
 			/* We expect errors as the qp is drained during shutdown */
 			if (wc.status == IB_WC_SUCCESS) {
 				rds_iw_process_recv(conn, recv, wc.byte_len, &state);
diff --git a/net/rds/rds.h b/net/rds/rds.h
index 0a88b8e..0247af8 100644
--- a/net/rds/rds.h
+++ b/net/rds/rds.h
@@ -624,6 +624,12 @@ rds_conn_up(struct rds_connection *conn)
 	return atomic_read(&conn->c_state) == RDS_CONN_UP;
 }
 
+static inline int
+rds_conn_connecting(struct rds_connection *conn)
+{
+	return atomic_read(&conn->c_state) == RDS_CONN_CONNECTING;
+}
+
 /* message.c */
 struct rds_message *rds_message_alloc(unsigned int nents, gfp_t gfp);
 struct rds_message *rds_message_copy_from_user(struct iovec *first_iov,