[rds-devel] [PATCH future/20090207 2/2] RDS/IB+IW: Allow recv handling while in CONNECTING state.

[Andy Grover]

Thanks, applied both patches (fixed the "From:") -- Andy

Steve Wise wrote:
> From: None <None>
> 
> It is possible to get a recv completion with the QP in RTS _before_
> RDS has a change to process the RDMA-CM ESTABLISHED event.  The result is
> that the recv completion handler can be called with the RDS connection
> state set to CONNECTING.
> 
> The current logic only processes recv completions if the RDS connection
> state is UP.  Further, it _skips_ completions in this state which results
> in effectively missing incoming data.
> 
> I've reproduced this problem by running simultaneous rds-stress runs
> on 2 nodes over iwarp (one in each direction effectively generating
> bidir traffic).  The failure mode in this case is a connection drop due
> to an unexpected fragment.
> 
> The solution is to allow recv processing while in CONNECTING mode.
> This resolves the failure mode I'm seeing.
> 
> Signed-off-by: Steve Wise <swise at opengridcomputing.com>
> ---
> 
>  net/rds/ib_recv.c |    7 ++++++-
>  net/rds/iw_recv.c |    7 ++++++-
>  net/rds/rds.h     |    6 ++++++
>  3 files changed, 18 insertions(+), 2 deletions(-)
> 
> diff --git a/net/rds/ib_recv.c b/net/rds/ib_recv.c
> index 45b569d..e45638a 100644
> --- a/net/rds/ib_recv.c
> +++ b/net/rds/ib_recv.c
> @@ -803,7 +803,12 @@ void rds_ib_recv_cq_comp_handler(struct ib_cq *cq, void *context)
>  
>  		rds_ib_recv_unmap_page(ic, recv);
>  
> -		if (rds_conn_up(conn)) {
> +		/*
> +	 	 * Also process recvs in connecting state because it is possible
> +		 * to get a recv completion _before_ the rdmacm ESTABLISHED
> +		 * event is processed.
> +		 */
> +		if (rds_conn_up(conn) || rds_conn_connecting(conn)) {
>  			/* We expect errors as the qp is drained during shutdown */
>  			if (wc.status == IB_WC_SUCCESS) {
>  				rds_ib_process_recv(conn, recv, wc.byte_len, &state);
> diff --git a/net/rds/iw_recv.c b/net/rds/iw_recv.c
> index 05bccc5..5b89344 100644
> --- a/net/rds/iw_recv.c
> +++ b/net/rds/iw_recv.c
> @@ -803,7 +803,12 @@ void rds_iw_recv_cq_comp_handler(struct ib_cq *cq, void *context)
>  
>  		rds_iw_recv_unmap_page(ic, recv);
>  
> -		if (rds_conn_up(conn)) {
> +		/*
> +	 	 * Also process recvs in connecting state because it is possible
> +		 * to get a recv completion _before_ the rdmacm ESTABLISHED
> +		 * event is processed.
> +		 */
> +		if (rds_conn_up(conn) || rds_conn_connecting(conn)) {
>  			/* We expect errors as the qp is drained during shutdown */
>  			if (wc.status == IB_WC_SUCCESS) {
>  				rds_iw_process_recv(conn, recv, wc.byte_len, &state);
> diff --git a/net/rds/rds.h b/net/rds/rds.h
> index 0a88b8e..0247af8 100644
> --- a/net/rds/rds.h
> +++ b/net/rds/rds.h
> @@ -624,6 +624,12 @@ rds_conn_up(struct rds_connection *conn)
>  	return atomic_read(&conn->c_state) == RDS_CONN_UP;
>  }
>  
> +static inline int
> +rds_conn_connecting(struct rds_connection *conn)
> +{
> +	return atomic_read(&conn->c_state) == RDS_CONN_CONNECTING;
> +}
> +
>  /* message.c */
>  struct rds_message *rds_message_alloc(unsigned int nents, gfp_t gfp);
>  struct rds_message *rds_message_copy_from_user(struct iovec *first_iov,
>