[rds-devel] FW: RDS -- RDMA out of order

[Tang, Changqing]

Andy,
        Normally I see completion notification with remote error, then a lot of dropped notifications.
        But sometimes during normal run, I see a couple dropped notifications only. Do
You have explanation why there is only dropped notification without a leading error notification ?

        Thanks.

--CQ

-----Original Message-----
From: Andy Grover [mailto:andy.grover at oracle.com]
Sent: Monday, May 03, 2010 12:54 AM
To: Tang, Changqing
Cc: Tina Yang; RDS Devel
Subject: Re: [rds-devel] FW: RDS -- RDMA out of order

On 05/02/2010 10:05 PM, Tang, Changqing wrote:
> I checked that the messages were retransmitted by me after the
> messages were dropped. This comes to another question, if rds-rdma
> messages to the same target process are dropped, does RDS drop them
> in the same order as user posted ?

> What I noticed is that the dropped order is not the same as the
> order I posted, I want to check with you if this is the anticipated
> RDS behavior. I prefer RDS to drop messages in the same order as they
> are posted, the reason for it is that, application does not need to
> check the order of dropped messages, application only needs to decide
> to retransmit it or not.

Yes they should be dropped in order.

See rds_send_reset(). When the connection is reset, the entire retrans
queue gets spliced back onto the send_queue. Then, when going back
through send_xmit, any with rdma ops attached are dropped. I don't see
any reordering taking place that could explain the behavior you're
seeing, so I'm a bit baffled.

Regards -- Andy