[rds-devel] Re: comments on the send CQ completion handler

Olaf Kirch olaf.kirch at oracle.com
Wed Jan 9 08:34:55 PST 2008


On Wednesday 09 January 2008 17:01, Or Gerlitz wrote:
> Richard Frank wrote:
> > All send operations should have signaled set as the last fragment is 
> > xmitted - and may have it set for one or more of the intervening 
> > fragments - based on the number of fragments sent for a message.
> 
> Indeed, but this is not how rds_ib_xmit_rdma works, rdma is always 
> serves by one posting to the qp (i.e one fragment), and asking a 
> completion to be generated for this post is conditioned on the 
> max_unsig_bytes/wrs values.

No, after the loop there's this obscure little fragment:

	/* if we finished the message then send completion owns it */
        if (scat == &op->r_sg[op->r_count]) {
                prev->s_wr.send_flags = IB_SEND_SIGNALED | IB_SEND_SOLICITED;
                prev->s_op = op;
        }

> On the other thread you wrote that "Our planned use of immediate data 
> (notification message of rdma completion) keeps the immediate data (msg) 
> separate from the rdma data" from which I understand you first want to 
> get notification on the rdma completion and only after this (and 
> possibly some more processing is done) send the immediate data, did I 
> miss anything?

I think there may be some confusion around the term "immediate data".
What Rick means by that is not the same the IB spec means when it
talks about eg RDMA writes with immediate data.

When the application triggers an RDMA transfer, it does a sendmsg -
a normal message with some ancillary information that triggers the
RDMA. The ancillary information is passed via a socket control message
(msghdr.msg_control).

The kernel puts this into one rds_message, and attaches the rdma_op
to it.

The whole thing is being sent as

 -	first we queue up the RDMA transfer via rds_ib_xmit_rdma
 -	then we send the message (what Rick refers to as immediate
	data) via a normal SEND; but without waiting for its completion

Did this help?

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax



More information about the rds-devel mailing list