[rds-devel] Re: comments on the send CQ completion handler

[Richard Frank]

Or Gerlitz wrote:
> +void rds_ib_send_unmap_rm(struct rds_ib_connection *ic,
> +		          struct rds_ib_send_work *send)
> +{
> +	rdsdebug("ic %p send %p rm %p\n", ic, send, send->s_rm);
> +
> +	dma_unmap_sg(ic->i_cm_id->device->dma_device,
> +		     send->s_rm->m_sg, send->s_rm->m_nents,
> +		     DMA_TO_DEVICE);
> +
> +	/* raise rdma completion hwm */
> +	if (send->s_rm->m_rdma_op)
> +		rds_barrier_update(send->s_rm, send->s_rm->m_rdma_op->r_rdma_id);
> +	rds_message_put(send->s_rm);
> +	send->s_rm = NULL;
> +}
>
> +void rds_ib_send_cq_comp_handler(struct ib_cq *cq, void *context)
> +{
> ...
> +		completed = rds_ib_ring_completed(&ic->i_send_ring, wc.wr_id, oldest);
> +
> +		for (i = 0; i < completed; i++) {
> +			if (wc.opcode == IB_WC_SEND) {
> +				if (send->s_rm)
> +					rds_ib_send_unmap_rm(ic, send);
> +			}
>
> Actually, I started to look here, since I did not understand how the barriers design
> goes hand in hand with the selective send CQ signaling, that is the method of asking
> for completion for 1 out of m calls to ib_post_send.
>
> My understanding of the code is that for rm instance (rds_message) that represents
> the (possibly empty) message which is provided as the immediate data of rdma
> operation the m_rdma_op field points to the rdma op descriptor (am I correct?) and
> when rds sense the completion of this rm instance it would signal the barrier, correct?
>
> This suggests a possible live-lock or what ever you may call it, when the rdma
> and/or the send following it did not set the IB_SEND_SIGNALED bit in the send flags
> but there's no further outgoing traffic and the user calls for barrier.
>   
If I understand your point - we need to make sure to signal completion 
on send side - for the immediate data and or the rdma completing - to 
ensure we can update the barrier - otherwise the client waiting on a 
barrier may get stuck - until it issues a subsequent rdma operation 
(which hopefully is signaled).

All send operations should have signaled set as the last fragment is 
xmitted - and may have it set for one or more of the intervening 
fragments - based on the number of fragments sent for a message.

This raises another point - we do not need signaling of rdma completions 
- as all rdmas are always followed with the immediate data send - which 
should be signaled. We use the immediate data send completion to set the 
barrier  hwm .


> Or.
>
> _______________________________________________
> rds-devel mailing list
> rds-devel at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/rds-devel
>