[rds-devel] Re: comments on the send CQ completion handler

Or Gerlitz ogerlitz at voltaire.com
Tue Jan 8 06:59:21 PST 2008


+void rds_ib_send_unmap_rm(struct rds_ib_connection *ic,
+		          struct rds_ib_send_work *send)
+{
+	rdsdebug("ic %p send %p rm %p\n", ic, send, send->s_rm);
+
+	dma_unmap_sg(ic->i_cm_id->device->dma_device,
+		     send->s_rm->m_sg, send->s_rm->m_nents,
+		     DMA_TO_DEVICE);
+
+	/* raise rdma completion hwm */
+	if (send->s_rm->m_rdma_op)
+		rds_barrier_update(send->s_rm, send->s_rm->m_rdma_op->r_rdma_id);
+	rds_message_put(send->s_rm);
+	send->s_rm = NULL;
+}

+void rds_ib_send_cq_comp_handler(struct ib_cq *cq, void *context)
+{
...
+		completed = rds_ib_ring_completed(&ic->i_send_ring, wc.wr_id, oldest);
+
+		for (i = 0; i < completed; i++) {
+			if (wc.opcode == IB_WC_SEND) {
+				if (send->s_rm)
+					rds_ib_send_unmap_rm(ic, send);
+			}

Actually, I started to look here, since I did not understand how the barriers design
goes hand in hand with the selective send CQ signaling, that is the method of asking
for completion for 1 out of m calls to ib_post_send.

My understanding of the code is that for rm instance (rds_message) that represents
the (possibly empty) message which is provided as the immediate data of rdma
operation the m_rdma_op field points to the rdma op descriptor (am I correct?) and
when rds sense the completion of this rm instance it would signal the barrier, correct?

This suggests a possible live-lock or what ever you may call it, when the rdma
and/or the send following it did not set the IB_SEND_SIGNALED bit in the send flags
but there's no further outgoing traffic and the user calls for barrier.

Or.



More information about the rds-devel mailing list