[rds-devel] RDS rdma-read logic

[Steve Wise]

Hey guys,

I just walked through the RDS kernel module logic for issuing RDMA READ 
operations.  I wanted to confirm my understanding of how it works.  Can 
you lemme know if this is correct:

Application gets a remote cookie for the source of the RDMA READ from 
its peer.

Application posts an RDMA_ARGS control message which requests an RDMA 
READ and describes  the source of the read (the remote cookie) as well 
as the sink memory of the RDMA READ (basically a virtual address and 
length of the application's memory to be used).

In rds_rdma_prepare(), the kernel pins the application memory and builds 
an rdma op (rds_rdma_op) structure that describes the page list for this 
memory as well as the rdma operation in general. EG: this struct holds 
the peer's rkey and offset.

In rds_ib_xmit_rdma(), when the READ is going to be posted (after any 
flow control queueing), the application memory is dma mapped.

In this same function, the READ WR is built with the scatter gather list 
of DMA addresses from the mapping and a dma_mr lkey is used.

Once the READ WR completes and all refs are removed on the rds send 
message, rds_message_put() is called to clean up. The memory is unmapped 
in rds_ib_send_unmap_rdma().  The memory is unpinned and the rdma op 
struct is freed in rds_rdma_free_op() called by rds_message_purge().

Is this basically correct?

Thanks,

Steve.