[rds-devel] what is rdma immediate data and how is it used ?

Or Gerlitz ogerlitz at voltaire.com
Tue Jan 8 04:01:16 PST 2008


Richard Frank wrote:
> Consider the case of an rdma server used to implement a simple zero copy 
> disk block server over  zero copy RDS sockets.

> To write a disk block, the rdma client requests that the rdma server 
> issue an rdma read from rdma client host memory into the rdma server 
> memory, and then to issue the disk write and send back a completion 
> message. What's interesting in this case is that the immediate data 
> which could be sent as part of the rdma read is optional (could be send 
> size of zero). If the immediate data is sent, the rdma client would know 
> that the rdma server has completed pulling the data from the rdma client 
> host memory - so it's possible for the rdma client to re-use the write 
> buffer at that point. Of course this assumes that either the rdma client 
> is willing to live possible data loss in light of a path failure - or 
> that the rdma disk server is guaranteeing to commit the data. When the 
> actual disk write completes a separate completion message is sent back 
> to the rdma client.

Hi Rick,

For the cases you mentioned here, where the immediate data send is 
posted to the HCA immediatly after issuing the rdma read, the fence 
indicator bit (IBV_SEND_FENCE) must be set in the send wqe flags, else 
bunch of mismatches can happen, as specified in the IB spec C10-100 
comment at section 10.8.3.3 , basically as specified in section 9.5 
comment c9-31 a work request with the fence set shall block until all 
prior rdma-read work requests have completed.

Or

If you agree to this claim, I will post Olaf a patch to fix that, 
basically my thinking is to have rds_send_xmit to do:

if (!rm->m_rdma_op->r_write && rm-has-associated-send)
	conn->trans->xmit(conn,rm,,,FENCE=1)
else
	conn->trans->xmit(conn,rm,,,FENCE=0)

this is not very elegant and I can think on few other ways to do so, but 
let first agree on the architectural issue.




More information about the rds-devel mailing list