[rds-devel] Re: send-with-invalidate

[Or Gerlitz]

Olaf Kirch wrote:
>>     The major motivation for the next three patches was
>>     the desire to have MRs with "fire and forget" behavior.
>>     I.e. the application allocates a MR, asks the peer
>>     to RDMA to/from it and have the kernel release the
>>     MR as soon as the RDMA completes. This eliminates
>>     one system call (we no longer need to call FREE_MR),
>>     and it makes more efficient use of MRs, as they are
>>     a very scarce resource.

Note that you have re-invented here the "send-with-invalidate" from the 
IB spec... which is fine, since at least from what I know, Mellanox does 
not show any intension to implement it in FW/SW.

Anyway, with this at hand, maybe you want to go further and eliminate 
the need for the get_mr system call on the client side, that mark to rds 
to register these bytes and send the rkey/addr in the extended header 
attached to the app message and provide the header to the app on the 
remote side or an id which will let rds get the rkey/addr from it tables 
etc.

>>
>> [RDS] Add support for extension headers
>>     This adds the plumbing for RDS header extensions.
>>     It reserves 16 bytes of extra space at the end of
>>     the RDS header, which can be used for various purposes.
>>
>> [RDS] Add support for RDMA use_once buffers
>>     This implements the above mentioned "fire and forget"
>>     MRs. It does so by sending the MR key inside a
>>     RDS header extension, as part of the RDS message
>>     immediately following the RDMA.
>>
>> [RDS] Transmit RDS protocol version in first packet
>>     This adds another extension header, including the
>>     RDS protocol version, and sends this header as part
>>     of the very first packet going across the wire when
>>     a connection is (re-) established. This will allow
>>     future RDS protocol changes to be implemented with
>>     backwards compatibility, so that we can perform
>>     rolling updates of a cluster.