[rds-devel] Re: send-with-invalidate

[Richard Frank]

Olaf Kirch wrote:
> On Thursday 10 January 2008 12:39, Or Gerlitz wrote:
>   
>> Note that you have re-invented here the "send-with-invalidate" from the 
>> IB spec... which is fine, since at least from what I know, Mellanox does 
>> not show any intension to implement it in FW/SW.
>>     
>
> I wasn't aware of that. But if it's not implemented anyway, then it's fine :)
>
>   
>> Anyway, with this at hand, maybe you want to go further and eliminate 
>> the need for the get_mr system call on the client side, that mark to rds 
>> to register these bytes and send the rkey/addr in the extended header 
>> attached to the app message and provide the header to the app on the 
>> remote side or an id which will let rds get the rkey/addr from it tables 
>> etc.
>>     
>
> Yeah, I was thinking along the same line - ie have the application
> pass the memory region in a control message along with some "normal"
> message; 
So this would be special send operation (rdma_request ?) - or a  build  
routine to add the memory region to the control msg (build_rdma_request()) ?

> the kernel allocates a MR and passes its key along in a RDS
> extension header; finally the remote kernel translates the MR into an
> opaque token, which the application receives as a control message
> along with the message data itself.
>
>   
Not exposing the keys is probably a good thing.
> However, some applications may want to create long-lived MRs, so
> we probably shouldn't disable the GET_MR call completely.
>   
We have a pre-existing app which is based on having keys exposed. We'd 
need to do some significant rotor tilling  to change this. Maybe we can 
do this in the next major release... for now we need the current model.

However, all new uses for RDMA could employ the newly proposed model / 
interface .

WRT to use_once - we need to add the ability to request that the mr be 
invalidated when it is freed by the system.
When the mr is created and marked for use_once we need to add an 
"invalidate" flag too....

> Olaf
>