[rds-devel] Re: send-with-invalidate

[Or Gerlitz]

Olaf Kirch wrote:
> However, some applications may want to create long-lived MRs, so
> we probably shouldn't disable the GET_MR call completely.

Many times, for example in the storage world, long lived MRs are somehow 
more natural in the --target-- side, eg you preallocate and pin set of 
big chunks of memory that would serve incoming I/O request for that 
connection.

In the RDS scheme, only the client side can cache the registration (i.e 
call get_mr but never call free_mr. I think we have to have at least 
symmetric, it would also make the code simpler: allow apps to pin,fmr 
the set of pages once, and then just dma_map/unmap it for each I/O.

If we want to refine that, we can separate the pinning from fmr-ing, so 
an app has set of pages pinned once, and then it fmrs subsets of this 
page set to serve an I/O. Note that this is exactly how it goes in the 
scsi world with the iser and srp initiator. The kernel buffer cache pins 
a set of buffers, per I/O an sg pointing to a subset of pages is carved 
from the cache, the initiator FMRs it and send the target the 
<rkey,addr,len> triplet, the targets gets a buffer from its pool, etc.

Or.