[rds-devel] Re: [PATCH] Implement our own RDMA MR pool

[Or Gerlitz]

Olaf Kirch wrote:
> On Wednesday 30 January 2008 16:06, Or Gerlitz wrote:

>> this is a hell of good catch, so the fmr pool expose the system to 
>> someone scribble over pages it was given access before. I wonder if 
>> there's a way for the fmr pool to handle that. Maybe allocate some pages 
>> by the pool and have each MR which is dirty and in the free list more 
>> then 10 seconds to be mapped to this pages? Roland?

> I think the only real answer is to coordinate the unmapping of pages
> between the FMR pool and the ULP. What I did in the RDS fmr pool was to
> give the pool control over the scatterlist, so that it can unpin the
> pages when it is safe to do so. However, this also means we want to do
> some accounting of how much memory we have actually pinned by unused FMRs.

OK, I think that aspect is the most interesting and important part of 
this discussion... one thing to consider here if you want to protect 
against remote side writing on pages which are:

- not pinned (so the remote writes over --random-- memory)
- re-mapped under a different rkey than the one used by the remote
- not mapped under any rkey

I assume we want to prevent all these options, correct?

If indeed, we must either rely on the remote side when it said "I am 
done with this mapping" OR implement some maybe sophisticated protocol 
for invalidation of mapping provided to the remote side, SCSI I/O abort 
is an example for  this, so the SCSI midlayer will not release back to 
the buffer cache the bunch of pages pointed by this scatterlist till it 
either gets the response or issues abort, etc.

I don't see how your solution to "not unpin the pages till its safe to 
do so" protects against the case of the remote side using some rkey 
(mapping) which from your perspective is not relevant any more,  since 
you can not assume that sets of pages used for different I/O's are disjoint.

I will be happy to hear yours and Roland's thoughts on this matter, lets 
  see.

Or.