[rds-devel] The meaning of MR invalidation

[Or Gerlitz]

Olaf Kirch wrote:
> On Monday 11 February 2008 12:46, Or Gerlitz wrote:

>> I have read the comments made by Rick and yourself over this thread, and 
>> I was not sure whether you think that other then the wrapping scheme I 
>> describe above same R_Key can be seen by the server twice.

> It was a concern about reusing a FMR, and getting the same R_Key. That
> would not be safe from our point of view.

Sorry, but there's some devil in the details here... indeed, once an fmr 
is remapped you get a new rkey. But, as Rick mentioned, only after the 
HW driver issues a SYNC_TPT command on the HCA (as does mthca_unmap_fmr) 
its guaranteed that remote parties will not be able to use the "past set 
of rkeys associated with this fmr" (the "old" rkey is not valid from the 
HCA network MMU point of view, but this table has a cache (aka TPT) and 
the cache is flushed only with you issue the sync-tpt command.

> The RDS fmr pool in its current version has 3 lists - one for unused FMRs
> that still have a mapping; one for unused FMRs that have been cleaned (ie
> they have been through ib_fmr_unmap); and a third one for unused FMRs that
> have reached the max_remap limit.

So what is the difference between the first and third lists? if an fmr 
has reached its max_remap limit and has not gone yet through 
ib_fmr_unmap it has a valid mapping from the HCA view point, isn't it?

> If we decide to never reuse FMRs, all we'd need would be one list; and
> we could do away with a fair amount of code dealing with FMR reuse.

Just to make sure, this means that each fmr would be used once and then 
ib_fmr_unmap would be called before the next time, so there will not be 
any remap unless an unmap takes place? there is some chance that in this 
case you would lose the "F" from the FMR concept.

> But, as you say, the R_Key changes whenever we remap an FMR, so we
> better keep this code around I guess :)

Again, I don't follow, lets first sync (...) on what I wrote above and 
my question to the role of these lists with your assumptions after we 
are in ync.

Or