[rds-devel] The meaning of MR invalidation

[Olaf Kirch]

On Tuesday 12 February 2008 11:05, Or Gerlitz wrote:
> Sorry, but there's some devil in the details here... indeed, once an fmr 
> is remapped you get a new rkey. But, as Rick mentioned, only after the 
> HW driver issues a SYNC_TPT command on the HCA (as does mthca_unmap_fmr) 
> its guaranteed that remote parties will not be able to use the "past set 
> of rkeys associated with this fmr" (the "old" rkey is not valid from the 
> HCA network MMU point of view, but this table has a cache (aka TPT) and 
> the cache is flushed only with you issue the sync-tpt command.

Yes, that's understood - and that's just fine. The application will have
to issue an invalidate when it wants to make sure that all previously granted
access rights are revoked.

> > The RDS fmr pool in its current version has 3 lists - one for unused FMRs
> > that still have a mapping; one for unused FMRs that have been cleaned (ie
> > they have been through ib_fmr_unmap); and a third one for unused FMRs that
> > have reached the max_remap limit.
> 
> So what is the difference between the first and third lists? if an fmr 
> has reached its max_remap limit and has not gone yet through 
> ib_fmr_unmap it has a valid mapping from the HCA view point, isn't it?

The difference between these is that you can reuse FMRs from the first
two lists, but not from the third. We could certainly make it a single
list, but that means we need to do more work in the allocation code
as we walk the list to find a suitable unused FMR.

> Just to make sure, this means that each fmr would be used once and then 
> ib_fmr_unmap would be called before the next time, so there will not be 
> any remap unless an unmap takes place? there is some chance that in this 
> case you would lose the "F" from the FMR concept.

That's one of the things I have no real understanding of. What is the
actual difference in performance when you use an FMR exactly once?

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax