[rds-devel] The meaning of MR invalidation

[Or Gerlitz]

Dror Goldenberg wrote:
> I am not sure I understood what you meant in the paragraph below. Maybe
> more info or example will do.

Hi Dror,

My understanding (which I have validated by looking on the mthca driver
xxx_map_phys_fmr / xxx_fmr_unmap code) is that once an FMR is created 
its being associated with a 32bits key, where each remap of the fmr 
changes some of the lower bits of the key. At some point a set of fmrs 
are unmapped where unmapping does two things, first, for each fmr it 
resets the key to the initial value, and second it issues SYNC_TPT. Once 
an fmr is now remapped, it would get the initial key it had and so on.

This means the ULP can not count on fmr umapping as the only mean for
invalidation of rkey held by remote party, since with remapping this fmr 
,they get the same set of rkeys that were used before unmapping, and 
hence the unmap operation is a synchronization point, no more.

ULPs like the SCSI based iser and srp, would not invalidate the mapping 
until either SCSI response was received (so here there's trust on the 
target that it will not attempt to use this rkey) or the initiator 
issued SCSI abort and there was response on the abort or the target 
declared as being non-responsive, etc.

The point I was trying to make with Olaf is that first, the RDS design 
should count on the remote side when it sends a response on a mapping 
and second, --if-- he wants to go an extra step, I am not sure that 
counting on fmr unmapping and even destroying provides any real 
advantage over doing nothing, as of all the reasons discussed above. So 
what's needed here is some invalidation protocol where the client says 
"I want to revoke this <transaction ID, rkey> pair" and the server says 
"fine, I will not this pair any more", etc.

Or

> -----Original Message-----
> From: Or Gerlitz [mailto:ogerlitz at voltaire.com] 
> However, the minute this synched-FMR is used again (mapped) one gets the
> --same set-- of R_Keys before the synchronization, so unless some
> explicit synchronization with the remote party is added, the  other
> synch does not worth much from the aspect you are interested in, at
> least as I understand it, since the remote may still try to use the
> old-from-your-perspective instance of the R_Key