[rds-devel] The meaning of MR invalidation

[Or Gerlitz]

>> From: Olaf Kirch 
>> So there is a strong case to reuse FMRs. The last nagging worry I have
>> is, what happens when I remap a FMR - does that invalidate the old
>> mapping, or not?

Dror Goldenberg wrote:
> FMR remapping does not guarantee invalidation of the old mapping. The MR
> number changes when remapped, so it guaranteed that the new MR number
> makes the new mapping accessible. The old mapping may or may not exist
> until SYNC_TPT is submitted (and completes).

OK, that was my thinking as well, and just to clarify, my understanding 
is that "MR number" refers to R_Key. So at the bottom line, through the 
life time of an FMR there are synchronization points (SYNC_TPT) where 
you know that from now and on old mapping possibly kept by a remote peer 
are invalid.

However, the minute this synched-FMR is used again (mapped) one gets the 
--same set-- of R_Keys before the synchronization, so unless some 
explicit synchronization with the remote party is added, the  other 
synch does not worth much from the aspect you are interested in, at 
least as I understand it, since the remote may still try to use the 
old-from-your-perspective instance of the R_Key

I have brought the point of a need for some invalidation protocol 
earlier over this thread, and gave the SCSI aborts as an example.

> From: Olaf Kirch [mailto:olaf.kirch at oracle.com]
> ... and the bad news is, if I change the code to use FMRs exactly once,
> I lose roughly 10% in performance. I ran with 1, 2, and 4 tasks, a wide
> range of message sizes and RDMA sizes from 64k to 1M. The degradation
> was pretty similar across all test runs.

So how many IOs (=FMR mappings) per second are needed to serve this load?

Or.