[rds-devel] Re: The meaning of MR invalidation

[Richard Frank]

Olaf Kirch wrote:
> Here's something that came up in talking to Or, which came back to me when
> I hacked on the rds-rdma manpage again.
>
> What is the purpose of invalidating MRs? IIRC the underlying motivation
> is to prevent future access to this memory range in case the server
> still holds on to the key after we're done with it, and decides it's
> not done yet and wants to write to that memory again. (As an aside - this
> is another good reason to avoid retransmitting RDMA ops)
>
>   
Yes, this is what we're after..

> So, scenario 1:
> 	client obtains R_Key
> 	sends to server
> 	server scribbles to memory
> 	client releases the MR and invalidates
> 	server tries to scribble to memory again => boink
>
> Which is nice.
>
>   
yep
> Now, scenario 2:
> 	client obtains R_Key
> 	sends to server
> 	server scribbles to memory
> 	client releases the MR and invalidates
> 	another application calls GET_MR and obtains the same R_Key
> 	server tries to scribble to memory again => oops, corruption!
>
>   
Not a good thing.

> The problem here is that the FMR pools (both the one in the IB stack
> and the one I wrote) try to reuse FMRs - so the applications will see
> the same R_Key over and over again; just the mapping changed.
>
>   
re-use of keys is one reason we shut off the existing fmr caching - the 
r_keys were getting re-used - which leads to scenario 2 above.

> What's currently happening when we release a MR is we put it on the
> pool's free list. If the application asks for invalidation, we additionally
> call the pool's flush function, which tears down the mapping of all
> MRs on the free list.
>
> We could make this a little more robust I guess by also freeing MRs
> when the application asks for an invalidate. At least the mthca
> driver seems to generate unique R_Keys (the top bits of the R_Key
> are used as a kind of counter, so we'd really have to cycle through
> 4 billion MRs before getting the same R_Key).
>
>   
Yes - this is what we want.

WRT to our favorite client - we ask for an invalidate to fence prior 
granted operations - the memory itself of course can be re-used for new 
operations.  This does not happen frequently - but when it does we want 
the hard fence.

Some scenarios where we need to fence include - 1) process death, 2) 
when the app decides that it wants to abort some prior operations - say 
as nodes join or leave, etc.

> Comments?
>
> Olaf
>