[rds-devel] The meaning of MR invalidation

[Olaf Kirch]

Here's something that came up in talking to Or, which came back to me when
I hacked on the rds-rdma manpage again.

What is the purpose of invalidating MRs? IIRC the underlying motivation
is to prevent future access to this memory range in case the server
still holds on to the key after we're done with it, and decides it's
not done yet and wants to write to that memory again. (As an aside - this
is another good reason to avoid retransmitting RDMA ops)

So, scenario 1:
	client obtains R_Key
	sends to server
	server scribbles to memory
	client releases the MR and invalidates
	server tries to scribble to memory again => boink

Which is nice.

Now, scenario 2:
	client obtains R_Key
	sends to server
	server scribbles to memory
	client releases the MR and invalidates
	another application calls GET_MR and obtains the same R_Key
	server tries to scribble to memory again => oops, corruption!

The problem here is that the FMR pools (both the one in the IB stack
and the one I wrote) try to reuse FMRs - so the applications will see
the same R_Key over and over again; just the mapping changed.

What's currently happening when we release a MR is we put it on the
pool's free list. If the application asks for invalidation, we additionally
call the pool's flush function, which tears down the mapping of all
MRs on the free list.

We could make this a little more robust I guess by also freeing MRs
when the application asks for an invalidate. At least the mthca
driver seems to generate unique R_Keys (the top bits of the R_Key
are used as a kind of counter, so we'd really have to cycle through
4 billion MRs before getting the same R_Key).

Comments?

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax