[rds-devel] The meaning of MR invalidation

[Or Gerlitz]

Olaf Kirch wrote:
> Now, scenario 2:
>         client obtains R_Key
>         sends to server
>         server scribbles to memory
>         client releases the MR and invalidates
>         another application calls GET_MR and obtains the same R_Key
>         server tries to scribble to memory again => oops, corruption!

> The problem here is that the FMR pools (both the one in the IB stack
> and the one I wrote) try to reuse FMRs - so the applications will see
> the same R_Key over and over again; just the mapping changed.

Olaf,

I just want to make sure we are on the same page here, so lets first
take a minute to clarify things:

With the FMRs you are using, the R_Key is broken to (at least) two
parts, the first one (say the "prefix") is unique and the second one
(say the lower XX "suffix" bits) are running counter which changes every
re-map of this fmr and wraps around at some point, such that the
consumer must issue an unmap on the fmr when this happens.

With this at hand (and please let me know if you disagree or have
clarification questions), we have that:

(A) no two FMRs on the same HCA would be ever mapped to the same R_Key
(b/c of the "prefix")

(B) after some period at time a specific FMR would be mapped to an R_Key
which it was mapped to in the past. The core fmr pool code does a device
query and uses the max_map_per_fmr attribute to decide when to put an
FMR in the dirty list. On my MT25208 system I see that the value of  the
attribute is 0x7fff so after 32K remaps an unmap would take place and
then the fmr would be mapped to the same (sequence of) rkeys.

I have read the comments made by Rick and yourself over this thread, and 
I was not sure whether you think that other then the wrapping scheme I 
describe above same R_Key can be seen by the server twice.

> We could make this a little more robust I guess by also freeing MRs
> when the application asks for an invalidate. At least the mthca
> driver seems to generate unique R_Keys (the top bits of the R_Key
> are used as a kind of counter, so we'd really have to cycle through
> 4 billion MRs before getting the same R_Key).

its not 4 billion MRs but rather 32K remaps of the same MR

For example I failed to follow Olaf's comment that fair amount of code 
can be removed from the pool (which one the RDS or the IB core?) when 
taking the above paragraph into account.

Or