[rds-devel] The meaning of MR invalidation

[Olaf Kirch]

On Monday 18 February 2008 08:17, Or Gerlitz wrote:
> However, the minute this synched-FMR is used again (mapped) one gets the 
> --same set-- of R_Keys before the synchronization, so unless some 
> explicit synchronization with the remote party is added, the  other 
> synch does not worth much from the aspect you are interested in, at 
> least as I understand it, since the remote may still try to use the 
> old-from-your-perspective instance of the R_Key

Ouch, that's the rub indeed. One could argue that this is a bug in the FMR
implementation/specification. The map count should *not* be reset when
we call invalidate; it should be allowed to increment until we hit max_remaps,
at which point the FMR needs to be destroyed.

But of course there's always a residual chance that we issue the same R_Key
"soon enough" after invalidation to cause real harm.

> I have brought the point of a need for some invalidation protocol 
> earlier over this thread, and gave the SCSI aborts as an example.

Yes, it's possible that there's no way around it :-(

> > From: Olaf Kirch [mailto:olaf.kirch at oracle.com]
> > ... and the bad news is, if I change the code to use FMRs exactly once,
> > I lose roughly 10% in performance. I ran with 1, 2, and 4 tasks, a wide
> > range of message sizes and RDMA sizes from 64k to 1M. The degradation
> > was pretty similar across all test runs.
> 
> So how many IOs (=FMR mappings) per second are needed to serve this load?

With 4K messages and 64K RDMAs, I hit 10,000 ops/second. With smaller messages
even more.

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax