[rds-devel] Has anyone tried RDS on IA64 Linux?

[Or Gerlitz]

Olaf Kirch wrote:
> You do have at most 32 transactions in parallel. But each of them pins
> 2*1Meg - once on the requester side through the FMR that was set
> up, and once on the responder side when the buffer is pinned for the
> duration of the RDMA.
OK, so you say that each rds-stress transaction is  "bidirectional"  
from the sender to the receiver and also the other way, and this applies 
also to RDMA.
> Also, note that we're not destroying the FMRs right away, but let
> them accumulate to amortize the cost of the TPT flush. With a 2K
> FMR pool you really can end up with up to 2K FMRs that have some
> memory pinned - and if each FMR is 1Meg, this may indeed end up
> blowing the limits of the IOMMU.
I see here possible confusion / little design bug between memory pinning 
and dma mapping - for buffer to be handed to an RDMA device it has to go 
through:
- memory pinning
- memory registration
- dma mapping

Now, when the RDMA (in the wide sense which covers also Send/Receive) 
completes it has to go through the reverse sequence
- dma un-mapping
- memory de-registration
- memory un-pinning

The dma un mapping must takes place before there is any attempt to touch 
this buffer from the CPU. If the RDS code deferred this to the time it 
issues memory de-registration - its a bug.

Or.