[rds-devel] Has anyone tried RDS on IA64 Linux?
Or Gerlitz
ogerlitz at voltaire.com
Tue Apr 29 04:32:25 PDT 2008
Olaf Kirch wrote:
> You do have at most 32 transactions in parallel. But each of them pins
> 2*1Meg - once on the requester side through the FMR that was set
> up, and once on the responder side when the buffer is pinned for the
> duration of the RDMA.
OK, so you say that each rds-stress transaction is "bidirectional"
from the sender to the receiver and also the other way, and this applies
also to RDMA.
> Also, note that we're not destroying the FMRs right away, but let
> them accumulate to amortize the cost of the TPT flush. With a 2K
> FMR pool you really can end up with up to 2K FMRs that have some
> memory pinned - and if each FMR is 1Meg, this may indeed end up
> blowing the limits of the IOMMU.
I see here possible confusion / little design bug between memory pinning
and dma mapping - for buffer to be handed to an RDMA device it has to go
through:
- memory pinning
- memory registration
- dma mapping
Now, when the RDMA (in the wide sense which covers also Send/Receive)
completes it has to go through the reverse sequence
- dma un-mapping
- memory de-registration
- memory un-pinning
The dma un mapping must takes place before there is any attempt to touch
this buffer from the CPU. If the RDS code deferred this to the time it
issues memory de-registration - its a bug.
Or.
More information about the rds-devel
mailing list