[rds-devel] Has anyone tried RDS on IA64 Linux?

[Olaf Kirch]

On Tuesday 29 April 2008 10:51:29 Or Gerlitz wrote:
> rds has a module param through which you can control how many FMRs (a 
> mapping associated with each RDMA issued by rds) it would use at max, 
> the default is 2K, as each FMR maps at most 256 pages, this would 
> consume 512K entries at the HCA MMU (MTT) during setup time (when the 
> fmr pool is allocated), but nothing at the I/O MMU at this point.
> 
> I/O MMU wise, assuming each RDMA is wrapped with dma_map/unmap on both 
> sides, it depends how many transactions are inflight at your test. I 
> think that -t4 -d8 means that you have at most 32 transactions running 
> in parallel (Olaf, am I correct?) which will consume 8K (32*256) 
> pages-entries at the MMU, is it too much for this system?

You do have at most 32 transactions in parallel. But each of them pins
2*1Meg - once on the requester side through the FMR that was set
up, and once on the responder side when the buffer is pinned for the
duration of the RDMA.

Also, note that we're not destroying the FMRs right away, but let
them accumulate to amortize the cost of the TPT flush. With a 2K
FMR pool you really can end up with up to 2K FMRs that have some
memory pinned - and if each FMR is 1Meg, this may indeed end up
blowing the limits of the IOMMU.

For architectures like Itanium it may indeed make sense to use a much
lower limit for the FMR pool.

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax