[rds-devel] FMR_POOL_SIZE = 1024 for RDS ?

[Olaf Kirch]

On Friday 04 January 2008 22:48, Richard Frank wrote:
> Is there any reason we would not increase this to 16k or more - what are 
> the limitations / resource issues here.
> 
> Allowing only 1024 I/Os outstanding seems low ?

Okay, I tried 16K, and it doesn't seem to give me any trouble.
There is some memory overhead here, however - the stack will
allocate a whole lot of memory in advance. The FMR pool itself
allocates pool_size * max_pages_per_fmr * sizeof(u64) for the
page vectors, plus some more memory per FMR. At a pool size of
16K and 256 pages per FMR, we're in the vicinity of 30-40Megs

The other limit that may be too low is RDS_FMR_SIZE, which is
currently at 256. So we're limited to RDMA messages of 1Meg.
I think we should at least make this a module option. See draft
patch below.

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax
---------
---
 net/rds/ib.c      |    5 ++++-
 net/rds/ib.h      |    2 ++
 net/rds/ib_rdma.c |    4 ++--
 3 files changed, 8 insertions(+), 3 deletions(-)

Index: ofa-kernel-1.3/net/rds/ib.c
===================================================================
--- ofa-kernel-1.3.orig/net/rds/ib.c
+++ ofa-kernel-1.3/net/rds/ib.c
@@ -41,9 +41,12 @@
 #include "ib.h"
 
 static int fmr_pool_size = RDS_FMR_POOL_SIZE;
+unsigned int fmr_message_size = RDS_FMR_SIZE;
 
 module_param(fmr_pool_size, int, 0444);
 MODULE_PARM_DESC(fmr_pool_size, " Max number of fmr per HCA");
+module_param(fmr_message_size, int, 0444);
+MODULE_PARM_DESC(fmr_message_size, " Max size of a RDMA transfer");
 
 struct list_head rds_ib_devices;
 
@@ -99,7 +102,7 @@ void rds_ib_add_one(struct ib_device *de
 	fmr_param.pool_size	    = fmr_pool_size;
 	fmr_param.dirty_watermark   = fmr_pool_size / 4;
 	fmr_param.cache		    = 0;
-	fmr_param.max_pages_per_fmr = RDS_FMR_SIZE;
+	fmr_param.max_pages_per_fmr = fmr_message_size;
 	fmr_param.page_shift	    = rds_ibdev->fmr_page_shift;
 	fmr_param.access	    = (IB_ACCESS_LOCAL_WRITE |
 				       IB_ACCESS_REMOTE_READ |
Index: ofa-kernel-1.3/net/rds/ib.h
===================================================================
--- ofa-kernel-1.3.orig/net/rds/ib.h
+++ ofa-kernel-1.3/net/rds/ib.h
@@ -159,6 +159,8 @@ extern void rds_ib_add_one(struct ib_dev
 extern void rds_ib_remove_one(struct ib_device *device);
 extern struct ib_client rds_ib_client;
 
+extern unsigned int fmr_message_size;
+
 /* ib_cm.c */
 int rds_ib_conn_alloc(struct rds_connection *conn, gfp_t gfp);
 void rds_ib_conn_free(void *arg);
Index: ofa-kernel-1.3/net/rds/ib_rdma.c
===================================================================
--- ofa-kernel-1.3.orig/net/rds/ib_rdma.c
+++ ofa-kernel-1.3/net/rds/ib_rdma.c
@@ -129,8 +129,8 @@ int ib_map_fmr(struct rds_ib_device *rds
 	}
 
 	page_cnt += len >> rds_ibdev->fmr_page_shift;
-	if (page_cnt > RDS_FMR_SIZE)
-		return -ENOMEM;
+	if (page_cnt > fmr_message_size)
+		return -EINVAL;
 
 	dma_pages = kmalloc(sizeof(u64) * page_cnt, GFP_ATOMIC);
 	if (!dma_pages)