[rds-devel] Re: [PATCH] Implement our own RDMA MR pool

[Or Gerlitz]

Olaf Kirch wrote:
> On Wednesday 30 January 2008 16:06, Or Gerlitz wrote:

>>>      -	there is no longer a hard upper limit on the number of MRs,
>>>      	as was the case of the fmr_pool

>> limits are good, no-limits is problematic... you say there's no "hard 
>> upper limit", is there any limit?

> As I said in another message, I was relying on ib_alloc_fmr to return an error
> when it hits some internal limit. It does that now, but apparently it leaves
> the driver in an inconsistent state.

Again, my thinking here is that the rds design has to go hand in hand 
with being limited at some point, best if this limit can be part of the 
configuration. After all  nowadays environments are always open systems 
to some extent, so its somehow incorrect if one software component in 
the system exhausts all the resources of some type from the HCA. In the 
example of database you need the interconnect for connections going out, 
IPC, storage and management.

So, if you are willing to take this approach, you can create fmr pool 
large enough to serve the required load,  for that end, I have started 
to check how to implement the max_fmr device attribute.

As for the problem you have pointed in ib_alloc_fmr / mthca driver, 
Roland sent here a patch, I will be able to test it when you clarify 
what you mean by "it leaves the driver in an inconsistent state" since 
after applying the patch you sent to mthca, I don't see that crashes we 
were discussing before when running rds-stress, etc.

Or