[rds-devel] RDS - resource leakage - recv_ring counters - looks buggy

[Viral Mehta]

Hi,
I am again suspicious about RDS code.

I modified RDS code a little bit to confirm the same. I added a call to 
ib_cq_poll() after rdma_disconnect() call in rds_iw_conn_shutdown() 
function definition.

And as expected, I got CQ completion entry with cqe_flush status 
(IB_WC_WR_FLUSH_ERR) which I am not getting in normal code which means 
ib_cq_poll() is not being called when we are in disconnect path (or when 
modprobe -r rds is done).

If you can shed some light I can debug more.

Viral Mehta wrote:
> Hi Andy,
> Thanks for your response.
>
> Yes, I agree with you. rdma_disconnect should free up all SQ and RQ 
> WQEs. Also iwarp sepc confirms the same.
>
> So, looks like RDS has no problem. I will let you know if I find 
> something else.
>
> Thanks again,
>
> Andy Grover wrote:
>   
>> Viral Mehta wrote:
>>   
>>     
>>> I am, somehow, not able to forward this to netdev list.
>>>
>>> When we run any rds-ping test, it creates a connection. It sets up
>>> QP. And then it posts (1024 or whatever mentioned through sysctl)
>>> recvs. Basically these are pre-post recvs. Extra recvs will be posted
>>> again if it goes below low-watermark.
>>>
>>> By design, Connection and all RDMA resources are destroyed only when
>>> module is unloaded. Now in unloading process, before destroying RDMA
>>> resources, it  waits till all send_ring/recv_ring becomes empty. 
>>> =========== iw_cm.c:600:
>>> wait_event(rds_iw_ring_empty_wait, iw_cm.c-601-
>>> rds_iw_ring_empty(&ic->i_send_ring) && iw_cm.c-602-
>>> rds_iw_ring_empty(&ic->i_recv_ring)); ===========
>>>
>>> Ring empty means diff (i.e., ring->w_alloc_ctr - ring->free_ctr)
>>> should be zero. w_alloc_ctr are number of posted recvs and free_ctr
>>> is number of recvs consumed. Ideally, this can never be zero as we
>>> always want some pre-posted recvs and thus recv_ring will never be
>>> empty.
>>>     
>>>       
>> Hi Viral, sorry for the delay in responding.
>>
>> I believe what happens is that the rdma_disconnect() above the
>> wait_event causes all the outstanding recv wrs to be completed with an
>> error. This causes them to be freed. They are not refilled, and so the
>> ring becomes empty.
>>
>> Does this analysis appear correct to you?
>>
>> Thanks -- Regards -- Andy
>>
>>
>>
>> Email Scanned for Virus & Dangerous Content by : www.CleanMailGateway.com
>>
>>
>>   
>>     

-- 
Thanks, Viral Mehta, Embedded Software Engineer, www.einfochips.com