<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<br>
Well,<br>
The ball is again in different plate. It is confirmed that low-level
driver/hw has problem.<br>
<br>
Anyway, iwarp spec, in section 6.6.2.3 still says that <br>
<br>
"Once in the Error state, the RI flushes all incomplete WQEs on<br>
both the Send and Receive Queues by completing them with the<br>
Flushed Completion Status. The Consumer would presumably reap<br>
all of the Work Completions to ensure all resources are cleaned<br>
up. Once the Consumer believes all Work Completions have been<br>
reaped, it should attempt to transition the QP to the Idle state<br>
by performing a Modify QP. If the transition is successful, the<br>
Consumer knows it can either re-use the QP for another LLP<br>
Stream or it can invoke Destroy QP (see Section 6.1.3 -<br>
Modifying Queue Pair Attributes and Section 6.1.4 - Destroying a<br>
Queue Pair). If the Modify QP returns with an error (presumably<br>
because Work Requests are still being flushed), the Consumer<br>
must try at a later time to transition to the Idle state. The<br>
RDMA Verbs Specification 25 Apr 2003<br>
Hilland, et al. [Page 84]<br>
Consumer might arm a timeout. If the Consumer is unable to<br>
transition to the Idle state after some amount of time, it<br>
should destroy the QP (presumably because the QP can not recover<br>
from an internal error)."<br>
<br>
<br>
RDS should have wait_event_timeout() instead of just wait_event(), IMHO.<br>
<br>
<br>
<br>
Andy Grover wrote:
<blockquote cite="mid:4A1DE1AE.80705@oracle.com" type="cite">
<pre wrap="">I'm very curious what the rdsdebug() inside the ib_poll_cq loop in
rds_iw_recv_cq_comp_handler would say. Can you turn on RDS_DEBUG, or
perhaps change that rdsdebug to a printk so we just get that one line of
output? I would guess you will see completions with errors for all
outstanding recv WRs. Can you try this and see what happens?
I'm pretty sure those WRs have to be completed *somewhere*, since as you
pointed out, otherwise we'd hang indefinitely on unload.
Thanks -- Regards -- Andy
Viral Mehta wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi, I am again suspicious about RDS code.
I modified RDS code a little bit to confirm the same. I added a call
to ib_cq_poll() after rdma_disconnect() call in
rds_iw_conn_shutdown() function definition.
And as expected, I got CQ completion entry with cqe_flush status
(IB_WC_WR_FLUSH_ERR) which I am not getting in normal code which
means ib_cq_poll() is not being called when we are in disconnect path
(or when modprobe -r rds is done).
If you can shed some light I can debug more.
Viral Mehta wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Andy, Thanks for your response.
Yes, I agree with you. rdma_disconnect should free up all SQ and RQ
WQEs. Also iwarp sepc confirms the same.
So, looks like RDS has no problem. I will let you know if I find
something else.
Thanks again,
Andy Grover wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Viral Mehta wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I am, somehow, not able to forward this to netdev list.
When we run any rds-ping test, it creates a connection. It sets
up QP. And then it posts (1024 or whatever mentioned through
sysctl) recvs. Basically these are pre-post recvs. Extra recvs
will be posted again if it goes below low-watermark.
By design, Connection and all RDMA resources are destroyed only
when module is unloaded. Now in unloading process, before
destroying RDMA resources, it waits till all
send_ring/recv_ring becomes empty. =========== iw_cm.c:600:
wait_event(rds_iw_ring_empty_wait, iw_cm.c-601-
rds_iw_ring_empty(&ic->i_send_ring) && iw_cm.c-602-
rds_iw_ring_empty(&ic->i_recv_ring)); ===========
Ring empty means diff (i.e., ring->w_alloc_ctr -
ring->free_ctr) should be zero. w_alloc_ctr are number of
posted recvs and free_ctr is number of recvs consumed. Ideally,
this can never be zero as we always want some pre-posted recvs
and thus recv_ring will never be empty.
</pre>
</blockquote>
<pre wrap="">Hi Viral, sorry for the delay in responding.
I believe what happens is that the rdma_disconnect() above the
wait_event causes all the outstanding recv wrs to be completed
with an error. This causes them to be freed. They are not
refilled, and so the ring becomes empty.
Does this analysis appear correct to you?
Thanks -- Regards -- Andy
Email Scanned for Virus & Dangerous Content by :
<a class="moz-txt-link-abbreviated" href="http://www.CleanMailGateway.com">www.CleanMailGateway.com</a>
</pre>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!---->
Email Scanned for Virus & Dangerous Content by : <a class="moz-txt-link-abbreviated" href="http://www.CleanMailGateway.com">www.CleanMailGateway.com</a>
</pre>
</blockquote>
<br>
<div class="moz-signature">-- <br>
Thanks,
Viral Mehta,
Embedded Software Engineer,
<a class="moz-txt-link-abbreviated" href="http://www.einfochips.com">www.einfochips.com</a>
</div>
</body>
</html>