On Nov 13, 2007 6:25 PM, Olaf Kirch <<a href="mailto:olaf.kirch@oracle.com">olaf.kirch@oracle.com</a>> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
From: Olaf Kirch <<a href="mailto:olaf.kirch@oracle.com">olaf.kirch@oracle.com</a>><br><br>[EXPERIMENTAL] RDS/IB: add connection level flow control<br><br>Rather than relying on RNR NAKs to fix recv queue overruns for us,
<br>this patch introduces flow control on the RC connection.<br><br>This flow control mechanism is based on the count of posted receive<br>buffers.<br><br>We add an IB specific extension to the RDS header, which includes<br>
the following two 32bit quantities:<br><br> - a sequence number. Each IB work request has a distinct number,<br> even for fragments of the same RDS message.<br><br> This is different from the RDS sequence number space, which
<br> may not be consecutive, and which uses one sequence number for the<br> entirety of an RDS message.<br><br> - a window sequence number, denoting the end of the receive window.<br> Essentially, this is the last sequence number received from the
<br> peer, plus the number of posted receive buffers.<br><br> The peer should not transmit packets with a sequence number equal<br> or greater to this value (modulo wrap-around).<br><br>These quantities are transmitted with every packet. When the sender
<br>notices that it is running low on send credits, it explicitly requests<br>an ACK packet (which would carry updated window information).</blockquote><div><br>I think this flow can be simplified to have the --receiver-- send credit update packet
<br>when it sees that the credits at the sender are below some low water mark.<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
When we find we're unable to refill the recv queue from the recv CQ<br>completion handler, and refilling is deferred to the worker thread,<br>we also send an ACK packet after refilling the queue.<br><br>In practice, it would probably be sufficient to make the sequence and window
<br>quantities 16 bits wide, unless we expect to deal with receive queues of 32000 entries.<br><br></blockquote></div><br>Also: with this approach, can we remove the ACK packets from RDS/IB altogether? since you know what
<br>was the last sequence number, and the sequence number is incremental, IB RC in order delivery<br>guarantees that all the packets (RDS fragments) up to this number are in memory.<br><br>Second point, I have failed so far to fully understand the RDS congestion management, however, it is possible
<br>that with the flow control protocol, there's no need to have congestion control wire protocol and it can be based<br>on the information present in the credit management?<br><br>Or.<br>