[rds-devel] [PATCH 2/2] RDS/IB: add connection level flow control

Or Gerlitz or.gerlitz at gmail.com
Wed Nov 14 02:23:01 PST 2007


On Nov 13, 2007 6:25 PM, Olaf Kirch <olaf.kirch at oracle.com> wrote:

> From: Olaf Kirch <olaf.kirch at oracle.com>
>
> [EXPERIMENTAL] RDS/IB: add connection level flow control
>
> Rather than relying on RNR NAKs to fix recv queue overruns for us,
> this patch introduces flow control on the RC connection.
>
> This flow control mechanism is based on the count of posted receive
> buffers.
>
> We add an IB specific extension to the RDS header, which includes
> the following two 32bit quantities:
>
>  -      a sequence number. Each IB work request has a distinct number,
>        even for fragments of the same RDS message.
>
>        This is different from the RDS sequence number space, which
>        may not be consecutive, and which uses one sequence number for the
>        entirety of an RDS message.
>
>  -      a window sequence number, denoting the end of the receive window.
>        Essentially, this is the last sequence number received from the
>        peer, plus the number of posted receive buffers.
>
>        The peer should not transmit packets with a sequence number equal
>        or greater to this value (modulo wrap-around).
>
> These quantities are transmitted with every packet. When the sender
> notices that it is running low on send credits, it explicitly requests
> an ACK packet (which would carry updated window information).


I think this flow can be simplified to have the --receiver-- send credit
update packet
when it sees that the credits at the sender are below some low water mark.

When we find we're unable to refill the recv queue from the recv CQ
> completion handler, and refilling is deferred to the worker thread,
> we also send an ACK packet after refilling the queue.
>
> In practice, it would probably be sufficient to make the sequence and
> window
> quantities 16 bits wide, unless we expect to deal with receive queues of
> 32000 entries.
>
>
Also: with this approach, can we remove the ACK packets from RDS/IB
altogether? since you know what
was the last sequence number, and the sequence number is incremental, IB RC
in order delivery
guarantees that  all the  packets (RDS fragments) up to this number are in
memory.

Second point, I have failed so far to fully understand the RDS congestion
management, however, it is possible
that with the flow control protocol, there's no need to have congestion
control wire protocol and it can be based
on the information present in the credit management?

Or.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/rds-devel/attachments/20071114/64db73f0/attachment.html


More information about the rds-devel mailing list