[rds-devel] Re: [PATCH RFC RDS/IB] version 2. increase number of unsignaled work requests

[Zach Brown]

> I tried rds_ib_ring_free(&.., completed) and my server stuck with no
> memory in this case. There is probably some memory link issue in
> rds_ib_ring_free. With rds_ib_ring_free(&ic->i_send_ring, 1) it works
> fine.

Well, that's disappointing.  I guess it can be worried about when the  
ring locks show up on profiles.  (It's not too hard to imagine  
producer and consumers each writing to their own cacheline.)

> This scheme would not work well for large messages because it does not
> allow for pipelining. For example, consider a stream of messages, each
> of which consumes the whole descriptor ring.
>
> In this case, you can only have a single message in flight! In  
> contrast,
> by signaling completions once every couple of fragments, you can start
> pipelining the next message immediately after the first fragment  
> (of the
> previous message) has completed.

True, but right now RDS doesn't have multiple partial messages in  
flight for a given connection.  The next message is only put on the  
wire once the previous has been fully posted.  This is a conscious  
trade-off to keep the receive path absurdly simple.  See  
rds_recv_incoming().

So yes, I agree that we'd want to post completions regularly if we  
were interleaving messages.  That behaviour would be introduced along  
with the changes to the wire headers and receive path to support  
reassembly of concurrent messages.  It doesn't seem worth the effort  
in the current code.

I could also mention the possibility of reworking the interface  
transports use to get sending messages so that the IB send path could  
have an easier timing only signaling the last of multiple queued  
small messages.  Right now the IB send path is only handed a message  
at a time.  Frankly, I think the efficiency of small (1k to 8k?)  
messages is a higher priority than larger messages.

- z