[rds-devel] Re: [PATCH RFC RDS/IB] version 2. increase number of unsignaled work requests

[Vladimir Sokolovsky]

On Wed, 2007-01-24 at 11:24 -0800, Zach Brown wrote:
> > This scheme would not work well for large messages because it does not
> > allow for pipelining. For example, consider a stream of messages, each
> > of which consumes the whole descriptor ring.
> >
> > In this case, you can only have a single message in flight! In  
> > contrast,
> > by signaling completions once every couple of fragments, you can start
> > pipelining the next message immediately after the first fragment  
> > (of the
> > previous message) has completed.
> 
> True, but right now RDS doesn't have multiple partial messages in  
> flight for a given connection.  The next message is only put on the  
> wire once the previous has been fully posted.  This is a conscious  
> trade-off to keep the receive path absurdly simple.  See  
> rds_recv_incoming().
> 
> So yes, I agree that we'd want to post completions regularly if we  
> were interleaving messages.  That behaviour would be introduced along  
> with the changes to the wire headers and receive path to support  
> reassembly of concurrent messages.  It doesn't seem worth the effort  
> in the current code.
> 
> I could also mention the possibility of reworking the interface  
> transports use to get sending messages so that the IB send path could  
> have an easier timing only signaling the last of multiple queued  
> small messages.  Right now the IB send path is only handed a message  
> at a time.  Frankly, I think the efficiency of small (1k to 8k?)  
> messages is a higher priority than larger messages.
> 
> - z

Although the code does not have interleaving fragments from different
messages, it seems that multiple RDS messages can be in-flight
simultaneously because rds_send_xmit() does not wait for completions: it
continues to process the next message as soon as the transport layer
(e.g., IB) reports that the messages "have been sent".
With IB, it just means that the messages have been scheduled for
transmission by the HW.

Did I miss something?


-- 
Vladimir Sokolovsky <vlad at dev.mellanox.co.il>
Mellanox Technologies Ltd.