[rds-devel] RDS-HA

[Or Gerlitz]

Zach Brown wrote:
>> With Oracle/RDS being an open source project, lets move to do all
>> rds devel related communication over this list.
> 
> Indeed, we've been moving more aggressively in that direction.

OK

> If the IB transport sees an error on the connection it queues shutdown
> work on the connection thread.  That thread calls back into the IB
> transport to shutdown the transport-specific bits of the connection via
> rds_ib_conn_shutdown().  Then later the thread will try to re-establish
> the connection by calling rds_ib_conn_connect().
> 
> rds_ib_conn_connect() uses rdma_resolve_addr() to kick off the
> connection establishment.  So as long as Or's IPoIB HA patches result in
> completion errors and rdma_resolve_addr() pointing at the now-active
> side of a failover then RDS will be able to bring up a new RC QP and
> start sending messages down it again.
> 
> The passive listening side of connection re-establishment can be seen in
> rds_ib_cm_handle_connect().  RDS maintains only one transport connection
> for a given peer IP address.  If it sees an incoming connection request
> that already has an established connection it tears down the existing
> connection and tries reconnecting again.  It's keeping the number of
> moving pieces minimized by handling the stale connection case like the
> case where both nodes race to establish connections.

One thing to note here is that the new connection might be established 
over a different rdma/dma device (when ib0 and ib1 map to different 
HCAs). So the RDS code must not assume it can always recycle resources 
(eg PD/CQ/QP/DMA-MR) created for the old IB connection and use them with 
the new IB connection. Looking at the code, i see that rds_ib_setup_qp 
is called for each new connection and that it allocates everything from 
scratch


> Or, if you're about to send a refresh of the patch series to
> openib-devel maybe we can focus on working with those?

Just sent them out today...

Or.