[Ocfs2-devel] [patch 2/8] ocfs2: fix deadlock when two nodes are converting same lock from PR to EX and idletimeout closes conn

[Mark Fasheh]

On Mon, Jun 09, 2014 at 01:04:00PM -0700, Andrew Morton wrote:
> From: Tariq Saeed <tariq.x.saeed at oracle.com>
> Subject: ocfs2: fix deadlock when two nodes are converting same lock from PR to EX and idletimeout closes conn
> 
> Orabug: 18639535
> 
> Two node cluster and both nodes hold a lock at PR level and both want to
> convert to EX at the same time.  Master node 1 has sent BAST and then
> closes the connection due to idletime out.  Node 0 receives BAST, sends
> unlock req with cancel flag but gets error -ENOTCONN.  The problem is this
> error is ignored in dlm_send_remote_unlock_request() on the **incorrect**
> assumption that the master is dead.  See NOTE in comment why it returns
> DLM_NORMAL.  Upon getting DLM_NORMAL, node 0 proceeds to sends convert
> (without cancel flg) which fails with -ENOTCONN.  waits 5 sec and resends.
>  This time gets DLM_IVLOCKID from the master since lock not found in grant
> , it had been moved to converting queue in response to conv PR->EX req. 
> No way out.
> 
> Node 1 (master)				Node 0
> ==============				======
> 
> lock mode PR				PR
> 
> convert PR -> EX
> mv grant -> convert and que BAST
> ...
>                      <-------- convert PR -> EX
> convert que looks like this: ((node 1, PR -> EX) (node 0, PR -> EX))
> ...
> 			BAST (want PR -> NL)
>                      ------------------>
> ...
> idle timout, conn closed
>                                 ...
> 				In response to BAST,
> 				sends unlock with cancel convert flag
> 				gets -ENOTCONN. Ignores and
>                                 sends remote convert request
>                                 gets -ENOTCONN, waits 5 Sec, retries
> ...
> reconnects
>                    <----------------- convert req goes through on next try
> does not find lock on grant que
>                    status DLM_IVLOCKID
>                    ------------------>
> ...
> 
> No way out.  Fix is to keep retrying unlock with cancel flag until it
> succeeds or the master dies.
> 
> Signed-off-by: Tariq Saeed <tariq.x.saeed at oracle.com>
> Cc: Mark Fasheh <mfasheh at suse.com>
> Cc: Joel Becker <jlbec at evilplan.org>
> Signed-off-by: Andrew Morton <akpm at linux-foundation.org>

That looks good, thanks Tariq, in particular for the detailed explanation.

Reviewed-by: Mark Fasheh <mfasheh at suse.de>
	--Mark

--
Mark Fasheh