[rds-devel] [PATCH net v3] rds: ib: Increment i_fastreg_wrs before bailing out

[Allison Henderson]

On Wed, 2025-09-10 at 13:04 +0200, Håkon Bugge wrote:
> We need to increment i_fastreg_wrs before we bail out from
> rds_ib_post_reg_frmr().
> 
> We have a fixed budget of how many FRWR operations that can be
> outstanding using the dedicated QP used for memory registrations and
> de-registrations. This budget is enforced by the atomic_t
> i_fastreg_wrs. If we bail out early in rds_ib_post_reg_frmr(), we will
> "leak" the possibility of posting an FRWR operation, and if that
> accumulates, no FRWR operation can be carried out.
Hi Håkon,

This sounds much clearer, thank you!

> 
> Fixes: 1659185fb4d0 ("RDS: IB: Support Fastreg MR (FRMR) memory registration mode")
> Fixes: 3a2886cca703 ("net/rds: Keep track of and wait for FRWR segments in use upon shutdown")
> Cc: stable at vger.kernel.org
> Signed-off-by: Håkon Bugge <haakon.bugge at oracle.com>
> 
> ---
> 
> v2 -> v3:
>    * Amended commit message
>    * Removed indentation of this section
>    * Fixing error path from ib_post_send()
> 
> v1 -> v2: Added Cc: stable at vger.kernel.org
> ---
>  net/rds/ib_frmr.c | 20 ++++++++++++--------
>  1 file changed, 12 insertions(+), 8 deletions(-)
> 
> diff --git a/net/rds/ib_frmr.c b/net/rds/ib_frmr.c
> index 28c1b00221780..395a99b5a65ca 100644
> --- a/net/rds/ib_frmr.c
> +++ b/net/rds/ib_frmr.c
> @@ -133,12 +133,15 @@ static int rds_ib_post_reg_frmr(struct rds_ib_mr *ibmr)
>  
>  	ret = ib_map_mr_sg_zbva(frmr->mr, ibmr->sg, ibmr->sg_dma_len,
>  				&off, PAGE_SIZE);
> -	if (unlikely(ret != ibmr->sg_dma_len))
> -		return ret < 0 ? ret : -EINVAL;
> +	if (unlikely(ret != ibmr->sg_dma_len)) {
> +		ret = ret < 0 ? ret : -EINVAL;
> +		goto out_inc;
> +	}
>  
> -	if (cmpxchg(&frmr->fr_state,
> -		    FRMR_IS_FREE, FRMR_IS_INUSE) != FRMR_IS_FREE)
> -		return -EBUSY;
> +	if (cmpxchg(&frmr->fr_state, FRMR_IS_FREE, FRMR_IS_INUSE) != FRMR_IS_FREE) {
> +		ret = -EBUSY;
> +		goto out_inc;
> +	}
>  
>  	atomic_inc(&ibmr->ic->i_fastreg_inuse_count);
>  
> @@ -166,11 +169,10 @@ static int rds_ib_post_reg_frmr(struct rds_ib_mr *ibmr)
>  		/* Failure here can be because of -ENOMEM as well */
>  		rds_transition_frwr_state(ibmr, FRMR_IS_INUSE, FRMR_IS_STALE);
>  
> -		atomic_inc(&ibmr->ic->i_fastreg_wrs);
>  		if (printk_ratelimit())
>  			pr_warn("RDS/IB: %s returned error(%d)\n",
>  				__func__, ret);
> -		goto out;
Just one nit:  This was the only place the out label was used isnt it?  If so, lets go ahead and clear it out.  

> +		goto out_inc;
>  	}
>  
>  	/* Wait for the registration to complete in order to prevent an invalid
> @@ -178,9 +180,11 @@ static int rds_ib_post_reg_frmr(struct rds_ib_mr *ibmr)
>  	 * being accessed while registration is still pending.
>  	 */
>  	wait_event(frmr->fr_reg_done, !frmr->fr_reg);
> -
>  out:
With that fixed, you can go ahead and add my rvb:
Reviewed-by: Allison Henderson <allison.henderson at oracle.com>

Thank you!

> +	return ret;
>  
> +out_inc:
> +	atomic_inc(&ibmr->ic->i_fastreg_wrs);
>  	return ret;
>  }
>