[rds-devel] Has anyone tried RDS on IA64 Linux?

[Or Gerlitz]

Olaf Kirch wrote:
>> It seems that now we have two issues
>> (1) code correctness for which the dma_sync calls would serve
>
> Here's a draft patch, uncompiled. Would this work?
I think there are some errors there, see some comments below

Or

> --- linux-2.6.orig/net/rds/ib_rdma.c
> +++ linux-2.6/net/rds/ib_rdma.c
>  
> +void rds_ib_dma_sync_sg(struct ib_device *dev,
> +		struct scatterlist *sg, unsigned int sg_dma_len, int direction)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < sg_dma_len; ++i) {
> +		unsigned int dma_len = ib_sg_dma_len(dev, &sg[i]);
> +		u64 dma_addr = ib_sg_dma_address(dev, &sg[i]);
> +
> +		ib_dma_sync_single_for_device(dev,
> +				ib_sg_dma_address(dev, &sg[i]),
> +				ib_sg_dma_len(dev, &sg[i]),
> +				direction);
> +	}
this function always does _sync_for_device, where its callers sometimes 
want it to sync for the CPU. Also why not use ib_dma_sync_sg_for_device() ?

> @@ -268,7 +284,7 @@ static int rds_ib_map_fmr(struct rds_ib_
>  	for (i = 0; i < sg_dma_len; ++i) {
>  		unsigned int dma_len = ib_sg_dma_len(dev, &scat[i]);
>  		u64 dma_addr = ib_sg_dma_address(dev, &scat[i]);
> -	
> +
empty lines snipped in... and also in the next change
> @@ -302,7 +318,9 @@ static int rds_ib_map_fmr(struct rds_ib_
>  			dma_pages[page_cnt++] = 
>  				(dma_addr & rds_ibdev->fmr_page_mask) + j;
>  	}
> -				
> +
> +	rds_ib_dma_sync_sg(dev, scat, sg_dma_len, DMA_TO_DEVICE);
earlier in this function you did a dma mapping, which also does sync for 
device, unless the bidi directive does not let it do so correctly?
> --- linux-2.6.orig/net/rds/ib_send.c
> +++ linux-2.6/net/rds/ib_send.c
> @@ -53,6 +53,15 @@ void rds_ib_send_unmap_rm(struct rds_ib_
>  
>  	/* raise rdma completion hwm */
>  	if (rm->m_rdma_op && success) {
> +		struct rds_rdma_op *op = rm->m_rdma_op;
> +
> +		/* If this was a RDMA READ, make sure the CPU sees all
> +		 * the updates. */
> +		if (!op->r_write)
> +			rds_ib_dma_sync_sg(ic->i_cm_id->device,
> +					op->r_sg, op->r_count,
> +					DMA_FROM_DEVICE);
here you want a sync_for_cpu where  rds_ib_dma_sync_sg does sync_for_device