[rds-devel] dma_sync [was: Has anyone tried RDS on IA64 Linux?]

Olaf Kirch olaf.kirch at oracle.com
Wed Apr 30 01:46:20 PDT 2008


On Tuesday 29 April 2008 14:25:25 Olaf Kirch wrote:
> I don't think a dma unmap is really required. But you're right,
> this code needs dma_sync_* calls in a number of places.

Rick raised an interesting question yesterday. Do normal send/recv
operations need dma_sync as well? My immediate reaction was "of
course not" but then I gave it a second thought.

RDS headers are in DMA consistent memory, so there's no explicit
dma_sync required. Correct?

When sending data, dma_map(TO_DEVICE) will implicitly do the sync.

When setting up WRs for receiving data, we use dma_map_page(FROM_DEVICE)
to map one page, and then we carve it up into fragment sized chunks
(currently 2K) and post WRs with these chunks. And here's the rub.

When we have posted two fragment sized receive buffers, and *one*
packet comes in, we will just grab the completed WR, and queue up
the fragment on the destination socket.

If a *second* fragment arrives just in time, we unmap the page, and
that should implicitly perform a dma_sync(FROM_DEVICE). However, if
there is no second fragment, we don't sync.

Given that logic, we would really need to sync explicitly when
we receive a RDS packet that does not use the last fragment of the
mapped page, correct? Proposed patch below...

Now the practical value of this question is that somebody is seeing
odd, reproduceable corruptions in an Oracle certification run. I'm
not sure which hardware this is on, so I can't say whether it may
be related or not. But for instance with swiotlb this problem would
certainly show up at some point.

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax
---------------
 net/rds/ib_recv.c |    4 ++++
 1 file changed, 4 insertions(+)

Index: ofa_kernel-1.3/net/rds/ib_recv.c
===================================================================
--- ofa_kernel-1.3.orig/net/rds/ib_recv.c
+++ ofa_kernel-1.3/net/rds/ib_recv.c
@@ -75,6 +75,10 @@ static void rds_ib_recv_unmap_page(struc
                ib_dma_unmap_page(ic->i_cm_id->device,
                               recv->r_sge[0].addr - recv->r_frag->f_offset,
                               PAGE_SIZE, DMA_FROM_DEVICE);
+       else
+               ib_dma_sync_single_for_device(ic->i_cm_id->device,
+                               recv->r_sge[0].addr - recv->r_frag->f_offset,
+                               PAGE_SIZE, DMA_FROM_DEVICE);
        recv->r_sge[0].addr = 0;
 }





More information about the rds-devel mailing list