[rds-devel] RDS update

[Andy Grover]

Hi all,

mainline:

up to date, but not too relevant until enterprise distros move to 2.6.30+.

OFED 1.4:

critical bugfixes only

OFED 1.5:

RDS done for 1.5 except for re-adding RDS's pf_hack proc interface to
find out PF_RDS (got lost when ofed 1.5 rebased to mainline, which
doesn't have this.)

OFED 1.6: in development: see ~agrover/ofed_1_5/future/20091123 git tree
on openfabrics.

There do seem to be a constant trickle of rds bugs coming in, but I've
pushed those to the side for now, otherwise RDS would never make any
forward progress. Please still feel free to send me patches :) and I'll
carve out a big chunk of some at some point soon to deal with them en masse.

My current git branch so far is mainly changes to cut down on memory
allocs, and integrate rdma a little more. net/rds/rdma.h is folded into
rds.h, and m_rdma_op is directly embedded in rds_message. I had to add a
little complexity to make this work, but the end result is one fewer
kmalloc per rdma needed.

The next spot I'm looking at is allocations for the kernel data buffer
on send. RDS currently has a fancy mechanism (in page.c) to alloc a page
on each CPU and then divvy it up for data buffers until it is used up. I
have two possible thoughts:

1) We can get rid of this and just use kmalloc for small allocations,
alloc pages for larger, and get rid of all this code! Linux kmalloc
should be pretty smart, and I believe it is already doing per-cpu pools,
but I am not positive. Anyone know?

2) If we decide to keep this code or not, for small datagrams we can use
either page_remainder_alloc or kmalloc to allocate additional contiguous
space for the struct rds_message, since both objects have the same
lifetime. This should save another kmalloc when data length is nonzero.

This direction is somewhat speculative and not necessarily how future
rds will end up. I'm going on a bit of a hunch that if I rework how some
parts of rds interact, it will make further development (like dropping
in atomics support) easier and less likely to cause regressions. So far
so good...

Regards -- Andy