[rds-devel] Help with sample RDS RDMA code by Jon

Mon Mar 24 13:08:37 PDT 2014

Hi Pierre,

Yes, the bcopy works perfectly fine. See the log below.
Yes, I am using Mellanox HCAs. The firmware version is 2.9.1000
(P.S. I am running this on Debian Linux)

root at ib1:~# rds-stress
waiting for incoming connection on 0.0.0.0:4000
accepted connection from 192.168.16.2:52913 on 192.168.16.1:4000
negotiated options, tasks will start in 2 seconds
Starting up....
tsks   tx/s   rx/s  tx+rx K/s    mbi K/s    mbo K/s tx us/c   rtt us cpu %
   1    346    345    5562.51       0.00       0.00   34.62  2886.89 -1.00
   1    350    347    5610.66       0.00       0.00   33.72  2851.56 -1.00

root at ib2:~/milind/rds-tools/rds-tools/examples# rds-stress -q 8k -s
192.168.16.1
connecting to 192.168.16.1:4000
negotiated options, tasks will start in 2 seconds
Starting up....
tsks   tx/s   rx/s  tx+rx K/s    mbi K/s    mbo K/s tx us/c   rtt us cpu %
   1    345    346    5567.12       0.00       0.00 1326.71  2350.08 -1.00
   1    347    349    5606.91       0.00       0.00 1326.07  2339.70 -1.00
   1    342    345    5535.02       0.00       0.00 1127.62  2386.98 -1.00
   1     80     81    1296.89       0.00       0.00  648.65  2219.30 -1.00

Thanks,
-Milind

On Mon, Mar 24, 2014 at 8:40 PM, pierre orzechowski <
pierre.e.orzechowski at oracle.com> wrote:

>  I have seen setsockopt(RDS_FREE_MR) failing with errno 22 when we
> originally introduced QoS support in rds-stress and that caused interop
> issues when running against a system with an older version.
> I am not familiar with the *ENOPROTOOPT* in this context : it does not
> look like this is permission or memory registration issue*.*
>
> A couple questions :
> i) Does rds-stress runs fine with bcopy ? (use -q 8k instead of -D)
> ii) are you using a Mellanox HCA ? firmware version  ?
>
> Bang, Venkat, do you know if the git listed hereunder has 2.0.7-1.12
> version of rds-tools ?
>
> Thanks,
> Pierre
>
>
>
> On 3/24/2014 10:42 AM, Milind Dumbare wrote:
>
> Hi Pierre,
>
>  I checked out latest sources from. *git://oss.oracle.com/git/rds-tools.git
> <http://oss.oracle.com/git/rds-tools.git>*
>
>  It say its 2.0.7 but not sure if its "1.10" or "1.12". Can you point me
> to the sources/binaries for 2.0.7-1.12
>
>  *One more thing. In following code setsockopt() returns -1 and errno
> -ENOPROTOOPT. Does that mean I have setup issues with my infiniband stack
> and it does not support RDMA/zero copy.*
>
>  fd = socket(PF_RDS, SOCK_SEQPACKET, 0);
> sin.sin_family = AF_INET;
>  sin.sin_addr.s_addr = inet_addr(address);
>  sin.sin_port = TESTPORT;
>  *setsockopt(fd, sol, RDS_FREE_MR, &args, sizeof(args))*
>
>  I see rds-stress checks this at rds-stress.c:2802 to check if rdma is
> supported or not.
>  Do we have any other way to check if zero copy RDMA is setup properly.
>
>  Thanks,
> -Milind
>
>
>  On Mon, Mar 24, 2014 at 6:26 PM, pierre orzechowski <
> pierre.e.orzechowski at oracle.com> wrote:
>
>>  Hi Milind,
>>
>> Do yo have the rds-tools package already to run rds-stress ?
>> The most form of RDMA test is :
>> server : rds-stress -r <server_IPoIB_IP>
>> client : rds-stres -r <client_IPoIB_IP> -s <server_IPoIB_IP> -D 8k
>>
>> If you are using the 2.6.39-113.1 kernel onwards, I believe there was a
>> kernel fix that exposed a flaw in rds-stress.
>>  "Don't check for msg_namelen for Control msgs" is fixed in rds-tools
>> 2.0.7-1.10.
>>  2.0.7-1.12 is recommended.
>>
>> Thanks,
>> Pierre
>>
>>
>> On 3/24/2014 10:08 AM, Pierre Orzechowski wrote:
>>
>> Sure, wii do.
>>
>> Thanks,
>> Pierre
>>
>> On Mar 24, 2014, at 9:17 AM, Sumanta Chatterjee <
>> sumanta.chatterjee at oracle.com> wrote:
>>
>>   Hi Pierre,
>>
>> Can you respond to this question.  Thanks.
>>
>>
>>
>> -------- Original Message --------  Subject: Re: [rds-devel] Help with
>> sample RDS RDMA code by Jon  Date: Mon, 24 Mar 2014 12:43:33 +0100  From:
>> Milind Dumbare <milind at linux.com> <milind at linux.com>  To: rds-devel
>> <rds-devel at oss.oracle.com> <rds-devel at oss.oracle.com>
>>
>> Hello,
>>
>>  Anyone actively using/developing application on RDS? Especially the
>> zero copy RDMA area? I need help running basic things like rds-stress.
>>
>>  Thanks,
>>  -Milind
>>
>>
>> On Wed, Feb 19, 2014 at 1:40 PM, Milind Dumbare <milind at linux.com> wrote:
>>
>>>  Hi,
>>>
>>>  I was trying to run sample example to to RDS RDMA communication. But I
>>> can't make it work. Following thing works great on without "--rdma" on
>>> rds_tcp stack. I do have infiniband and "rds_ping" is working on it.
>>>
>>>  If you have any other sample code which can run RDS RDMA please give
>>> me the pointers.
>>>
>>>  On Server:
>>> ./a.out --rdma -s 192.168.16.2
>>>
>>>  On client
>>> ./a.out --rdma -s 192.168.16.2 -c 192.168.16.1
>>>
>>>  And the Client says it sent the packet over RDMA but Server prints
>>> blank payload and then exits. While client is waiting on the recvmsg()
>>>  call.
>>>
>>>  This is probably because of the following code
>>>
>>>  156: if (msg.msg_controllen) {
>>> 157:               rc = do_rdma_read(sock, &msg, buf);
>>>
>>>  Server never gets non-zero "msg.msg_controllen" and it never goes and
>>> runs "do_rdma_read()".
>>>
>>>  Please help.
>>>
>>>  Thanks,
>>> -Milind
>>>
>>
>>
>>
>>    <Attached Message Part>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/rds-devel/attachments/20140324/5b6a625d/attachment.html 