<div dir="ltr">Hi Pierre,<div><br></div><div style>Yes, the bcopy works perfectly fine. See the log below.</div><div style>Yes, I am using Mellanox HCAs. The firmware version is 2.9.1000</div><div style>(P.S. I am running this on Debian Linux)</div>
<div style><br></div><div style><div><font face="courier new, monospace">root@ib1:~# rds-stress </font></div><div><font face="courier new, monospace">waiting for incoming connection on <a href="http://0.0.0.0:4000">0.0.0.0:4000</a></font></div>
<div><font face="courier new, monospace">accepted connection from <a href="http://192.168.16.2:52913">192.168.16.2:52913</a> on <a href="http://192.168.16.1:4000">192.168.16.1:4000</a></font></div><div><font face="courier new, monospace">negotiated options, tasks will start in 2 seconds</font></div>
<div><font face="courier new, monospace">Starting up....</font></div><div><font face="courier new, monospace">tsks tx/s rx/s tx+rx K/s mbi K/s mbo K/s tx us/c rtt us cpu %</font></div><div><font face="courier new, monospace"> 1 346 345 5562.51 0.00 0.00 34.62 2886.89 -1.00</font></div>
<div><font face="courier new, monospace"> 1 350 347 5610.66 0.00 0.00 33.72 2851.56 -1.00</font></div><div><font face="courier new, monospace"><br></font></div><div><div><font face="courier new, monospace">root@ib2:~/milind/rds-tools/rds-tools/examples# rds-stress -q 8k -s 192.168.16.1</font></div>
<div><font face="courier new, monospace">connecting to <a href="http://192.168.16.1:4000">192.168.16.1:4000</a></font></div><div><font face="courier new, monospace">negotiated options, tasks will start in 2 seconds</font></div>
<div><font face="courier new, monospace">Starting up....</font></div><div><font face="courier new, monospace">tsks tx/s rx/s tx+rx K/s mbi K/s mbo K/s tx us/c rtt us cpu %</font></div><div><font face="courier new, monospace"> 1 345 346 5567.12 0.00 0.00 1326.71 2350.08 -1.00</font></div>
<div><font face="courier new, monospace"> 1 347 349 5606.91 0.00 0.00 1326.07 2339.70 -1.00</font></div><div><font face="courier new, monospace"> 1 342 345 5535.02 0.00 0.00 1127.62 2386.98 -1.00</font></div>
<div><font face="courier new, monospace"> 1 80 81 1296.89 0.00 0.00 648.65 2219.30 -1.00</font></div></div><div><font face="courier new, monospace"><br></font></div><div style><font face="courier new, monospace">Thanks,</font></div>
<div style><font face="courier new, monospace">-Milind</font></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Mar 24, 2014 at 8:40 PM, pierre orzechowski <span dir="ltr"><<a href="mailto:pierre.e.orzechowski@oracle.com" target="_blank">pierre.e.orzechowski@oracle.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
I have seen setsockopt(RDS_FREE_MR) failing with errno 22 when we
originally introduced QoS support in rds-stress and that caused
interop issues when running against a system with an older version.<br>
I am not familiar with the <span style="font-family:arial,sans-serif;font-size:13px"><b>ENOPROTOOPT</b></span>
in this context : it does not look like this is permission or memory
registration issue<b>.</b><span style="font-family:arial,sans-serif;font-size:13px"></span><br>
<br>
A couple questions :<br>
i) Does rds-stress runs fine with bcopy ? (use -q 8k instead of -D)<br>
ii) are you using a Mellanox HCA ? firmware version ?<br>
<br>
Bang, Venkat, do you know if the git listed hereunder has 2.0.7-1.12
version of rds-tools ?<br>
<br>
Thanks,<br>
Pierre<div><div class="h5"><br>
<br>
<br>
<div>On 3/24/2014 10:42 AM, Milind Dumbare
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Pierre,
<div><br>
</div>
<div><span style="font-family:arial,sans-serif;font-size:13px">I
checked out latest sources from. </span><font face="arial,
sans-serif"><b>git://<a href="http://oss.oracle.com/git/rds-tools.git" target="_blank">oss.oracle.com/git/rds-tools.git</a></b></font><br>
</div>
<div><font face="arial, sans-serif"><b><br>
</b></font></div>
<div><font face="arial, sans-serif">It say its 2.0.7 but not
sure if its "1.10" or "1.12". Can you point me to the
sources/binaries for</font><span style="font-family:arial,sans-serif;font-size:13px"> 2.0.7-1.12</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><b>One
more thing. In following code setsockopt() returns -1 and
errno -ENOPROTOOPT. Does that mean I have setup issues
with my infiniband stack and it does not support RDMA/zero
copy.</b></span></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace">fd = socket(PF_RDS,
SOCK_SEQPACKET, 0);</font></div>
<div><font face="courier new, monospace">sin.sin_family =
AF_INET;<br>
</font></div>
<div><font face="courier new, monospace">sin.sin_addr.s_addr =
inet_addr(address);<br>
</font></div>
<div><font face="courier new, monospace">sin.sin_port =
TESTPORT;<br>
</font></div>
<div><span style="font-family:'courier new',monospace"><b>setsockopt(fd,
sol, RDS_FREE_MR, &args, sizeof(args))</b></span><br>
</div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">I
see rds-stress checks this at rds-stress.c:2802 to check if
rdma is supported or not.</span><br>
</div>
<div><span style="font-family:arial,sans-serif;font-size:13px">Do
we have any other way to check if zero copy RDMA is setup
properly.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">Thanks,</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">-Milind</span></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On Mon, Mar 24, 2014 at 6:26 PM, pierre orzechowski <span dir="ltr"><<a href="mailto:pierre.e.orzechowski@oracle.com" target="_blank">pierre.e.orzechowski@oracle.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi Milind,<br>
<br>
Do yo have the rds-tools package already to run rds-stress
?<br>
The most form of RDMA test is :<br>
server : rds-stress -r <server_IPoIB_IP><br>
client : rds-stres -r <client_IPoIB_IP> -s
<server_IPoIB_IP> -D 8k<br>
<br>
If you are using the <a href="tel:2.6.39-113.1" value="+4526391131" target="_blank">2.6.39-113.1</a> kernel onwards, I
believe there was a kernel fix that exposed a flaw in
rds-stress.<br>
"Don't check for msg_namelen for Control msgs" is fixed
in rds-tools 2.0.7-1.10.<br>
2.0.7-1.12 is recommended.<br>
<br>
Thanks,<br>
Pierre<br>
<br>
<br>
<div>On 3/24/2014 10:08 AM, Pierre Orzechowski wrote:<br>
</div>
<blockquote type="cite">
<div>Sure, wii do.</div>
<div><br>
Thanks,<br>
Pierre</div>
<div><br>
On Mar 24, 2014, at 9:17 AM, Sumanta Chatterjee <<a href="mailto:sumanta.chatterjee@oracle.com" target="_blank">sumanta.chatterjee@oracle.com</a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div> Hi Pierre,<br>
<br>
Can you respond to this question. Thanks.
<div>
<div><br>
<div><br>
<br>
-------- Original Message --------
<table border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<th align="RIGHT" nowrap valign="BASELINE">Subject: </th>
<td>Re: [rds-devel] Help with sample RDS
RDMA code by Jon</td>
</tr>
<tr>
<th align="RIGHT" nowrap valign="BASELINE">Date: </th>
<td>Mon, 24 Mar 2014 12:43:33 +0100</td>
</tr>
<tr>
<th align="RIGHT" nowrap valign="BASELINE">From: </th>
<td>Milind Dumbare <a href="mailto:milind@linux.com" target="_blank"><milind@linux.com></a></td>
</tr>
<tr>
<th align="RIGHT" nowrap valign="BASELINE">To: </th>
<td>rds-devel <a href="mailto:rds-devel@oss.oracle.com" target="_blank"><rds-devel@oss.oracle.com></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<div dir="ltr">Hello,
<div><br>
</div>
<div>Anyone actively using/developing
application on RDS? Especially the zero
copy RDMA area? I need help running basic
things like rds-stress.</div>
<div><br>
</div>
<div>Thanks,</div>
<div> -Milind</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Feb 19,
2014 at 1:40 PM, Milind Dumbare <span dir="ltr"><<a href="mailto:milind@linux.com" target="_blank">milind@linux.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:arial,sans-serif;font-size:13px">Hi,</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">I
was trying to run sample example to
to RDS RDMA communication. But I
can't make it work. Following thing
works great on without "--rdma" on
rds_tcp stack. I do have infiniband
and "rds_ping" is working on it.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">If
you have any other sample code which
can run RDS RDMA please give me the
pointers.</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">On
Server:</div>
<div style="font-family:arial,sans-serif;font-size:13px"><font face="courier new, monospace">./a.out
--rdma -s 192.168.16.2</font><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">On
client</div>
<div style="font-family:arial,sans-serif;font-size:13px"><font face="courier new, monospace">./a.out
--rdma -s 192.168.16.2 -c
192.168.16.1</font><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">And
the Client says it sent the packet
over RDMA but Server prints blank
payload and then exits. While client
is waiting on the <font face="courier new, monospace">recvmsg()</font> call.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">This
is probably because of the following
code</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<div><font face="courier new,
monospace">156: if
(msg.msg_controllen) {</font></div>
<div><font face="courier new,
monospace">157: rc
= do_rdma_read(sock, &msg,
buf);</font></div>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Server
never gets non-zero "<font face="courier new, monospace">msg.msg_controllen</font>"
and it never goes and runs "<font face="courier new, monospace">do_rdma_read()</font>".</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Please
help.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">
Thanks,</div>
<div style="font-family:arial,sans-serif;font-size:13px">-Milind</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
</div>
<br>
</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><Attached Message Part></div>
</blockquote>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>