<div dir="ltr">Hi Smith,<div><br></div><div>Appearlty the rds-sample.c is broken for current stack. Here is the fix I got which should make rds-sample.c work.</div><div><b><br></b></div><div>
<p class=""><b> do {</b></p><p class=""><b> rc = recvmsg(sock, &msg, 0);</b></p>
<p class=""><b> if (rc < 0) {</b></p>
<p class=""><b> printf("%s: Error receiving message: %d %d\n", __func__, rc, errno);</b></p>
<p class=""><b> goto out2;</b></p>
<p class=""><b> }</b></p>
<p class=""><b> </b></p>
<p class=""><b> if (flags & VERBOSE_FLAG)</b></p>
<p class=""><b> printf("Received %s packet %d of len %d, cmsg len %d, on port %d\n",</b></p>
<p class=""><b> msg.msg_controllen ? "<span class="">RDS</span> RDMA" : "<span class="">RDS</span>",</b></p>
<p class=""><b> count,</b></p>
<p class=""><b> (uint32_t) iov[0].iov_len,</b></p>
<p class=""><b> (uint32_t) msg.msg_controllen,</b></p>
<p class=""><b> din.sin_port);</b></p>
<p class=""><b>+++ if (msg.msg_namelen == 0)</b></p>
<p class=""><b>+++ continue;</b></p>
<p class=""><b>+++ msg.msg_namelen = sizeof(din);</b></p><p class=""><b><br></b></p><p class="">-Milind</p></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 17, 2014 at 11:33 PM, Smith, Stan <span dir="ltr"><<a href="mailto:stan.smith@intel.com" target="_blank">stan.smith@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal">Hello,<u></u><u></u></p>
<p class="MsoNormal"> Pardon my 1<sup>st</sup> time post omissions which I may be unaware of.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">RHEL 6.5 (2.6.32-431-29.2) provided Infiniband stack over Mellanox IB hardware (mlx4), distro provided RDS stack.<u></u><u></u></p>
<p class="MsoNormal">rds-sample.c (rds-tools-2.0.6) works as expected for non-RDMA operations.<u></u><u></u></p>
<p class="MsoNormal">RDMA ops never complete?<u></u><u></u></p>
<p class="MsoNormal">With –rr (RDMA-read) the client successfully sends the RDS_CMSG_RDMA_MAP control message over an RDS socket bound to the IPoIB IPv4 address and waits for server RDMA ack.<u></u><u></u></p>
<p class="MsoNormal">Server correctly receives RDS_CMSG_RDMA_MAP message from an rds socket bound to the IPoIB IF IPv4 address (local loopback) and returns RDS_CMSG_RDMA_DEST control message with RDMA cookie as CMSG_DATA(msg) in response to readmsg().<u></u><u></u></p>
<p class="MsoNormal">Server casts CMSG_DATA(msg) ptr to ‘(struct rds_rdma_args*)’ overlaying cookie and fills in the remaining struct rds_rdma_args fields describing the RDMA-read op.<br>
rds_rdma_args.flags = RDS_RDMA_FENCE | RDS_RDMA_NOTIFY_ME;<u></u><u></u></p>
<p class="MsoNormal">Server calls sendmsg() to post the RDMA-read operation and gets a successful sendmsg() return.<u></u><u></u></p>
<p class="MsoNormal">Server issues recvmsg(sock, msg, MSG_DONTWAIT) same msg struct as used in previous sendmsg().<u></u><u></u></p>
<p class="MsoNormal">Server loops forever in<u></u><u></u></p>
<p class="MsoNormal"> do {<u></u><u></u></p>
<p class="MsoNormal"> rc = recvmsg(sock, msg, MSG_DONTWAIT);<u></u><u></u></p>
<p class="MsoNormal"> } while (rc < 0 && errno == EAGAIN);<u></u><u></u></p>
<p class="MsoNormal">When a bailout condition is inserted into the loop is encountered (waited 7 seconds) the expected RDMA-read data has not appeared; suggests the RDMA-read op was never posted or completed successfully.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Questions<u></u><u></u></p>
<p><u></u><span>1)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Does anyone have a pointer to a known working version of rds-sample.c for RDMA operations?<u></u><u></u></p>
<p><u></u><span>2)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Thoughts on why/how to move beyond the stalling do {} while()?
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thank you,<span class="HOEnZb"><font color="#888888"><u></u><u></u></font></span></p><span class="HOEnZb"><font color="#888888">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Stan.<u></u><u></u></p>
</font></span></div>
</div>
<br>_______________________________________________<br>
rds-devel mailing list<br>
<a href="mailto:rds-devel@oss.oracle.com">rds-devel@oss.oracle.com</a><br>
<a href="https://oss.oracle.com/mailman/listinfo/rds-devel" target="_blank">https://oss.oracle.com/mailman/listinfo/rds-devel</a><br></blockquote></div><br></div>