<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1064524053;
mso-list-type:hybrid;
mso-list-template-ids:484845400 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"> Pardon my 1<sup>st</sup> time post omissions which I may be unaware of.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">RHEL 6.5 (2.6.32-431-29.2) provided Infiniband stack over Mellanox IB hardware (mlx4), distro provided RDS stack.<o:p></o:p></p>
<p class="MsoNormal">rds-sample.c (rds-tools-2.0.6) works as expected for non-RDMA operations.<o:p></o:p></p>
<p class="MsoNormal">RDMA ops never complete?<o:p></o:p></p>
<p class="MsoNormal">With –rr (RDMA-read) the client successfully sends the RDS_CMSG_RDMA_MAP control message over an RDS socket bound to the IPoIB IPv4 address and waits for server RDMA ack.<o:p></o:p></p>
<p class="MsoNormal">Server correctly receives RDS_CMSG_RDMA_MAP message from an rds socket bound to the IPoIB IF IPv4 address (local loopback) and returns RDS_CMSG_RDMA_DEST control message with RDMA cookie as CMSG_DATA(msg) in response to readmsg().<o:p></o:p></p>
<p class="MsoNormal">Server casts CMSG_DATA(msg) ptr to ‘(struct rds_rdma_args*)’ overlaying cookie and fills in the remaining struct rds_rdma_args fields describing the RDMA-read op.<br>
rds_rdma_args.flags = RDS_RDMA_FENCE | RDS_RDMA_NOTIFY_ME;<o:p></o:p></p>
<p class="MsoNormal">Server calls sendmsg() to post the RDMA-read operation and gets a successful sendmsg() return.<o:p></o:p></p>
<p class="MsoNormal">Server issues recvmsg(sock, msg, MSG_DONTWAIT) same msg struct as used in previous sendmsg().<o:p></o:p></p>
<p class="MsoNormal">Server loops forever in<o:p></o:p></p>
<p class="MsoNormal"> do {<o:p></o:p></p>
<p class="MsoNormal"> rc = recvmsg(sock, msg, MSG_DONTWAIT);<o:p></o:p></p>
<p class="MsoNormal"> } while (rc < 0 && errno == EAGAIN);<o:p></o:p></p>
<p class="MsoNormal">When a bailout condition is inserted into the loop is encountered (waited 7 seconds) the expected RDMA-read data has not appeared; suggests the RDMA-read op was never posted or completed successfully.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Questions<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Does anyone have a pointer to a known working version of rds-sample.c for RDMA operations?<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Thoughts on why/how to move beyond the stalling do {} while()?
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Stan.<o:p></o:p></p>
</div>
</body>
</html>