<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Aaaarrrggghhh – this is a bit of a hack I’m afraid to compensate for a feature (currently) only of the ptllnd.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The issue stems from the fact that LNDs _<i>must</i>_ keep sufficient numbers of buffers posted at all times to ensure they remain responsive to their peers and play their part in the buffer credits protocol.
All other LNDs have buffer size == message size – i.e. they post 1 buffer for every buffer credit their peers have, but the ptllnd posts large buffers that are expected to receive many messages. This means that a failure to handle messages eagerly by upper
levels could leave a whole ptllnd buffer of ‘n’ messages pinned and therefore out of service. This would violate the credit protocol and lead to deadlock. So the ptllnd has an ‘eager receive’ method to copy messages that can’t be handled immediately into
a temporary buffer to avoid this problem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">IIRC, we had to use this feature in the ptllnd because if we’d posted 1 buffer for every message we could receive at any time, we’d have run out of portals MDs. Allowing multiple incoming messages to share a
single buffer has some obvious benefits (LNET does this too!), but IMHO it can be a bit of a two-edged sword with some non-obvious consequences. For example, buffers must be considered full when there isn’t enough space left to receive the longest message
a peer might send. If you have considerable variation in message length and buffers are only sized large enough for 1 maximum message, you end up with significant buffer underutilisation. I believe this was the root cause of recent network hangs at scale
on Blue Waters after the maximum MDS request size was increased significantly to permit wide striping.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> Eric<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> lustre-devel-bounces@lists.lustre.org [mailto:lustre-devel-bounces@lists.lustre.org]
<b>On Behalf Of </b>Chuck Fossen<br>
<b>Sent:</b> Thursday, September 27, 2012 11:36 PM<br>
<b>To:</b> lustre-devel@lists.lustre.org<br>
<b>Subject:</b> [Lustre-devel] lnet eager receive path<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US">LNET experts,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We are currently using the eager receive path to buffer rx messages removing them from the wire so as to not stall the network.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Is this necessary for the proper operation of LNET? I don’t see that any of the other LNDs use the eager receive path.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Is there some history as to why the eager receive was added?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks for any input.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Chuck Fossen<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Cray Inc.<o:p></o:p></span></p>
</div>
<p>---------------------------------------------------------------------<br>
Intel Corporation (UK) Limited<br>
Registered No. 1134945 (England)<br>
Registered Office: Pipers Way, Swindon SN3 1RJ<br>
VAT No: 860 2173 47</p>
<p>This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.</p></body>
</html>