<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Fair enough.
<div class=""><br class="">
</div>
<div class="">The code I’m talking about, that allows concurrent_sends to be set less than peer_credits, was apparently implemented as a “workaround” for bug 15983.</div>
<div class=""><br class="">
</div>
<div class="">Chris Horn</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Aug 19, 2015, at 2:54 PM, Alexey Lyashkov <<a href="mailto:alexey.lyashkov@seagate.com" class="">alexey.lyashkov@seagate.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">In credit based theory - credits should never to be less zero, as credit is resources to send. If credits set to zero we just stop send.
<div class="">My assumption base on book "Traffic Management for High-speed Networks International Science Lecture Series ; 4th Lecture".</div>
<div class=""><br class="">
</div>
<div class=""><span style="font-size:12.8000001907349px" class="">> This is counterintuitive, and I don’t understand why the code was written this way.</span><br class="">
</div>
<div class=""><span style="font-size:12.8000001907349px" class="">if i understand correctly it code was originally written for a direct connected network, but credits distribution model forget to change when lnet routers was created. So we continue to count
credits in peers base instead of distribute credits only in next hop base where a router should distribute a provided credits over connected clients. It provide a router overloading with huge parallel sends and new limit was added.</span></div>
<div class=""><span style="font-size:12.8000001907349px" class=""><br class="">
</span></div>
<div class=""><br class="">
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Wed, Aug 19, 2015 at 10:38 PM, Chris Horn <span dir="ltr" class="">
<<a href="mailto:hornc@cray.com" target="_blank" class="">hornc@cray.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class=""><span class="">
<blockquote type="cite" class="">But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work
in that situation..</blockquote>
<div class=""><br class="">
</div>
</span>
<div class="">Well, I think that depends on how the flow control is supposed to work. The code definitely prevents *more* than peer_credits sends to a single peer. And negative number of credits indeed indicates a queue which, I think, is fine. What is non-obvious
is that it appears that the code may prevent fewer than peer_credits sends to a single peer, even if there aren’t any other outstanding sends to other peers. i.e. we hit the concurrent_sends limit before we hit the peer_credits limit. This is counterintuitive,
and I don’t understand why the code was written this way.</div>
<span class="HOEnZb"><font color="#888888" class="">
<div class=""><br class="">
</div>
<div class="">Chris Horn</div>
</font></span>
<div class="">
<div class="h5">
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Aug 19, 2015, at 1:58 PM, Alexey Lyashkov <<a href="mailto:alexey.lyashkov@seagate.com" target="_blank" class="">alexey.lyashkov@seagate.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
Chris,
<div class=""><br class="">
</div>
<div class="">In current code it's may be same. Let me finish some bugs before look to code. But with "good" code - you should have a different cost (credits count) for different messages. 1Mb bulk transfer should eats more credits than simple 4kb message.
So you "should" have a two limits first one for parallel send with zero/low cost messages and second one for large messages.</div>
<div class="">But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work in that situation..</div>
<div class=""><br class="">
</div>
<div class="">It's know bug for me but need large time to create a network model to create a correct credits distribution.</div>
<div class=""><br class="">
</div>
</div>
<div class="gmail_extra" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br class="">
<div class="gmail_quote">On Wed, Aug 19, 2015 at 9:24 PM, Chris Horn<span class=""> </span><span dir="ltr" class=""><<a href="mailto:hornc@cray.com" target="_blank" class="">hornc@cray.com</a>></span><span class=""> </span>wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<div class="">They sure look pretty related to me. In kiblnd_post_tx_locked() we return EAGAIN if ibc_nsends_posted == IBLND_CONCURRENT_SENDS. ibc_nsends_posted is incremented on every send. So it looks like you couldn’t ever send more than concurrent_sends
messages to a single peer, which is exactly what peer_credits is supposed to govern, no? What am I missing?</div>
<span class=""><font color="#888888" class="">
<div class=""><br class="">
</div>
<div class="">Chris Horn</div>
</font></span>
<div class="">
<div class="">
<div class=""><br class="">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Aug 19, 2015, at 12:28 PM, Alexey Lyashkov <<a href="mailto:alexey.lyashkov@seagate.com" target="_blank" class="">alexey.lyashkov@seagate.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">Chris,
<div class=""><br class="">
<div class=""><span style="font-size:17.6000003814697px" class="">concurrent_sends is measurement about parallel operations, but credits is flow control artifact.</span><br class="">
</div>
<div class=""><span style="font-size:17.6000003814697px" class="">each operations may eats different number credits and calculated in per link and per destination basic, so it's completely different attributes.</span></div>
<div class=""><span style="font-size:17.6000003814697px" class=""><br class="">
</span></div>
<div class=""><span style="font-size:17.6000003814697px" class=""><br class="">
</span></div>
<div class=""><br class="">
</div>
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Wed, Aug 19, 2015 at 7:31 PM, Chris Horn<span class=""> </span><span dir="ltr" class=""><<a href="mailto:hornc@cray.com" target="_blank" class="">hornc@cray.com</a>></span><span class=""> </span>wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
A thread on HPDD-discuss made me think about this question. AFAICT, the o2iblnd driver code will not let you have more that concurrent_sends messages in flight at the same time(in fact, we LASSERT on this fact in kiblnd_check_sends). Thus peer_credits is effectively
limited by concurrent_sends anyways. What’s the reasoning behind allowing peer_credits to be larger than concurrent_sends?<br class="">
<span class=""><font color="#888888" class=""><br class="">
Chris Horn<br class="">
_______________________________________________<br class="">
lustre-devel mailing list<br class="">
<a href="mailto:lustre-devel@lists.lustre.org" target="_blank" class="">lustre-devel@lists.lustre.org</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=mGEpe9i1Xe69mkwImOAP_rhH7F-u64SSh70zy-1fqz4&s=B9m6PgjBPBZiV_CIxXPJRU5EoTEXV1rQYtuQGjwcOiU&e=" rel="noreferrer" target="_blank" class="">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><br class="">
</font></span></blockquote>
</div>
<br class="">
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class=""> </span><br class="">
<div class="">
<div dir="ltr" class="">Alexey Lyashkov <strong class="">·</strong> Technical lead for a Morpheus team<br class="">
Seagate Technology, LLC<br class="">
<a href="http://www.seagate.com/" target="_blank" class="">www.seagate.com</a><br class="">
<div class=""><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lustre.org_&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=mGEpe9i1Xe69mkwImOAP_rhH7F-u64SSh70zy-1fqz4&s=ndWLD_9DiIMxtYFvQrJRumfN-MZTMVBJQdeid6tdKAw&e=" target="_blank" class="">www.lustre.org</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class=""> </span><br class="">
<div class="">
<div dir="ltr" class="">Alexey Lyashkov <strong class="">·</strong> Technical lead for a Morpheus team<br class="">
Seagate Technology, LLC<br class="">
<a href="http://www.seagate.com/" target="_blank" class="">www.seagate.com</a><br class="">
<div class=""><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lustre.org_&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=xj-N9a-E5CgJ_rYAs23V93tTuZ1HyQw2yltXnCNAgSI&s=HES5frmlzutMi-g6kgm_00bfdDBM6r3ot5yMOL5buP0&e=" target="_blank" class="">www.lustre.org</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<br clear="all" class="">
<div class=""><br class="">
</div>
-- <br class="">
<div class="gmail_signature">
<div dir="ltr" class="">Alexey Lyashkov <strong class="">·</strong> Technical lead for a Morpheus team<br class="">
Seagate Technology, LLC<br class="">
<a href="http://www.seagate.com/" target="_blank" class="">www.seagate.com</a><br class="">
<div class=""><a href="http://www.lustre.org/" target="_blank" class="">www.lustre.org</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>