<div dir="ltr">In credit based theory - credits should never to be less zero, as credit is resources to send. If credits set to zero we just stop send.<div>My assumption base on book "Traffic Management for High-speed Networks International Science Lecture Series ; 4th Lecture".</div><div><br></div><div><span style="font-size:12.8000001907349px">> This is counterintuitive, and I don’t understand why the code was written this way.</span><br></div><div><span style="font-size:12.8000001907349px">if i understand correctly it code was originally written for a direct connected network, but credits distribution model forget to change when lnet routers was created. So we continue to count credits in peers base instead of distribute credits only in next hop base where a router should distribute a provided credits over connected clients. It provide a router overloading with huge parallel sends and new limit was added.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 19, 2015 at 10:38 PM, Chris Horn <span dir="ltr"><<a href="mailto:hornc@cray.com" target="_blank">hornc@cray.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><span class="">
<blockquote type="cite">But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work
in that situation..</blockquote>
<div><br>
</div>
</span><div>Well, I think that depends on how the flow control is supposed to work. The code definitely prevents *more* than peer_credits sends to a single peer. And negative number of credits indeed indicates a queue which, I think, is fine. What is non-obvious
is that it appears that the code may prevent fewer than peer_credits sends to a single peer, even if there aren’t any other outstanding sends to other peers. i.e. we hit the concurrent_sends limit before we hit the peer_credits limit. This is counterintuitive,
and I don’t understand why the code was written this way.</div><span class="HOEnZb"><font color="#888888">
<div><br>
</div>
<div>Chris Horn</div></font></span><div><div class="h5">
<div><br>
<div>
<blockquote type="cite">
<div>On Aug 19, 2015, at 1:58 PM, Alexey Lyashkov <<a href="mailto:alexey.lyashkov@seagate.com" target="_blank">alexey.lyashkov@seagate.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Chris,
<div><br>
</div>
<div>In current code it's may be same. Let me finish some bugs before look to code. But with "good" code - you should have a different cost (credits count) for different messages. 1Mb bulk transfer should eats more credits than simple 4kb message.
So you "should" have a two limits first one for parallel send with zero/low cost messages and second one for large messages.</div>
<div>But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work in that situation..</div>
<div><br>
</div>
<div>It's know bug for me but need large time to create a network model to create a correct credits distribution.</div>
<div><br>
</div>
</div>
<div class="gmail_extra" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
<div class="gmail_quote">On Wed, Aug 19, 2015 at 9:24 PM, Chris Horn<span> </span><span dir="ltr"><<a href="mailto:hornc@cray.com" target="_blank">hornc@cray.com</a>></span><span> </span>wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>They sure look pretty related to me. In kiblnd_post_tx_locked() we return EAGAIN if ibc_nsends_posted == IBLND_CONCURRENT_SENDS. ibc_nsends_posted is incremented on every send. So it looks like you couldn’t ever send more than concurrent_sends
messages to a single peer, which is exactly what peer_credits is supposed to govern, no? What am I missing?</div>
<span><font color="#888888">
<div><br>
</div>
<div>Chris Horn</div>
</font></span>
<div>
<div>
<div><br>
</div>
<br>
<div>
<blockquote type="cite">
<div>On Aug 19, 2015, at 12:28 PM, Alexey Lyashkov <<a href="mailto:alexey.lyashkov@seagate.com" target="_blank">alexey.lyashkov@seagate.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr">Chris,
<div><br>
<div><span style="font-size:17.6000003814697px">concurrent_sends is measurement about parallel operations, but credits is flow control artifact.</span><br>
</div>
<div><span style="font-size:17.6000003814697px">each operations may eats different number credits and calculated in per link and per destination basic, so it's completely different attributes.</span></div>
<div><span style="font-size:17.6000003814697px"><br>
</span></div>
<div><span style="font-size:17.6000003814697px"><br>
</span></div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Aug 19, 2015 at 7:31 PM, Chris Horn<span> </span><span dir="ltr"><<a href="mailto:hornc@cray.com" target="_blank">hornc@cray.com</a>></span><span> </span>wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
A thread on HPDD-discuss made me think about this question. AFAICT, the o2iblnd driver code will not let you have more that concurrent_sends messages in flight at the same time(in fact, we LASSERT on this fact in kiblnd_check_sends). Thus peer_credits is effectively
limited by concurrent_sends anyways. What’s the reasoning behind allowing peer_credits to be larger than concurrent_sends?<br>
<span><font color="#888888"><br>
Chris Horn<br>
_______________________________________________<br>
lustre-devel mailing list<br>
<a href="mailto:lustre-devel@lists.lustre.org" target="_blank">lustre-devel@lists.lustre.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=mGEpe9i1Xe69mkwImOAP_rhH7F-u64SSh70zy-1fqz4&s=B9m6PgjBPBZiV_CIxXPJRU5EoTEXV1rQYtuQGjwcOiU&e=" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><br>
</font></span></blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
--<span> </span><br>
<div>
<div dir="ltr">Alexey Lyashkov <strong>·</strong> Technical lead for a Morpheus team<br>
Seagate Technology, LLC<br>
<a href="http://www.seagate.com/" target="_blank">www.seagate.com</a><br>
<div><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lustre.org_&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=mGEpe9i1Xe69mkwImOAP_rhH7F-u64SSh70zy-1fqz4&s=ndWLD_9DiIMxtYFvQrJRumfN-MZTMVBJQdeid6tdKAw&e=" target="_blank">www.lustre.org</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
--<span> </span><br>
<div>
<div dir="ltr">Alexey Lyashkov <strong>·</strong> Technical lead for a Morpheus team<br>
Seagate Technology, LLC<br>
<a href="http://www.seagate.com/" target="_blank">www.seagate.com</a><br>
<div><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lustre.org_&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=xj-N9a-E5CgJ_rYAs23V93tTuZ1HyQw2yltXnCNAgSI&s=HES5frmlzutMi-g6kgm_00bfdDBM6r3ot5yMOL5buP0&e=" target="_blank">www.lustre.org</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Alexey Lyashkov <strong>·</strong> Technical lead for a Morpheus team<br>
Seagate Technology, LLC<br>
<a href="http://www.seagate.com" target="_blank">www.seagate.com</a><br><div><a href="http://www.lustre.org" target="_blank">www.lustre.org</a></div></div></div>
</div>