<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<blockquote type="cite" class="">But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work
in that situation..</blockquote>
<div class=""><br class="">
</div>
<div class="">Well, I think that depends on how the flow control is supposed to work. The code definitely prevents *more* than peer_credits sends to a single peer. And negative number of credits indeed indicates a queue which, I think, is fine. What is non-obvious
is that it appears that the code may prevent fewer than peer_credits sends to a single peer, even if there aren’t any other outstanding sends to other peers. i.e. we hit the concurrent_sends limit before we hit the peer_credits limit. This is counterintuitive,
and I don’t understand why the code was written this way.</div>
<div class=""><br class="">
</div>
<div class="">Chris Horn</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Aug 19, 2015, at 1:58 PM, Alexey Lyashkov <<a href="mailto:alexey.lyashkov@seagate.com" class="">alexey.lyashkov@seagate.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Chris,
<div class=""><br class="">
</div>
<div class="">In current code it's may be same. Let me finish some bugs before look to code. But with "good" code - you should have a different cost (credits count) for different messages. 1Mb bulk transfer should eats more credits than simple 4kb message.
So you "should" have a two limits first one for parallel send with zero/low cost messages and second one for large messages.</div>
<div class="">But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work in that situation..</div>
<div class=""><br class="">
</div>
<div class="">It's know bug for me but need large time to create a network model to create a correct credits distribution.</div>
<div class=""><br class="">
</div>
</div>
<div class="gmail_extra" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<br class="">
<div class="gmail_quote">On Wed, Aug 19, 2015 at 9:24 PM, Chris Horn<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:hornc@cray.com" target="_blank" class="">hornc@cray.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class="">
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">
<div style="word-wrap: break-word;" class="">
<div class="">They sure look pretty related to me. In kiblnd_post_tx_locked() we return EAGAIN if ibc_nsends_posted == IBLND_CONCURRENT_SENDS. ibc_nsends_posted is incremented on every send. So it looks like you couldn’t ever send more than concurrent_sends
messages to a single peer, which is exactly what peer_credits is supposed to govern, no? What am I missing?</div>
<span class="HOEnZb"><font color="#888888" class="">
<div class=""><br class="">
</div>
<div class="">Chris Horn</div>
</font></span>
<div class="">
<div class="h5">
<div class=""><br class="">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Aug 19, 2015, at 12:28 PM, Alexey Lyashkov <<a href="mailto:alexey.lyashkov@seagate.com" target="_blank" class="">alexey.lyashkov@seagate.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">Chris,
<div class=""><br class="">
<div class=""><span style="font-size: 17.6000003814697px;" class="">concurrent_sends is measurement about parallel operations, but credits is flow control artifact.</span><br class="">
</div>
<div class=""><span style="font-size: 17.6000003814697px;" class="">each operations may eats different number credits and calculated in per link and per destination basic, so it's completely different attributes.</span></div>
<div class=""><span style="font-size: 17.6000003814697px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 17.6000003814697px;" class=""><br class="">
</span></div>
<div class=""><br class="">
</div>
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Wed, Aug 19, 2015 at 7:31 PM, Chris Horn<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:hornc@cray.com" target="_blank" class="">hornc@cray.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class="">
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">
A thread on HPDD-discuss made me think about this question. AFAICT, the o2iblnd driver code will not let you have more that concurrent_sends messages in flight at the same time(in fact, we LASSERT on this fact in kiblnd_check_sends). Thus peer_credits is effectively
limited by concurrent_sends anyways. What’s the reasoning behind allowing peer_credits to be larger than concurrent_sends?<br class="">
<span class=""><font color="#888888" class=""><br class="">
Chris Horn<br class="">
_______________________________________________<br class="">
lustre-devel mailing list<br class="">
<a href="mailto:lustre-devel@lists.lustre.org" target="_blank" class="">lustre-devel@lists.lustre.org</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=mGEpe9i1Xe69mkwImOAP_rhH7F-u64SSh70zy-1fqz4&s=B9m6PgjBPBZiV_CIxXPJRU5EoTEXV1rQYtuQGjwcOiU&e=" rel="noreferrer" target="_blank" class="">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><br class="">
</font></span></blockquote>
</div>
<br class="">
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class="Apple-converted-space"> </span><br class="">
<div class="">
<div dir="ltr" class="">Alexey Lyashkov <strong class="">·</strong> Technical lead for a Morpheus team<br class="">
Seagate Technology, LLC<br class="">
<a href="http://www.seagate.com/" target="_blank" class="">www.seagate.com</a><br class="">
<div class=""><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lustre.org_&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=mGEpe9i1Xe69mkwImOAP_rhH7F-u64SSh70zy-1fqz4&s=ndWLD_9DiIMxtYFvQrJRumfN-MZTMVBJQdeid6tdKAw&e=" target="_blank" class="">www.lustre.org</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class="Apple-converted-space"> </span><br class="">
<div class="gmail_signature">
<div dir="ltr" class="">Alexey Lyashkov <strong class="">·</strong> Technical lead for a Morpheus team<br class="">
Seagate Technology, LLC<br class="">
<a href="http://www.seagate.com/" target="_blank" class="">www.seagate.com</a><br class="">
<div class=""><a href="http://www.lustre.org/" target="_blank" class="">www.lustre.org</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>