[lustre-devel] Why can you set concurrent_sends < peer_credits ?

[Alexey Lyashkov]

In credit based theory - credits should never to be less zero, as credit is
resources to send. If credits set to zero we just stop send.
My assumption base on book "Traffic Management for High-speed Networks
International Science Lecture Series ; 4th Lecture".

> This is counterintuitive, and I don’t understand why the code was written
this way.
if i understand correctly it code was originally written for a direct
connected network, but credits distribution model forget to change when
lnet routers was created. So we continue to count credits in peers base
instead of distribute credits only in next hop base where a router should
distribute a provided credits over connected clients. It provide a router
overloading with huge parallel sends and new limit was added.



On Wed, Aug 19, 2015 at 10:38 PM, Chris Horn <hornc at cray.com> wrote:

> But as i understand credit based flow control don't work now - i see
> several situations when server have a negative number a credits, which
> indicate we have sending queue more than limits and parallel sends limits
> will work in that situation..
>
>
> Well, I think that depends on how the flow control is supposed to work.
> The code definitely prevents *more* than peer_credits sends to a single
> peer. And negative number of credits indeed indicates a queue which, I
> think, is fine. What is non-obvious is that it appears that the code may
> prevent fewer than peer_credits sends to a single peer, even if there
> aren’t any other outstanding sends to other peers. i.e. we hit the
> concurrent_sends limit before we hit the peer_credits limit. This is
> counterintuitive, and I don’t understand why the code was written this way.
>
> Chris Horn
>
> On Aug 19, 2015, at 1:58 PM, Alexey Lyashkov <alexey.lyashkov at seagate.com>
> wrote:
>
> Chris,
>
> In current code it's may be same. Let me finish some bugs before look to
> code. But with "good" code - you should have a different cost (credits
> count) for different messages. 1Mb bulk transfer should eats more credits
> than simple 4kb message. So you "should" have a two limits first one for
> parallel send with zero/low cost messages and second one for large messages.
> But as i understand credit based flow control don't work now - i see
> several situations when server have a negative number a credits, which
> indicate we have sending queue more than limits and parallel sends limits
> will work in that situation..
>
> It's know bug for me but need large time to create a network model to
> create a correct credits distribution.
>
>
> On Wed, Aug 19, 2015 at 9:24 PM, Chris Horn <hornc at cray.com> wrote:
>
>> They sure look pretty related to me. In kiblnd_post_tx_locked() we return
>> EAGAIN if ibc_nsends_posted == IBLND_CONCURRENT_SENDS. ibc_nsends_posted is
>> incremented on every send. So it looks like you couldn’t ever send more
>> than concurrent_sends messages to a single peer, which is exactly what
>> peer_credits is supposed to govern, no?  What am I missing?
>>
>> Chris Horn
>>
>>
>> On Aug 19, 2015, at 12:28 PM, Alexey Lyashkov <
>> alexey.lyashkov at seagate.com> wrote:
>>
>> Chris,
>>
>> concurrent_sends is measurement about parallel operations, but credits is
>> flow control artifact.
>> each operations may eats different number credits and calculated in per
>> link and per destination basic, so it's completely different attributes.
>>
>>
>>
>>
>> On Wed, Aug 19, 2015 at 7:31 PM, Chris Horn <hornc at cray.com> wrote:
>>
>>> A thread on HPDD-discuss made me think about this question. AFAICT, the
>>> o2iblnd driver code will not let you have more that concurrent_sends
>>> messages in flight at the same time(in fact, we LASSERT on this fact in
>>> kiblnd_check_sends). Thus peer_credits is effectively limited by
>>> concurrent_sends anyways. What’s the reasoning behind allowing peer_credits
>>> to be larger than concurrent_sends?
>>>
>>> Chris Horn
>>> _______________________________________________
>>> lustre-devel mailing list
>>> lustre-devel at lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=mGEpe9i1Xe69mkwImOAP_rhH7F-u64SSh70zy-1fqz4&s=B9m6PgjBPBZiV_CIxXPJRU5EoTEXV1rQYtuQGjwcOiU&e=>
>>>
>>
>>
>>
>> --
>> Alexey Lyashkov *·* Technical lead for a Morpheus team
>> Seagate Technology, LLC
>> www.seagate.com
>> www.lustre.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lustre.org_&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=mGEpe9i1Xe69mkwImOAP_rhH7F-u64SSh70zy-1fqz4&s=ndWLD_9DiIMxtYFvQrJRumfN-MZTMVBJQdeid6tdKAw&e=>
>>
>>
>>
>
>
> --
> Alexey Lyashkov *·* Technical lead for a Morpheus team
> Seagate Technology, LLC
> www.seagate.com
> www.lustre.org
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lustre.org_&d=AwMGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=xj-N9a-E5CgJ_rYAs23V93tTuZ1HyQw2yltXnCNAgSI&s=HES5frmlzutMi-g6kgm_00bfdDBM6r3ot5yMOL5buP0&e=>
>
>
>


-- 
Alexey Lyashkov *·* Technical lead for a Morpheus team
Seagate Technology, LLC
www.seagate.com
www.lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20150819/7a154e2d/attachment-0001.htm>