[lustre-discuss] lustre-discuss Digest, Vol 208, Issue 6

Mon Jul 10 03:09:24 PDT 2023

Hi Anna,

This isn’t Lustre specific, but last time I spoke to Mellanox when I was worried about over subscription between InfiniBand L1 and L2 switches. They said that the IB ASICs monitor congestion on the HBAs of the connected channel and will throttle back HBA channel transfer rates if congestion on one or both is detected.

This may help to smear out some of the imbalances, but you would probably still get Lustre ‘waiting’ type warnings.

Cheers

Marc

From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of lustre-discuss-request at lists.lustre.org <lustre-discuss-request at lists.lustre.org>
Date: Friday, 7 July 2023 at 21:12
To: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
Subject: lustre-discuss Digest, Vol 208, Issue 6
Send lustre-discuss mailing list submissions to
        lustre-discuss at lists.lustre.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.lustre.org%2flistinfo.cgi%2flustre-discuss-lustre.org&c=E,1,udOzn8uiss34fCFAWaJTKs9MrvTzb7uSTeisCdxzZ76x5DnzLe2J3JMmFIyIv-IJtUaBMKZaiBbYuF1c5RF6m8rF-jHYOcR0evY3ZEutI-rZ&typo=1
or, via email, send a message with subject or body 'help' to
        lustre-discuss-request at lists.lustre.org

You can reach the person managing the list at
        lustre-discuss-owner at lists.lustre.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of lustre-discuss digest..."

Today's Topics:

   1. Imbalanced incoming and outgoing network load (Anna Fuchs)
   2. Re: Imbalanced incoming and outgoing network load
      (Kulyavtsev, Alex Ivanovich)

----------------------------------------------------------------------

Message: 1
Date: Fri, 7 Jul 2023 13:48:34 +0200
From: Anna Fuchs <anna.fuchs at uni-hamburg.de>
To: <lustre-discuss at lists.lustre.org>
Subject: [lustre-discuss] Imbalanced incoming and outgoing network
        load
Message-ID: <ad173ee0-101a-29ee-6995-fa1a66aa5290 at uni-hamburg.de>
Content-Type: text/plain; charset="UTF-8"; format=flowed

Dear all,

I have some questions regarding the following scenario:
 ?- A large HPC system.
- Let's assume that Job X is running on 1 compute node and is reading a
very large file with a stripecount (>>1)..-1. Alternatively, tons of
files are read at once with smaller striping each, but distributed
across all OSS/OSTs.
- The compute node is connected, for example, with a 100Gb/s link, and
there are 50 servers, each with a 200Gb/s link. This generates a network
load of 50x200Gb/s, which is processed at 100Gb/s.
- Job Y, which requires the same network and potentially doesn't even
perform I/O, suffers a lot as a result.

Does this scenario sound familiar to you?
Is the sequence of events correct?
What could be done in this situation?

To avoid:
a) having such single/few-nodes jobs
b) striping large files with up to -1
c) reading millions of files at once
One could try, but I have concerns that the users will persist in doing
it, either intentionally or accidentally, and it would only shift the
problem, rather than solving it.
One could tweak the network design, reconfigure it, separate I/O from
communication, but it would hardly optimize all use cases. Virtual lanes
could potentially be a solution as well. Though, that might not help if
the Job Y also involves some I/O.

Wouldn't it be better if Lustre somehow recognized this imbalance
between incoming and outgoing network traffic and loaded the
file(s)/data gradually rather than all at once, saturating or slightly
overloading the consumer 100Gb/s connection rather than by a factor of
100? Does this sound reasonable, and is there already a solution for it?
I would appreciate any opinions.

Best regards
Anna

--
Anna Fuchs
Universit?t Hamburg
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwr.informatik.uni-hamburg.de%2fpeople%2fanna_fuchs&c=E,1,5lU4ZdOq0v8v03GynIpPmNbhvtS_2QTSByMPxhQ3oiVaSAIfEjlmOuf0py53AEnmokksRCIU8P50mJcXWmHPYkklq_Gcbhq8AQUcD3kGWcq9rDuUR_K1pk96&typo=1

------------------------------

Message: 2
Date: Fri, 7 Jul 2023 16:18:55 +0000
From: "Kulyavtsev, Alex Ivanovich" <alexku at anl.gov>
To: Anna Fuchs <anna.fuchs at uni-hamburg.de>
Cc: "lustre-discuss at lists.lustre.org"
        <lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] Imbalanced incoming and outgoing network
        load
Message-ID: <5F635719-E080-4CA0-BE2E-55ED330C6A7F at anl.gov>
Content-Type: text/plain; charset="utf-8"

There is QoS in lustre, the feature called NRS - Network Request Scheduler.
It is possible to set different policies.
Will it address the issue ?

The manual has entry and there were few presentations on LUG/LAD.

I did not use NRS myself but I would like to learn.
Alex.

> On Jul 7, 2023, at 06:48, Anna Fuchs via lustre-discuss <lustre-discuss at lists.lustre.org> wrote:
>
> Dear all,
>
> I have some questions regarding the following scenario:
>  - A large HPC system.
> - Let's assume that Job X is running on 1 compute node and is reading a very large file with a stripecount (>>1)..-1. Alternatively, tons of files are read at once with smaller striping each, but distributed across all OSS/OSTs.
> - The compute node is connected, for example, with a 100Gb/s link, and there are 50 servers, each with a 200Gb/s link. This generates a network load of 50x200Gb/s, which is processed at 100Gb/s.
> - Job Y, which requires the same network and potentially doesn't even perform I/O, suffers a lot as a result.
>
> Does this scenario sound familiar to you?
> Is the sequence of events correct?
> What could be done in this situation?
>
> To avoid:
> a) having such single/few-nodes jobs
> b) striping large files with up to -1
> c) reading millions of files at once
> One could try, but I have concerns that the users will persist in doing it, either intentionally or accidentally, and it would only shift the problem, rather than solving it.
> One could tweak the network design, reconfigure it, separate I/O from communication, but it would hardly optimize all use cases. Virtual lanes could potentially be a solution as well. Though, that might not help if the Job Y also involves some I/O.
>
> Wouldn't it be better if Lustre somehow recognized this imbalance between incoming and outgoing network traffic and loaded the file(s)/data gradually rather than all at once, saturating or slightly overloading the consumer 100Gb/s connection rather than by a factor of 100? Does this sound reasonable, and is there already a solution for it?
> I would appreciate any opinions.
>
> Best regards
> Anna
>
> --
> Anna Fuchs
> Universit?t Hamburg
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwr.informatik.uni-hamburg.de%2fpeople%2fanna_fuchs&c=E,1,tE0uhLIlDzWctIlGAdMLaXv5rg1mMzZb43E_JUBZ-5wIPT2fsxnmMiVa4Nnjp9_x58lz7HSfjkbJOnQWnmQBPcdZ1ickmZDSMUA98IQ_VQ,,&typo=1
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.lustre.org%2flistinfo.cgi%2flustre-discuss-lustre.org&c=E,1,RdA3hxSQ2vRMTbs0l--PQyom2TjdaN7ziaZ-dcCmFkL565YonIZRF6_wWL8RFV4Pgeb4uSMCtgWb2NMFa9yzfNdU1uaZKrsuICISLwNNYiIDOxt6qFSXuhVa&typo=1

------------------------------

Subject: Digest Footer

_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.lustre.org%2flistinfo.cgi%2flustre-discuss-lustre.org&c=E,1,dcSAQBOQ0m4cALCt1T7jYhggyItOaWAFRfKYaJFwekxVjZ9jzl7BcKTulPJhqJpr2AKONUYh8Zq_y2mRYMnVb61m_i6sAo6VphEn2T2aIY9LsCiRAhdT&typo=1

------------------------------

End of lustre-discuss Digest, Vol 208, Issue 6
**********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20230710/b2f33919/attachment-0001.htm>