[lustre-discuss] IPoIB best practises

[E.S. Rosenberg]

Hi Angelos,
Thanks for the answer!
According to the lecture Lauz just sent multirail is on by default no?
So I don't need to set IPs on each interface (using
interfaces/netplan/network scripts) just a single NID at the lnet level?
Thanks again,
Eli

On Sun, Jan 16, 2022 at 5:20 PM Angelos Ching <angelosching at clustertech.com>
wrote:

> Hi Eli,
>
> I think you can leave out the IP so long as you specific the interface,
> but IIRC you can't setup multirail via modprobe, you should setup Lnet
> using lnetctl
>
> Best regards,
> Angelos
> (Sent from mobile, please pardon me for typos and cursoriness.)
>
> 16/1/2022 21:38、E.S. Rosenberg <esr+lustre at mail.hebrew.edu>のメール:
>
> 
> Hey Lauz,
>
> Thanks for the reply!
> Indeed it sounds like multirail is what I am referring to (my wrong memory
> of multirail was a failover technique allowing connection of both TCP and
> IPoIB or multiple IPoIB between nodes at the same time), but I am still
> left with a question.
> In the modprobe arguments for lnet I'm binding all the interfaces into a
> single network but I don't define IPs for the interfaces that is done at
> the OS level.
> Lustre as far as I recall does use IPoIB so IPs are needed or could I even
> leave out the IP definitions and let lustre figure out what it wants there?
> It just seems kind of excessive and adding points of failure that a node
> with 4 dual port IB cards would end up needing 8 IP addresses...
> Thanks again,
> Eli
>
>
>
> On Sun, Jan 16, 2022 at 2:32 PM Laurence Horrocks-Barlow <
> laurence at qsplace.co.uk> wrote:
>
>> The limit of IPoIB is active/backup when using traditional bonding,
>> however I believe you are wanting to multirail your IB. This is achieved
>> buy using multiple LNet's (assuming it uses the same fabric), you should be
>> able to configure for active/active.
>>
>> https://wiki.whamcloud.com/display/LNet/Multi-Rail+Overview
>>
>> This should help with most of the concepts.
>>
>> -- Lauz
>>
>> On 16 January 2022 11:56:16 GMT, "E.S. Rosenberg" <
>> esr+lustre at mail.hebrew.edu> wrote:
>>>
>>> Hey everyone,
>>>
>>> This is probably off-topic but I can't find any documents on the subject
>>> and since Lustre uses IPoIB I suspect others here have dealt with this
>>> question.
>>>
>>> If I have a node connected with multiple IB links should each connected
>>> IB port have it's own IP address or is there a way similar to LACP on the
>>> Ethernet side to bond all the links and use only a single IP address to
>>> refer to the node? And what is the better method?
>>>
>>> In the past I never had this luxury, but now I'm starting a small new
>>> cluster currently made up of a few GPU nodes and a Lustre filesystem so
>>> plenty of IB ports to go around.
>>>
>>> Thanks!
>>> Eli
>>>
>> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220116/c4ed9321/attachment-0001.html>