[lustre-discuss] IPoIB best practises

[Angelos Ching]

Hi Eli,

I think you can leave out the IP so long as you specific the interface, but IIRC you can't setup multirail via modprobe, you should setup Lnet using lnetctl 

Best regards,
Angelos
(Sent from mobile, please pardon me for typos and cursoriness.)

> 16/1/2022 21:38、E.S. Rosenberg <esr+lustre at mail.hebrew.edu>のメール:
> 
> 
> Hey Lauz,
> 
> Thanks for the reply!
> Indeed it sounds like multirail is what I am referring to (my wrong memory of multirail was a failover technique allowing connection of both TCP and IPoIB or multiple IPoIB between nodes at the same time), but I am still left with a question.
> In the modprobe arguments for lnet I'm binding all the interfaces into a single network but I don't define IPs for the interfaces that is done at the OS level.
> Lustre as far as I recall does use IPoIB so IPs are needed or could I even leave out the IP definitions and let lustre figure out what it wants there?
> It just seems kind of excessive and adding points of failure that a node with 4 dual port IB cards would end up needing 8 IP addresses...
> Thanks again,
> Eli
> 
> 
> 
>> On Sun, Jan 16, 2022 at 2:32 PM Laurence Horrocks-Barlow <laurence at qsplace.co.uk> wrote:
>> The limit of IPoIB is active/backup when using traditional bonding, however I believe you are wanting to multirail your IB. This is achieved buy using multiple LNet's (assuming it uses the same fabric), you should be able to configure for active/active.
>> 
>> https://wiki.whamcloud.com/display/LNet/Multi-Rail+Overview
>> 
>> This should help with most of the concepts.
>> 
>> -- Lauz
>> 
>>> On 16 January 2022 11:56:16 GMT, "E.S. Rosenberg" <esr+lustre at mail.hebrew.edu> wrote:
>>> Hey everyone,
>>> 
>>> This is probably off-topic but I can't find any documents on the subject and since Lustre uses IPoIB I suspect others here have dealt with this question.
>>> 
>>> If I have a node connected with multiple IB links should each connected IB port have it's own IP address or is there a way similar to LACP on the Ethernet side to bond all the links and use only a single IP address to refer to the node? And what is the better method?
>>> 
>>> In the past I never had this luxury, but now I'm starting a small new cluster currently made up of a few GPU nodes and a Lustre filesystem so plenty of IB ports to go around.
>>> 
>>> Thanks!
>>> Eli
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220116/39529916/attachment.html>