[lustre-discuss] IPoIB best practises

[Horn, Chris]

Local LNet configuration can be done either via modprobe config or via lnetctl/yaml. We are slowly moving away from modprobe config (kernel module parameters) in favor of lnetctl/yaml because the latter provides more flexibility.

For IB and TCP networks, every interface needs an IP address assigned.

It is okay to have multiple interfaces on the same subnet as long as you have appropriate ip route/rules and ARP settings in place. Otherwise the network stack may not actually send traffic to/from the correct interfaces, or there may be connection failures, etc. There was some work to do this automatically for TCP networks in https://jira.whamcloud.com/browse/LU-14662 . There is some discussion of the issue on the wiki at https://wiki.lustre.org/LNet_Router_Config_Guide#ARP_flux_issue_for_MR_node but I’m not sure how up-to-date that guidance is.

LNet/ko2iblnd only uses IPoIB for connection setup via RDMA CM. After a connection is established all traffic between IB peers is via RDMA protocol.

The multi-rail feature requires more than just a local LNet configuration. It also requires configuration of the peer table. In Lustre 2.10, this peer table was statically defined. In Lustre 2.11 (and later), the LNet Dynamic Peer Discovery feature allows LNet to create the peer table dynamically.=

Chris Horn

From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of Åke Sandgren <ake.sandgren at hpc2n.umu.se>
Date: Monday, January 17, 2022 at 1:10 AM
To: Lustre discussion <lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] IPoIB best practises


On 1/17/22 2:36 AM, Angelos Ching via lustre-discuss wrote:
> Hi Eli,
>
> Yes & no; part of my info is a bit rusty because I carried them from
> version around 2.10. MR is now turned on by default.
>
> But you'll need to have an IP setup on each IPoIB interface, and for all
> ib0 & all ib1 interface, they should be in different subnet. Eg: all ib0
> on 192.168.100.0/24 and all ib1 on 192.168.101.0/24

The multirail setup we have is that both ib0 and ib1 are on the same
subnet, that's how DDN configured it for us.

ip a s ib0 | grep inet
    inet 172.27.1.30/24 brd 172.27.1.255 scope global ib0
ip a s ib1 | grep inet
    inet 172.27.1.50/24 brd 172.27.1.255 scope global ib1

and the modprobe config is

options lnet networks="o2ib1(ib0,ib1)"

--
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: ake at hpc2n.umu.se   Phone: +46 90 7866134 Fax: +46 90-580 14
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se<http://www.hpc2n.umu.se>
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220119/ee7df2cd/attachment.html>