[lustre-discuss] IPoIB best practises

[Åke Sandgren]

On 1/17/22 2:36 AM, Angelos Ching via lustre-discuss wrote:
> Hi Eli,
> 
> Yes & no; part of my info is a bit rusty because I carried them from
> version around 2.10. MR is now turned on by default.
> 
> But you'll need to have an IP setup on each IPoIB interface, and for all
> ib0 & all ib1 interface, they should be in different subnet. Eg: all ib0
> on 192.168.100.0/24 and all ib1 on 192.168.101.0/24

The multirail setup we have is that both ib0 and ib1 are on the same
subnet, that's how DDN configured it for us.

ip a s ib0 | grep inet
    inet 172.27.1.30/24 brd 172.27.1.255 scope global ib0
ip a s ib1 | grep inet
    inet 172.27.1.50/24 brd 172.27.1.255 scope global ib1

and the modprobe config is

options lnet networks="o2ib1(ib0,ib1)"

-- 
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: ake at hpc2n.umu.se   Phone: +46 90 7866134 Fax: +46 90-580 14
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se