<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Local LNet configuration can be done either via modprobe config or via lnetctl/yaml. We are slowly moving away from modprobe config (kernel module parameters) in favor of lnetctl/yaml because the latter provides more flexibility.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">For IB and TCP networks, every interface needs an IP address assigned.<br>
<br>
It is okay to have multiple interfaces on the same subnet as long as you have appropriate ip route/rules and ARP settings in place. Otherwise the network stack may not actually send traffic to/from the correct interfaces, or there may be connection failures,
etc. There was some work to do this automatically for TCP networks in <a href="https://jira.whamcloud.com/browse/LU-14662">https://jira.whamcloud.com/browse/LU-14662</a> . There is some discussion of the issue on the wiki at
<a href="https://wiki.lustre.org/LNet_Router_Config_Guide#ARP_flux_issue_for_MR_node">https://wiki.lustre.org/LNet_Router_Config_Guide#ARP_flux_issue_for_MR_node</a> but I’m not sure how up-to-date that guidance is.<br>
<br>
LNet/ko2iblnd only uses IPoIB for connection setup via RDMA CM. After a connection is established all traffic between IB peers is via RDMA protocol.<o:p></o:p></p>
<p class="MsoNormal"><br>
The multi-rail feature requires more than just a local LNet configuration. It also requires configuration of the peer table. In Lustre 2.10, this peer table was statically defined. In Lustre 2.11 (and later), the LNet Dynamic Peer Discovery feature allows LNet
to create the peer table dynamically.=<br>
<br>
Chris Horn<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Åke Sandgren <ake.sandgren@hpc2n.umu.se><br>
<b>Date: </b>Monday, January 17, 2022 at 1:10 AM<br>
<b>To: </b>Lustre discussion <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] IPoIB best practises<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
On 1/17/22 2:36 AM, Angelos Ching via lustre-discuss wrote:<br>
> Hi Eli,<br>
> <br>
> Yes & no; part of my info is a bit rusty because I carried them from<br>
> version around 2.10. MR is now turned on by default.<br>
> <br>
> But you'll need to have an IP setup on each IPoIB interface, and for all<br>
> ib0 & all ib1 interface, they should be in different subnet. Eg: all ib0<br>
> on 192.168.100.0/24 and all ib1 on 192.168.101.0/24<br>
<br>
The multirail setup we have is that both ib0 and ib1 are on the same<br>
subnet, that's how DDN configured it for us.<br>
<br>
ip a s ib0 | grep inet<br>
inet 172.27.1.30/24 brd 172.27.1.255 scope global ib0<br>
ip a s ib1 | grep inet<br>
inet 172.27.1.50/24 brd 172.27.1.255 scope global ib1<br>
<br>
and the modprobe config is<br>
<br>
options lnet networks="o2ib1(ib0,ib1)"<br>
<br>
-- <br>
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden<br>
Internet: ake@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90-580 14<br>
Mobile: +46 70 7716134 WWW: <a href="http://www.hpc2n.umu.se">http://www.hpc2n.umu.se</a>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<o:p></o:p></p>
</div>
</div>
</body>
</html>