<div dir="ltr"><div dir="ltr">Thanks for the replies, the nodes have multiple interfaces (four on compute nodes and 6 on the storage nodes), ens2f0 is the 100G Mellanox ConnectX5 card in slot 2 and they are all running 2.12.6 using the RPMS from the lustre site.</div><div dir="ltr"><br></div><div dir="ltr">I will remove one of the network definition files and add the lnetctl --backup config to the /etc/lnet.conf.... i did try an export and noticed it barfs on some of the parameters but I did not try the --backup option, so it gives me a few options to experiment with minimising the config.... just a bit of trial and error</div><div dir="ltr"><br></div><div dir="ltr">I gather then the lustre.conf file is not needed, just the /etc/modprobe.d/lnet.conf and the /etc/lnet.conf.<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div><br></div><div>Sid Young</div></div></div></div></div></div></div></div></div></div></div></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>---------- Forwarded message ----------<br>From: "Degremont, Aurelien" <<a href="mailto:degremoa@amazon.com" target="_blank">degremoa@amazon.com</a>><br>To: Sid Young <<a href="mailto:sid.young@gmail.com" target="_blank">sid.young@gmail.com</a>>, lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>Cc: <br>Bcc: <br>Date: Tue, 23 Feb 2021 08:47:27 +0000<br>Subject: Re: [lustre-discuss] need to always manually add network after reboot<br>
<div lang="FR" style="word-wrap:break-word">
<div class="gmail-m_-6260066770701151177WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hello<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">If I understand correctly, you're telling that you have 2 configuration files:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:35.4pt">/etc/modprobe.d/lnet.conf<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:35.4pt">options lnet networks=tcp<u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:35.4pt">[root@hpc-oss-03 ~]# cat /etc/modprobe.d/lustre.conf<br>
options lnet networks="tcp(ens2f0)"<br>
options lnet ip2nets="tcp(ens2f0) 10.140.93.*<u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">That means you are declaring twice the "networks" option for "lnet" kernel module. I don't know how 'modprobe' will behave regarding that.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">If you have a very simple configuration, where your nodes only have one Ethernet interface "ens2f0", you only need the following lines, from the 3 above:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">options lnet networks="tcp(ens2f0)"</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">If this interface is the only Ethernet interface on your host, you don't even need a network specific setup. By default, when loading Lustre, in the absence of a network configuration,
Lustre will automatically setup the only ethernet interface to use it for "tcp".<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Aurélien<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div style="border-style:solid none none;border-right-width:initial;border-bottom-width:initial;border-left-width:initial;border-right-color:initial;border-bottom-color:initial;border-left-color:initial;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span style="font-size:12pt;color:black">De :
</span></b><span style="font-size:12pt;color:black">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> au nom de Sid Young via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Répondre à : </b>Sid Young <<a href="mailto:sid.young@gmail.com" target="_blank">sid.young@gmail.com</a>><br>
<b>Date : </b>mardi 23 février 2021 à 06:59<br>
<b>À : </b>lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Objet : </b>[EXTERNAL] [lustre-discuss] need to always manually add network after reboot<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<table border="0" cellspacing="0" cellpadding="0" width="625" style="width:468.8pt;margin-left:35.4pt;border-collapse:collapse">
<tbody>
<tr style="height:15.25pt">
<td width="625" valign="top" style="width:468.8pt;border:1.5pt solid rgb(237,125,49);padding:0cm 5.4pt;height:15.25pt">
<p><strong><span style="font-family:Calibri,sans-serif;color:black;background-image:initial;background-color:rgb(255,255,153);background-size:initial;background-origin:initial;background-clip:initial;background-position:initial;background-repeat:initial">CAUTION</span></strong><span style="color:black;background-image:initial;background-color:rgb(255,255,153);background-size:initial;background-origin:initial;background-clip:initial;background-position:initial;background-repeat:initial">: This email originated from outside of the organization. Do not click links or open attachments unless
you can confirm the sender and know the content is safe.</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">G'Day all,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I'm finding that when I reboot any node in our new HPC, I need to keep manually adding the network using lnetctl net add --net tcp --if ens2f0<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Then I can do an lnetctl net show and see the tcp part active...<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I have options in /etc/modprobe.d/lnet.conf<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">options lnet networks=tcp<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">and <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">[root@hpc-oss-03 ~]# cat /etc/modprobe.d/lustre.conf<br>
options lnet networks="tcp(ens2f0)"<br>
options lnet ip2nets="tcp(ens2f0) 10.140.93.*<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I've read the doco and tried to understand the correct parameters for a simple Lustre config so this is what I worked out is needed... but I suspect its still wrong.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Any help appreciated :)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Sid Young<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br><br><br>---------- Forwarded message ----------<br>From: Angelos Ching <<a href="mailto:angelosching@clustertech.com" target="_blank">angelosching@clustertech.com</a>><br>To: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>Cc: <br>Bcc: <br>Date: Tue, 23 Feb 2021 18:06:02 +0800<br>Subject: Re: [lustre-discuss] need to always manually add network after reboot<br>
<div>
<p>Hi Sid,</p>
<p>Notice that you are using lnetctl net add to add the lnet
network, which means you should be using a recent version of
Lustre that depends on /etc/lnet.conf for boot time lnet
configuration.</p>
<p>You can save the current lnet configuration using command:
lnetctl export --backup > /etc/lnet.conf (make a backup of the
original file first if required)<br>
</p>
<p>On next boot, lnet.service will load your lnet configuration from
the file.</p>
<p>Or you can manually build lnet.conf as lnetctl seems to have
occasion problems with some of the fields exported by "lnetctl
export --backup"</p>
<p>Attaching my simple lnet.conf for your reference:</p>
<p>
</p><blockquote type="cite"><font face="monospace"># cat
/etc/lnet.conf<br>
ip2nets:<br>
- net-spec: o2ib<br>
ip-range:<br>
0: 10.2.8.*<br>
- net-spec: tcp<br>
ip-range:<br>
0: 10.5.9.*<br>
route:<br>
- net: o2ib<br>
gateway: 10.5.9.25@tcp<br>
hop: -1<br>
priority: 0<br>
- net: o2ib<br>
gateway: 10.5.9.24@tcp<br>
hop: -1<br>
priority: 0<br>
global:<br>
numa_range: 0<br>
max_intf: 200<br>
discovery: 1<br>
drop_asym_route: 0</font></blockquote>
Best regards,<br>
Angelos<p></p>
<div><br></div></div>
</blockquote></div></div>