<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Andale Mono";
panose-1:2 11 5 9 0 0 0 0 0 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Ah yes, I see what the lnet unit file is doing. OK, I think this is all straighten out and working great now. We have a fairly extensive init script (the lustre3 script in previous posts) that does various
checks in addition to loading modules and mounting/unmounting the filesystems. But at its core, the start is now doing this:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier"> /usr/bin/systemctl start lnet >& /dev/null<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier"> modprobe lustre<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier"> <mount lustre FS's><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The stop portion does:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier"> <umount lustre FS's><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier"> /usr/bin/systemctl stop lnet >& /dev/null<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier"> /usr/sbin/lustre_rmmod<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The final conf files I'm using are:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">lnet.conf:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">net:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> - net type: o2ib1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> local NI(s):<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> - interfaces:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> 0: ib0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">global:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> discovery: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">/etc/modprobe.d/lustre.conf:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">options ko2iblnd map_on_demand=32<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Using the lnet systemd unit file properly loads the configuration and shows discovery=0 (without any of lnet stuff in the modprobe conf file). We could properly enable the lnet unit file and make a dependency
to make sure our init script runs after the lnet service but its a little easier to just run the systemctl commands in our init script.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I would be interested if others have a cleaner way to do all mounting, etc. in a more native systemd manner. It probably just involves making a simple unit file to run a script. Probably six of one, half
dozen of the other but if anyone has experience with the pros and cons, please let me know.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks a ton for the help on this. Much appreciated.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Horn, Chris" <chris.horn@hpe.com><br>
<b>Date: </b>Tuesday, September 14, 2021 at 9:40 AM<br>
<b>To: </b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <darby.vicker-1@nasa.gov>, Riccardo Veraldi <riccardo.veraldi@cnaf.infn.it>, "lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[EXTERNAL] Re: Re: [lustre-discuss] Disabling multi-rail dynamic discovery<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">When you start LNet via ‘modprobe lnet; lctl net up’, that doesn’t load the configuration from /etc/lnet.conf. It is going to configure LNet based only on kernel module parameters. Since you removed the ‘options
lnet networks’ from your modprobe.conf file, it is going to use the default configuration which is @tcp on whatever the first ethernet interface w/ipv4 configured that it finds.<br>
<br>
To load /etc/lnet.conf you can use systemctl start lnet.service (or equivalent), or if you want to do it manually:<br>
<br>
modprobe lnet<br>
lnetctl lnet configure<br>
lnetctl lnet import < /etc/lnet.conf<br>
<br>
Also, I would try this for your lnet.conf<br>
<br>
</span><span style="font-size:11.0pt;font-family:"Andale Mono"">net:<br>
- net type: o2ib<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Andale Mono""> local NI(s):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Andale Mono""> - interfaces:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Andale Mono""> 0: ib0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Andale Mono"">global:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Andale Mono""> discovery: 0</span><span style="font-size:11.0pt"><br>
<br>
Chris Horn<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] <darby.vicker-1@nasa.gov><br>
<b>Date: </b>Tuesday, September 14, 2021 at 10:17 AM<br>
<b>To: </b>Horn, Chris <chris.horn@hpe.com>, Riccardo Veraldi <riccardo.veraldi@cnaf.infn.it>, lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [EXTERNAL] Re: [lustre-discuss] Disabling multi-rail dynamic discovery<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">So I"m a little confused. <o:p>
</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">When I take the "options lnet networks=o2ib1(ib0)" line out of the modprobe conf file and instead put that info in the lnet.conf file, things don't work properly.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]# cat /etc/modprobe.d/lustre.conf<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">options ko2iblnd map_on_demand=32<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]# cat /etc/lnet.conf
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">ip2nets:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">- net-spec: o2ib1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> interfaces:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> 0: ib0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">global:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> discovery: 0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]# modprobe lnet<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]# lctl network up<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">LNET configured<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]# service lustre3 start<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">Mounting /ephemeral... mount.lustre: mount 10.150.100.30@o2ib1:10.150.100.31@o2ib1:/scratch/work at /ephemeral failed: No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">Is the MGS specification correct?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">Is the filesystem name correct?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">If upgrading, is the copied client log valid? (see upgrade docs)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">FAILED.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">Mounting /nobackup... mount.lustre: mount 10.150.100.30@o2ib1:10.150.100.31@o2ib1:/hpfs-fsl/work at /nobackup failed: No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">Is the MGS specification correct?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">Is the filesystem name correct?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">If upgrading, is the copied client log valid? (see upgrade docs)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">FAILED.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]#<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The logs when this happens:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Sep 14 09:53:38 r1i1n18 kernel: LNet: Added LNI
<a href="mailto:10.159.0.39@tcp">10.159.0.39@tcp</a> [8/256/0/180]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Sep 14 09:53:38 r1i1n18 kernel: Lnet: Accept secure, port 988<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Sep 14 09:53:54 r1i1n18 kernel: Lustre: Lustre: Build Version: 2.12.6<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Sep 14 09:53:55 r1i1n18 kernel: LustreError: 34174:0:(ldlm_lib.c:494:client_obd_setup()) can't add initial connection<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Sep 14 09:53:55 r1i1n18 kernel: LustreError: 34174:0:(obd_config.c:559:class_setup()) setup
<a href="mailto:MGC10.150.100.30@o">MGC10.150.100.30@o</a>2ib1 failed (-2)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Sep 14 09:53:55 r1i1n18 kernel: LustreError: 34174:0:(obd_mount.c:202:lustre_start_simple())
<a href="mailto:MGC10.150.100.30@o">MGC10.150.100.30@o</a>2ib1 setup error -2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Sep 14 09:53:55 r1i1n18 kernel: LustreError: 34174:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-2)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Note the @tcp above – it looks like without the modprobe conf file, the lnet module isn't getting set up properly. When this happens, I'm not able to shut down lnet or unload the kernel modules to try again.
The only way I've been able to recover from this is to reboot the node. If I add the "options lnet" stuff back to the modprobe conf file, everything works as expected. Do I not have enough info in lnet.conf or are both just required?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Chris, adding lnet_peer_discovery_disabled=1 to my lnet options does indeed seem to work. Thanks!
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Darby<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Horn, Chris" <chris.horn@hpe.com><br>
<b>Date: </b>Monday, September 13, 2021 at 4:59 PM<br>
<b>To: </b>Riccardo Veraldi <riccardo.veraldi@cnaf.infn.it>, "Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <darby.vicker-1@nasa.gov>, "lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[EXTERNAL] Re: [lustre-discuss] Disabling multi-rail dynamic discovery<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m not sure why lnetctl import wouldn’t correctly set discovery. Might be a bug. You can try setting the kernel module parameter to disable discovery:<br>
<br>
options lnet lnet_peer_discovery_disabled=1<br>
<br>
This obviously requires LNet to be reloaded.<br>
<br>
I would not recommend toggling discovery via the CLI as there are some bugs with correctly dealing with the fallout of that (peers going from MR enabled to MR disabled).<br>
<br>
Chris Horn<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Riccardo Veraldi <riccardo.veraldi@cnaf.infn.it><br>
<b>Date: </b>Monday, September 13, 2021 at 5:25 PM<br>
<b>To: </b>Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] <darby.vicker-1@nasa.gov>, lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Disabling multi-rail dynamic discovery<o:p></o:p></span></p>
</div>
<p>I supposed you removed the /etc/modprobe.d/lustre.conf completely.<o:p></o:p></p>
<p>I only have the lnet service enabled at startup, I do not start any lustre3 service, but I am running lustre 2.12.0 sorry not 2.14<o:p></o:p></p>
<p>so something might be different.<o:p></o:p></p>
<p>Did you start over with a clean configuration ?<o:p></o:p></p>
<p>Did you reboot your system to make sure it picks up the new config ? At least for me sometimes the lnet module does not unload correctly.<o:p></o:p></p>
<p>Also I have to mention in my setup I did disable discovery also on the OSSes not only client side.<o:p></o:p></p>
<p>Generally it is not advisable to disable Multi-rail unless you have backward compatibility issues with older lustre peers.<o:p></o:p></p>
<p>But disabling discovery will also disable Multi-rail.<o:p></o:p></p>
<p>You can try with <o:p></o:p></p>
<p>lenetctl set discovery 0<o:p></o:p></p>
<p>as you already did,<o:p></o:p></p>
<p>then you do<o:p></o:p></p>
<p>lnetctl -b export > /etc/lnet.conf<o:p></o:p></p>
<p>check discovery is set to 0 in the file and if not edit it and set it to 0.<o:p></o:p></p>
<p>reboot and see if things changes.<o:p></o:p></p>
<p>If anyway you did not define any tcp interface in lnet.conf you should not see any tcp peers.<o:p></o:p></p>
<p><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 9/13/21 2:59 PM, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks Rick. I removed my lnet modprobe options and adapted my lnet.conf file to:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"># cat /etc/lnet.conf
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">ip2nets:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">- net-spec: o2ib1</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> interfaces:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> 0: ib0</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">global:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> discovery: 0</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">#</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Now "lnetctl export" doesn't have any reference to NIDs on the other networks, so that's good. However, I'm still seeing some values that concern me:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"># lnetctl export | grep -e Multi -e discover | sort -u</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> discovery: 1</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> Multi-Rail: True</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">#</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Any idea why discovery is still 1 if I'm specifying that to 0 in the lnet.conf file? I'm a little concerned that with Multi-Rail still True and discovery on, the client could still find its way back to the
TCP route. </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Riccardo Veraldi
<a href="mailto:riccardo.veraldi@cnaf.infn.it"><riccardo.veraldi@cnaf.infn.it></a><br>
<b>Date: </b>Monday, September 13, 2021 at 3:16 PM<br>
<b>To: </b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <a href="mailto:darby.vicker-1@nasa.gov">
<darby.vicker-1@nasa.gov></a>, <a href="mailto:lustre-discuss@lists.lustre.org">"lustre-discuss@lists.lustre.org"</a>
<a href="mailto:lustre-discuss@lists.lustre.org"><lustre-discuss@lists.lustre.org></a><br>
<b>Subject: </b>[EXTERNAL] Re: [lustre-discuss] Disabling multi-rail dynamic discovery</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p>I would use configuration on /etc/lnet.conf and I would not use anymore the older style configuration in
<o:p></o:p></p>
<p><span style="font-family:Courier">/etc/modprobe.d/lustre.conf </span><o:p></o:p></p>
<p class="MsoNormal">for example in my /etc/lnet.conf configuration I have: <o:p>
</o:p></p>
<p><b><span style="font-family:Courier">ip2nets:<br>
- net-spec: o2ib<br>
interfaces:<br>
0: ib0<br>
- net-spec: tcp<br>
interfaces:<br>
0: enp24s0f0<br>
global:<br>
discovery: 0</span></b><o:p></o:p></p>
<p>As I disabled the auto discovery.<o:p></o:p></p>
<p>Regarding ko2ib you can just use /etc/modprobe.d/ko2iblnd.conf<o:p></o:p></p>
<p>Mine looks like this:<o:p></o:p></p>
<p><b><span style="font-family:"Courier New"">options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048 map_on_demand=256 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4</span></b><o:p></o:p></p>
<p>Hope it helps.<o:p></o:p></p>
<p>Rick<o:p></o:p></p>
<p> <o:p></o:p></p>
<div>
<p class="MsoNormal">On 9/13/21 1:53 PM, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I would like to know how to turn off auto discovery of peers on a client. This seems like it should be straight forward but we can't get it to work. Please fill me in on what I'm missing.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We recently upgraded our servers to 2.14. Our servers are multi-homed (1 tcp network and 2 separate IB networks) but we want them to be single rail. On one of our clusters we are still using the 2.12.6 client
and it uses one of the IB networks for lustre. The modprobe file from one of the client nodes:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"># cat /etc/modprobe.d/lustre.conf
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">options lnet networks=o2ib1(ib0)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">options ko2iblnd map_on_demand=32</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">#</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The client does have a route to the TCP network. This is intended to allow jobs on the compute nodes to access licenese servers, not for any serious I/O. We recently discovered that due to some instability
in the IB fabric, the client was trying to fail over to tcp:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""># dmesg | grep Lustre</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[ 250.205912] Lustre: Lustre: Build Version: 2.12.6</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[ 255.886086] Lustre: Mounted scratch-client</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[ 287.247547] Lustre: 3472:0:(client.c:2146:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1630699139/real 0] req@ffff98deb9358480 x1709911947878336/t0(0)
o9-><a href="mailto:hpfs-fsl-OST0001-osc-ffff9880cfb80000@192.52.98.33@tcp:28/4">hpfs-fsl-OST0001-osc-ffff9880cfb80000@192.52.98.33@tcp:28/4</a> lens 224/224 e 0 to 1 dl 1630699145 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[ 739.832744] Lustre: 3526:0:(client.c:2146:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1630699591/real 0] req@ffff98deb935da00 x1709911947883520/t0(0)
o400-><a href="mailto:scratch-MDT0000-mdc-ffff98b0f1fc0800@192.52.98.31@tcp:12/10">scratch-MDT0000-mdc-ffff98b0f1fc0800@192.52.98.31@tcp:12/10</a> lens 224/224 e 0 to 1 dl 1630699598 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[ 739.832755] Lustre: 3526:0:(client.c:2146:ptlrpc_expire_one_request()) Skipped 5 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[ 739.832762] LustreError: 166-1: MGC10.150.100.30@o2ib1: Connection to MGS (at 192.52.98.30@tcp) was lost; in progress operations using this service will fail</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[ 739.832769] Lustre: hpfs-fsl-MDT0000-mdc-ffff9880cfb80000: Connection to hpfs-fsl-MDT0000 (at 192.52.98.30@tcp) was lost; in progress operations using this service will wait for
recovery to complete</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[ 1090.978619] LustreError: 167-0: scratch-MDT0000-mdc-ffff98b0f1fc0800: This client was evicted by scratch-MDT0000; in progress operations using this service will fail.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I'm pretty sure this is due to the auto discovery. Again, from a client:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<pre style="margin-left:.5in"># lnetctl export | grep -e Multi -e discover | sort -u<o:p></o:p></pre>
<pre style="margin-left:.5in"> discovery: 0<o:p></o:p></pre>
<pre style="margin-left:.5in"> Multi-Rail: True<o:p></o:p></pre>
<pre style="margin-left:.5in"># <o:p></o:p></pre>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We want to restrict lustre to only the IB NID but its not clear exactly how to do that.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Here is one attempt:<br>
<br>
<br>
</span><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]# service lustre3 stop</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">Shutting down lustre mounts
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">Lustre modules successfully unloaded</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]# lsmod | grep lnet</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]# cat /etc/lnet.conf
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">global:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> discovery: 0</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]# service lustre3 start</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">Mounting /ephemeral... done.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">Mounting /nobackup... done.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]# lnetctl export | grep -e Multi -e discover | sort -u</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> discovery: 1</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier"> Multi-Rail: True</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier">[root@r1i1n18 lnet]#</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">And a similar attempt (same lnet.conf file), but trying to turn off the discovery before doing the mounts:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<pre style="margin-left:.5in">[root@r1i1n18 lnet]# service lustre3 stop<o:p></o:p></pre>
<pre style="margin-left:.5in">Shutting down lustre mounts <o:p></o:p></pre>
<pre style="margin-left:.5in">Lustre modules successfully unloaded<o:p></o:p></pre>
<pre style="margin-left:.5in">[root@r1i1n18 lnet]# modprobe lnet<o:p></o:p></pre>
<pre style="margin-left:.5in">[root@r1i1n18 lnet]# lnetctl set discovery 0<o:p></o:p></pre>
<pre style="margin-left:.5in">[root@r1i1n18 lnet]# service lustre3 start<o:p></o:p></pre>
<pre style="margin-left:.5in">Mounting /ephemeral... done.<o:p></o:p></pre>
<pre style="margin-left:.5in">Mounting /nobackup... done.<o:p></o:p></pre>
<pre style="margin-left:.5in">[root@r1i1n18 lnet]# lnetctl export | grep -e Multi -e discover | sort -u<o:p></o:p></pre>
<pre style="margin-left:.5in"> discovery: 0<o:p></o:p></pre>
<pre style="margin-left:.5in"> Multi-Rail: True<o:p></o:p></pre>
<pre style="margin-left:.5in">[root@r1i1n18 lnet]# <o:p></o:p></pre>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If someone can point me in the right direction, I'd appreciate it.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Darby</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>lustre-discuss mailing list<o:p></o:p></pre>
<pre><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><o:p></o:p></pre>
<pre><a href="https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=04%7C01%7Cdarby.vicker-1%40nasa.gov%7C8943d25ba8254c75fded08d97795ee11%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637672308371242046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=HQWyVMEEAGjCk2zuSLAob3xa6tciNTahH8z0ZRncTjE%3D&reserved=0">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></pre>
</blockquote>
</blockquote>
</div>
</body>
</html>