<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Sep 14, 2021, at 11:17, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Ah yes, I see what the lnet unit file is doing. OK, I think this is all straighten out and working great now. We have a fairly extensive init script (the lustre3 script in previous posts) that does various checks in
addition to loading modules and mounting/unmounting the filesystems. But at its core, the start is now doing this:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> /usr/bin/systemctl start lnet >& /dev/null<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> modprobe lustre<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> <mount lustre FS's><o:p class=""></o:p></span></div>
</div>
</div>
</blockquote>
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class="">Strictly speaking, the mount command itself should automatically trigger "lustre" module loading, so the "modprobe lustre" is redundant.</o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""><br class="">
</o:p></span></div>
</div>
</div>
<blockquote type="cite" class="">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">The stop portion does:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> <span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> <umount lustre FS's><o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> /usr/bin/systemctl stop lnet >& /dev/null<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> /usr/sbin/lustre_rmmod<o:p class=""></o:p></span></div>
</div>
</div>
</blockquote>
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
</div>
</div>
<span style="font-size: 15px;" class="">In 2.15 the lustre_rmmod script will automatically run "lnetctl lnet unconfigure", and conversely lnet.service will run "lustre_rmmod" in the right places (assuming the filesystem was previously unmounted), so only one
or the other will be needed. Running both isn't harmful, just a bit redundant.</span>
<div class=""><span style="font-size: 15px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 15px;" class="">Cheers, Andreas</span><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">The final conf files I'm using are:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">lnet.conf:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">net:<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> - net type: o2ib1<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> local NI(s):<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> - interfaces:<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> 0: ib0<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">global:<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> discovery: 0<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">/etc/modprobe.d/lustre.conf:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">options ko2iblnd map_on_demand=32<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Using the lnet systemd unit file properly loads the configuration and shows discovery=0 (without any of lnet stuff in the modprobe conf file). We could properly enable the lnet unit file and make a dependency to make
sure our init script runs after the lnet service but its a little easier to just run the systemctl commands in our init script. <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">I would be interested if others have a cleaner way to do all mounting, etc. in a more native systemd manner. It probably just involves making a simple unit file to run a script. Probably six of one, half dozen of the
other but if anyone has experience with the pros and cons, please let me know. <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Thanks a ton for the help on this. Much appreciated. <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="" class="">"Horn, Chris" <<a href="mailto:chris.horn@hpe.com" style="color: blue; text-decoration: underline;" class="">chris.horn@hpe.com</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Tuesday, September 14, 2021 at 9:40 AM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>>, Riccardo
Veraldi <<a href="mailto:riccardo.veraldi@cnaf.infn.it" style="color: blue; text-decoration: underline;" class="">riccardo.veraldi@cnaf.infn.it</a>>, "<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>"
<<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>[EXTERNAL] Re: Re: [lustre-discuss] Disabling multi-rail dynamic discovery<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
</div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">When you start LNet via ‘modprobe lnet; lctl net up’, that doesn’t load the configuration from /etc/lnet.conf. It is going to configure LNet based only on kernel module parameters. Since you removed the ‘options lnet
networks’ from your modprobe.conf file, it is going to use the default configuration which is @tcp on whatever the first ethernet interface w/ipv4 configured that it finds.<br class="">
<br class="">
To load /etc/lnet.conf you can use systemctl start lnet.service (or equivalent), or if you want to do it manually:<br class="">
<br class="">
modprobe lnet<br class="">
lnetctl lnet configure<br class="">
lnetctl lnet import < /etc/lnet.conf<br class="">
<br class="">
Also, I would try this for your lnet.conf<br class="">
<br class="">
</span><span style="font-size: 11pt; font-family: "Andale Mono";" class="">net:<br class="">
- net type: o2ib<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Andale Mono";" class=""> local NI(s):<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Andale Mono";" class=""> - interfaces:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Andale Mono";" class=""> 0: ib0<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Andale Mono";" class="">global:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Andale Mono";" class=""> discovery: 0</span><span style="font-size: 11pt;" class=""><br class="">
<br class="">
Chris Horn<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: Calibri, sans-serif;">
<b class=""><span style="" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="" class="">Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Tuesday, September 14, 2021 at 10:17 AM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>Horn, Chris <<a href="mailto:chris.horn@hpe.com" style="color: blue; text-decoration: underline;" class="">chris.horn@hpe.com</a>>, Riccardo Veraldi <<a href="mailto:riccardo.veraldi@cnaf.infn.it" style="color: blue; text-decoration: underline;" class="">riccardo.veraldi@cnaf.infn.it</a>>,<span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a><span class="Apple-converted-space"> </span><<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [EXTERNAL] Re: [lustre-discuss] Disabling multi-rail dynamic discovery<o:p class=""></o:p></span></p>
</div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">So I"m a little confused. <span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">When I take the "options lnet networks=o2ib1(ib0)" line out of the modprobe conf file and instead put that info in the lnet.conf file, things don't work properly. <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]# cat /etc/modprobe.d/lustre.conf<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">options ko2iblnd map_on_demand=32<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]# cat /etc/lnet.conf<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">ip2nets:<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">- net-spec: o2ib1<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> interfaces:<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> 0: ib0<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">global:<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> discovery: 0<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]# modprobe lnet<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]# lctl network up<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">LNET configured<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]# service lustre3 start<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Mounting /ephemeral... mount.lustre: mount 10.150.100.30@o2ib1:10.150.100.31@o2ib1:/scratch/work at /ephemeral failed: No such file or directory<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Is the MGS specification correct?<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Is the filesystem name correct?<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">If upgrading, is the copied client log valid? (see upgrade docs)<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">FAILED.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Mounting /nobackup... mount.lustre: mount 10.150.100.30@o2ib1:10.150.100.31@o2ib1:/hpfs-fsl/work at /nobackup failed: No such file or directory<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Is the MGS specification correct?<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Is the filesystem name correct?<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">If upgrading, is the copied client log valid? (see upgrade docs)<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">FAILED.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]#<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">The logs when this happens:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Sep 14 09:53:38 r1i1n18 kernel: LNet: Added LNI<span class="Apple-converted-space"> </span><a href="mailto:10.159.0.39@tcp" style="color: blue; text-decoration: underline;" class="">10.159.0.39@tcp</a><span class="Apple-converted-space"> </span>[8/256/0/180]<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Sep 14 09:53:38 r1i1n18 kernel: Lnet: Accept secure, port 988<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Sep 14 09:53:54 r1i1n18 kernel: Lustre: Lustre: Build Version: 2.12.6<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Sep 14 09:53:55 r1i1n18 kernel: LustreError: 34174:0:(ldlm_lib.c:494:client_obd_setup()) can't add initial connection<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Sep 14 09:53:55 r1i1n18 kernel: LustreError: 34174:0:(obd_config.c:559:class_setup()) setup<span class="Apple-converted-space"> </span><a href="mailto:MGC10.150.100.30@o" style="color: blue; text-decoration: underline;" class="">MGC10.150.100.30@o</a>2ib1
failed (-2)<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Sep 14 09:53:55 r1i1n18 kernel: LustreError: 34174:0:(obd_mount.c:202:lustre_start_simple())<span class="Apple-converted-space"> </span><a href="mailto:MGC10.150.100.30@o" style="color: blue; text-decoration: underline;" class="">MGC10.150.100.30@o</a>2ib1
setup error -2<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Sep 14 09:53:55 r1i1n18 kernel: LustreError: 34174:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-2)<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Note the @tcp above – it looks like without the modprobe conf file, the lnet module isn't getting set up properly. When this happens, I'm not able to shut down lnet or unload the kernel modules to try again. The only
way I've been able to recover from this is to reboot the node. If I add the "options lnet" stuff back to the modprobe conf file, everything works as expected. Do I not have enough info in lnet.conf or are both just required? <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Chris, adding lnet_peer_discovery_disabled=1 to my lnet options does indeed seem to work. Thanks! <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Darby<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="" class="">"Horn, Chris" <<a href="mailto:chris.horn@hpe.com" style="color: blue; text-decoration: underline;" class="">chris.horn@hpe.com</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Monday, September 13, 2021 at 4:59 PM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>Riccardo Veraldi <<a href="mailto:riccardo.veraldi@cnaf.infn.it" style="color: blue; text-decoration: underline;" class="">riccardo.veraldi@cnaf.infn.it</a>>, "Vicker, Darby J. (JSC-EG111)[Jacobs
Technology, Inc.]" <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>>, "<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>"
<<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>[EXTERNAL] Re: [lustre-discuss] Disabling multi-rail dynamic discovery<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
</div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">I’m not sure why lnetctl import wouldn’t correctly set discovery. Might be a bug. You can try setting the kernel module parameter to disable discovery:<br class="">
<br class="">
options lnet lnet_peer_discovery_disabled=1<br class="">
<br class="">
This obviously requires LNet to be reloaded.<br class="">
<br class="">
I would not recommend toggling discovery via the CLI as there are some bugs with correctly dealing with the fallout of that (peers going from MR enabled to MR disabled).<br class="">
<br class="">
Chris Horn<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: Calibri, sans-serif;">
<b class=""><span style="" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="" class="">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss-bounces@lists.lustre.org</a>>
on behalf of Riccardo Veraldi <<a href="mailto:riccardo.veraldi@cnaf.infn.it" style="color: blue; text-decoration: underline;" class="">riccardo.veraldi@cnaf.infn.it</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Monday, September 13, 2021 at 5:25 PM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>>,<span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a><span class="Apple-converted-space"> </span><<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [lustre-discuss] Disabling multi-rail dynamic discovery<o:p class=""></o:p></span></p>
</div>
<div class="">I supposed you removed the /etc/modprobe.d/lustre.conf completely.<o:p class=""></o:p></div>
<div class="">I only have the lnet service enabled at startup, I do not start any lustre3 service, but I am running lustre 2.12.0 sorry not 2.14<o:p class=""></o:p></div>
<div class="">so something might be different.<o:p class=""></o:p></div>
<div class="">Did you start over with a clean configuration ?<o:p class=""></o:p></div>
<div class="">Did you reboot your system to make sure it picks up the new config ? At least for me sometimes the lnet module does not unload correctly.<o:p class=""></o:p></div>
<div class="">Also I have to mention in my setup I did disable discovery also on the OSSes not only client side.<o:p class=""></o:p></div>
<div class="">Generally it is not advisable to disable Multi-rail unless you have backward compatibility issues with older lustre peers.<o:p class=""></o:p></div>
<div class="">But disabling discovery will also disable Multi-rail.<o:p class=""></o:p></div>
<div class="">You can try with<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div class="">lenetctl set discovery 0<o:p class=""></o:p></div>
<div class="">as you already did,<o:p class=""></o:p></div>
<div class="">then you do<o:p class=""></o:p></div>
<div class="">lnetctl -b export > /etc/lnet.conf<o:p class=""></o:p></div>
<div class="">check discovery is set to 0 in the file and if not edit it and set it to 0.<o:p class=""></o:p></div>
<div class="">reboot and see if things changes.<o:p class=""></o:p></div>
<div class="">If anyway you did not define any tcp interface in lnet.conf you should not see any tcp peers.<o:p class=""></o:p></div>
<div class=""><o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
On 9/13/21 2:59 PM, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] wrote:<o:p class=""></o:p></div>
</div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Thanks Rick. I removed my lnet modprobe options and adapted my lnet.conf file to:</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""># cat /etc/lnet.conf</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">ip2nets:</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">- net-spec: o2ib1</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> interfaces:</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> 0: ib0</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">global:</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> discovery: 0</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">#</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Now "lnetctl export" doesn't have any reference to NIDs on the other networks, so that's good. However, I'm still seeing some values that concern me:</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""># lnetctl export | grep -e Multi -e discover | sort -u</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> discovery: 1</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> Multi-Rail: True</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">#</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Any idea why discovery is still 1 if I'm specifying that to 0 in the lnet.conf file? I'm a little concerned that with Multi-Rail still True and discovery on, the client could still find its way back to the TCP route. <span class="Apple-converted-space"> </span></span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="" class="">Riccardo Veraldi<span class="Apple-converted-space"> </span><a href="mailto:riccardo.veraldi@cnaf.infn.it" style="color: blue; text-decoration: underline;" class=""><riccardo.veraldi@cnaf.infn.it></a><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Monday, September 13, 2021 at 3:16 PM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]"<span class="Apple-converted-space"> </span><a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class=""><darby.vicker-1@nasa.gov></a>,<span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">"lustre-discuss@lists.lustre.org"</a><span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class=""><lustre-discuss@lists.lustre.org></a><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>[EXTERNAL] Re: [lustre-discuss] Disabling multi-rail dynamic discovery</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
</div>
<div class="">I would use configuration on /etc/lnet.conf and I would not use anymore the older style configuration in<o:p class=""></o:p></div>
<div class=""><span style="font-family: Courier;" class="">/etc/modprobe.d/lustre.conf<span class="Apple-converted-space"> </span></span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
for example in my /etc/lnet.conf configuration I have:<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div class=""><b class=""><span style="font-family: Courier;" class="">ip2nets:<br class="">
- net-spec: o2ib<br class="">
interfaces:<br class="">
0: ib0<br class="">
- net-spec: tcp<br class="">
interfaces:<br class="">
0: enp24s0f0<br class="">
global:<br class="">
discovery: 0</span></b><o:p class=""></o:p></div>
<div class="">As I disabled the auto discovery.<o:p class=""></o:p></div>
<div class="">Regarding ko2ib you can just use /etc/modprobe.d/ko2iblnd.conf<o:p class=""></o:p></div>
<div class="">Mine looks like this:<o:p class=""></o:p></div>
<div class=""><b class=""><span style="font-family: "Courier New";" class="">options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048 map_on_demand=256 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4</span></b><o:p class=""></o:p></div>
<div class="">Hope it helps.<o:p class=""></o:p></div>
<div class="">Rick<o:p class=""></o:p></div>
<div class=""> <o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
On 9/13/21 1:53 PM, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss wrote:<o:p class=""></o:p></div>
</div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Hello,</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">I would like to know how to turn off auto discovery of peers on a client. This seems like it should be straight forward but we can't get it to work. Please fill me in on what I'm missing. </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">We recently upgraded our servers to 2.14. Our servers are multi-homed (1 tcp network and 2 separate IB networks) but we want them to be single rail. On one of our clusters we are still using the 2.12.6 client and it
uses one of the IB networks for lustre. The modprobe file from one of the client nodes:</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""># cat /etc/modprobe.d/lustre.conf</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">options lnet networks=o2ib1(ib0)</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">options ko2iblnd map_on_demand=32</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">#</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">The client does have a route to the TCP network. This is intended to allow jobs on the compute nodes to access licenese servers, not for any serious I/O. We recently discovered that due to some instability in the IB
fabric, the client was trying to fail over to tcp:</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class=""># dmesg | grep Lustre</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">[ 250.205912] Lustre: Lustre: Build Version: 2.12.6</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">[ 255.886086] Lustre: Mounted scratch-client</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">[ 287.247547] Lustre: 3472:0:(client.c:2146:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1630699139/real 0] req@ffff98deb9358480 x1709911947878336/t0(0)
o9-><a href="mailto:hpfs-fsl-OST0001-osc-ffff9880cfb80000@192.52.98.33@tcp:28/4" style="color: blue; text-decoration: underline;" class="">hpfs-fsl-OST0001-osc-ffff9880cfb80000@192.52.98.33@tcp:28/4</a><span class="Apple-converted-space"> </span>lens 224/224
e 0 to 1 dl 1630699145 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">[ 739.832744] Lustre: 3526:0:(client.c:2146:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1630699591/real 0] req@ffff98deb935da00 x1709911947883520/t0(0)
o400-><a href="mailto:scratch-MDT0000-mdc-ffff98b0f1fc0800@192.52.98.31@tcp:12/10" style="color: blue; text-decoration: underline;" class="">scratch-MDT0000-mdc-ffff98b0f1fc0800@192.52.98.31@tcp:12/10</a><span class="Apple-converted-space"> </span>lens 224/224
e 0 to 1 dl 1630699598 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">[ 739.832755] Lustre: 3526:0:(client.c:2146:ptlrpc_expire_one_request()) Skipped 5 previous similar messages</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">[ 739.832762] LustreError: 166-1: MGC10.150.100.30@o2ib1: Connection to MGS (at 192.52.98.30@tcp) was lost; in progress operations using this service will fail</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">[ 739.832769] Lustre: hpfs-fsl-MDT0000-mdc-ffff9880cfb80000: Connection to hpfs-fsl-MDT0000 (at 192.52.98.30@tcp) was lost; in progress operations using this service will wait for recovery
to complete</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">[ 1090.978619] LustreError: 167-0: scratch-MDT0000-mdc-ffff98b0f1fc0800: This client was evicted by scratch-MDT0000; in progress operations using this service will fail.</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">I'm pretty sure this is due to the auto discovery. Again, from a client:</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class=""># lnetctl export | grep -e Multi -e discover | sort -u<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class=""> discovery: 0<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class=""> Multi-Rail: True<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class=""># <o:p class=""></o:p></pre>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">We want to restrict lustre to only the IB NID but its not clear exactly how to do that. </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Here is one attempt:<br class="">
<br class="">
<br class="">
</span><span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]# service lustre3 stop</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Shutting down lustre mounts</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Lustre modules successfully unloaded</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]# lsmod | grep lnet</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]# cat /etc/lnet.conf</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">global:</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> discovery: 0</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]# service lustre3 start</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Mounting /ephemeral... done.</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Mounting /nobackup... done.</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]# lnetctl export | grep -e Multi -e discover | sort -u</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> discovery: 1</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""> Multi-Rail: True</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@r1i1n18 lnet]#</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">And a similar attempt (same lnet.conf file), but trying to turn off the discovery before doing the mounts:</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class="">[root@r1i1n18 lnet]# service lustre3 stop<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class="">Shutting down lustre mounts <o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class="">Lustre modules successfully unloaded<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class="">[root@r1i1n18 lnet]# modprobe lnet<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class="">[root@r1i1n18 lnet]# lnetctl set discovery 0<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class="">[root@r1i1n18 lnet]# service lustre3 start<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class="">Mounting /ephemeral... done.<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class="">Mounting /nobackup... done.<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class="">[root@r1i1n18 lnet]# lnetctl export | grep -e Multi -e discover | sort -u<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class=""> discovery: 0<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class=""> Multi-Rail: True<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: "Courier New";" class="">[root@r1i1n18 lnet]# <o:p class=""></o:p></pre>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">If someone can point me in the right direction, I'd appreciate it. </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Thanks,</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Darby</span><o:p class=""></o:p></div>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: Calibri, sans-serif;">
<o:p class=""> </o:p></p>
<pre style="margin: 0in; font-size: 10pt; font-family: "Courier New";" class="">_______________________________________________<o:p class=""></o:p></pre>
<pre style="margin: 0in; font-size: 10pt; font-family: "Courier New";" class="">lustre-discuss mailing list<o:p class=""></o:p></pre>
<pre style="margin: 0in; font-size: 10pt; font-family: "Courier New";" class=""><a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a><o:p class=""></o:p></pre>
<pre style="margin: 0in; font-size: 10pt; font-family: "Courier New";" class=""><a href="https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=04%7C01%7Cdarby.vicker-1%40nasa.gov%7C8943d25ba8254c75fded08d97795ee11%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637672308371242046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=HQWyVMEEAGjCk2zuSLAob3xa6tciNTahH8z0ZRncTjE%3D&reserved=0" style="color: blue; text-decoration: underline;" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p class=""></o:p></pre>
</blockquote>
</blockquote>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">lustre-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">lustre-discuss@lists.lustre.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="color: blue; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>