<div dir="ltr">Awesome, thanks! Unfortunately the password reset site is not finding my UID. Maybe I never had access to the Lustre wiki. (I have so many accounts that sometimes my head spins.) I'm still willing to help. Is there a request password site?<br><div><br></div><div>Cheers,</div><div>megan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 8:54 PM Spitz, Cory James <<a href="mailto:cory.spitz@hpe.com">cory.spitz@hpe.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_4501481448440832035WordSection1">
<p class="MsoNormal">Megan,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">You wrote:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">PS. [I am willing to add/contribute to the
<a href="http://wiki.lustre.org/Infiniband_Configuration_Howto" target="_blank">
http://wiki.lustre.org/Infiniband_Configuration_Howto</a> but I think my account for wiki editing has expired (at least the one I thought I had did not work).<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thank you for your offer! Did you try <a href="http://wiki.lustre.org/Special:PasswordReset" target="_blank">
http://wiki.lustre.org/Special:PasswordReset</a>? If that didn’t work then I think that you could email
<a href="mailto:lustre.org@lists.opensfs.org" target="_blank">lustre.org@lists.opensfs.org</a>.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">-Cory<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">On 6/24/20, 3:33 PM, "lustre-discuss on behalf of Ms. Megan Larko" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a> on behalf of
<a href="mailto:dobsonunit@gmail.com" target="_blank">dobsonunit@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">On 22 Jun 2020 "guru.novice" wrote:<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Hi, all<br>
We setup up a cluster use mlx4 and mlx5 driver mixed?all things goes well.<br>
Later I find something in wiki<br>
<a href="http://wiki.lustre.org/Infiniband_Configuration_Howto" target="_blank">http://wiki.lustre.org/Infiniband_Configuration_Howto</a>
and<br>
<a href="http://lists.onebuilding.org/pipermail/lustre-devel-lustre.org/2016-May/003842.html" target="_blank">http://lists.onebuilding.org/pipermail/lustre-devel-lustre.org/2016-May/003842.html</a><br>
which was<br>
last edited on 2016.<br>
So do i need to change lnet configuration described in this page ?<br>
Or the problem has been resolved in new version (like 2.12.x) ?<br>
Anymore where can i find more details ?<br>
<br>
Any suggestions would be appreciated.<br>
Thanks?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Hello guru.novice,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Lustre 2.12.x has some nice LNet configuration abilities. The old /etc/modprobe.d/ config files have been superceded by /etc/lnet.conf. An install of Lustre 2.12.x provides a sample of this file (with the lines
commented out). Our experience has shown that not all lines are necessary; edit to suit. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">The Lustre 2.12.x has Multi-Rail (MR) on by default so Lustre will attempt to automatically find active and viable LNet paths to use. This should have no issue with your mlx4/5 mix environment; we have some mixed
IB and eth that work. To explicitly use MR one may set "Multi-Rail: true" in the "peer" NID section of the /etc/lnet.conf file. But that was not necessary for us. We used a simple /etc/lnet.conf for MR systems:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">File stub: /etc/lnet.conf<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">net:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> - net type: o2ib0<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> local NI(s):<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> - interfaces:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> 0: ib0<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> - net type: o2ib777<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> local NI(s):<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> - interfaces:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> 0: ib0:1<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">This allowed LNet to use any NID o2ib0 and o2ib777.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Whatever is placed in the /etc/lnet.conf file is loaded into the kernel modules used via the Lustre starting mechanism (CentOS uses /usr/lib/systemd/system). Because we are choosing _not_ to use MR on a different
box, we explicitly defined the available routes in /etc/lnet.conf using the lines:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">route:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> - net: tcp<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> gateway: 10.10.10.101@o2ib11111<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> - net: tcp<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> gateway: 10.10.10.102@o2ib1111<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">And so on up to 10.10.10.116@o2ib1111<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> In CentOS7, /usr/lib/systemd/system/lnet.service file is reproduced below. (details: lustre-2.12.4-1 with Mellanox OFED version 4.7-1.0.0.1 and kernel 3.10.957.27.2.el7)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">File lnet.service:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">[unit]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Description=lnet management<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Requires=network-online.target<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">After=network-online.target openibd.service rdma.service opa.service<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">ConditionsPathExists=!/proc/sys/lnet/<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">[Service]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Type=oneshot<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">RemainAfterExit=true<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">ExecStart=/sbin/modprobe lnet<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">ExecStart=/usr/sbin/lnetctl lnet configure<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">ExecStart=/usr/sbin/lnetctl set discover 0 <--Do NOT use this line if you want MR function<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">ExecStart=/usr/sbin/lnetctl import /etc/lnet.conf <--The file with router, credit and similar info<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">ExecStart=/usr/sbin/lnetctl peer add --nid 10.10.10.[101-116]@o2ib11111 --non_mr <--Omit non_rm if you want to use MR<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">ExecStop=/usr/sbin/lustre_rmmod ptlrpc<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">ExecStop=/usr/sbin/lnetctl lnet unconfigure<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">ExecStop=/usr/sbin/lustre_rmmod libcfs ldiskfs<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">[Install]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">WantedBy=multi-user.target<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">I hope this info can help you in the right direction.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Cheers,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">megan<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">PS. [I am willing to add/contribute to the
<a href="http://wiki.lustre.org/Infiniband_Configuration_Howto" target="_blank">
http://wiki.lustre.org/Infiniband_Configuration_Howto</a> but I think my account for wiki editing has expired (at least the one I thought I had did not work).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Our site had issues with Multi-Rail "not socially distancing appropriately" from other LNet networks so in our particular case we disabled MR. (An entirely different experience.) ]<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div>