<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Megan,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">You wrote:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">PS. [I am willing to add/contribute to the
<a href="http://wiki.lustre.org/Infiniband_Configuration_Howto">
http://wiki.lustre.org/Infiniband_Configuration_Howto</a> but I think my account for wiki editing has expired (at least the one I thought I had did not work).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you for your offer! Did you try <a href="http://wiki.lustre.org/Special:PasswordReset">
http://wiki.lustre.org/Special:PasswordReset</a>? If that didn’t work then I think that you could email
<a href="mailto:lustre.org@lists.opensfs.org">lustre.org@lists.opensfs.org</a>.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Cory<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 6/24/20, 3:33 PM, "lustre-discuss on behalf of Ms. Megan Larko" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a> on behalf of
<a href="mailto:dobsonunit@gmail.com">dobsonunit@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 22 Jun 2020 "guru.novice" wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi, all<br>
We setup up a cluster use mlx4 and mlx5 driver mixed?all things goes well.<br>
Later I find something in wiki<br>
<a href="http://wiki.lustre.org/Infiniband_Configuration_Howto">http://wiki.lustre.org/Infiniband_Configuration_Howto</a>
and<br>
<a href="http://lists.onebuilding.org/pipermail/lustre-devel-lustre.org/2016-May/003842.html">http://lists.onebuilding.org/pipermail/lustre-devel-lustre.org/2016-May/003842.html</a><br>
which was<br>
last edited on 2016.<br>
So do i need to change lnet configuration described in this page ?<br>
Or the problem has been resolved in new version (like 2.12.x) ?<br>
Anymore where can i find more details ?<br>
<br>
Any suggestions would be appreciated.<br>
Thanks?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hello guru.novice,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Lustre 2.12.x has some nice LNet configuration abilities. The old /etc/modprobe.d/ config files have been superceded by /etc/lnet.conf. An install of Lustre 2.12.x provides a sample of this file (with the lines
commented out). Our experience has shown that not all lines are necessary; edit to suit. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">The Lustre 2.12.x has Multi-Rail (MR) on by default so Lustre will attempt to automatically find active and viable LNet paths to use. This should have no issue with your mlx4/5 mix environment; we have some mixed
IB and eth that work. To explicitly use MR one may set "Multi-Rail: true" in the "peer" NID section of the /etc/lnet.conf file. But that was not necessary for us. We used a simple /etc/lnet.conf for MR systems:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">File stub: /etc/lnet.conf<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">net:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> - net type: o2ib0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> local NI(s):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> - interfaces:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> 0: ib0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> - net type: o2ib777<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> local NI(s):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> - interfaces:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> 0: ib0:1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">This allowed LNet to use any NID o2ib0 and o2ib777.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Whatever is placed in the /etc/lnet.conf file is loaded into the kernel modules used via the Lustre starting mechanism (CentOS uses /usr/lib/systemd/system). Because we are choosing _not_ to use MR on a different
box, we explicitly defined the available routes in /etc/lnet.conf using the lines:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">route:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> - net: tcp<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> gateway: 10.10.10.101@o2ib11111<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> - net: tcp<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> gateway: 10.10.10.102@o2ib1111<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">And so on up to 10.10.10.116@o2ib1111<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> In CentOS7, /usr/lib/systemd/system/lnet.service file is reproduced below. (details: lustre-2.12.4-1 with Mellanox OFED version 4.7-1.0.0.1 and kernel 3.10.957.27.2.el7)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">File lnet.service:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">[unit]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Description=lnet management<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Requires=network-online.target<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">After=network-online.target openibd.service rdma.service opa.service<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ConditionsPathExists=!/proc/sys/lnet/<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">[Service]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Type=oneshot<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">RemainAfterExit=true<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ExecStart=/sbin/modprobe lnet<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ExecStart=/usr/sbin/lnetctl lnet configure<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ExecStart=/usr/sbin/lnetctl set discover 0 <--Do NOT use this line if you want MR function<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ExecStart=/usr/sbin/lnetctl import /etc/lnet.conf <--The file with router, credit and similar info<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ExecStart=/usr/sbin/lnetctl peer add --nid 10.10.10.[101-116]@o2ib11111 --non_mr <--Omit non_rm if you want to use MR<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ExecStop=/usr/sbin/lustre_rmmod ptlrpc<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ExecStop=/usr/sbin/lnetctl lnet unconfigure<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ExecStop=/usr/sbin/lustre_rmmod libcfs ldiskfs<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">[Install]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">WantedBy=multi-user.target<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I hope this info can help you in the right direction.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Cheers,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">megan<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">PS. [I am willing to add/contribute to the
<a href="http://wiki.lustre.org/Infiniband_Configuration_Howto">
http://wiki.lustre.org/Infiniband_Configuration_Howto</a> but I think my account for wiki editing has expired (at least the one I thought I had did not work).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Our site had issues with Multi-Rail "not socially distancing appropriately" from other LNet networks so in our particular case we disabled MR. (An entirely different experience.) ]<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>