<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
There is a "Request Account" link at the top of every page: <a href="http://wiki.lustre.org/Special:RequestAccount" class="">
http://wiki.lustre.org/Special:RequestAccount</a>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 1, 2020, at 08:39, Ms. Megan Larko <<a href="mailto:dobsonunit@gmail.com" class="">dobsonunit@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Awesome, thanks! Unfortunately the password reset site is not finding my UID. Maybe I never had access to the Lustre wiki. (I have so many accounts that sometimes my head spins.) I'm still willing to help. Is there a request
password site?<br class="">
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class="">megan</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 8:54 PM Spitz, Cory James <<a href="mailto:cory.spitz@hpe.com" class="">cory.spitz@hpe.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div lang="EN-US" class="">
<div class="gmail-m_4501481448440832035WordSection1">
<p class="MsoNormal">Megan,<u class=""></u><u class=""></u></p>
<p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<p class="MsoNormal">You wrote:<u class=""></u><u class=""></u></p>
<p class="MsoNormal" style="margin-left:0.5in">PS. [I am willing to add/contribute to the
<a href="http://wiki.lustre.org/Infiniband_Configuration_Howto" target="_blank" class="">
http://wiki.lustre.org/Infiniband_Configuration_Howto</a> but I think my account for wiki editing has expired (at least the one I thought I had did not work).<u class=""></u><u class=""></u></p>
<p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<p class="MsoNormal">Thank you for your offer! Did you try <a href="http://wiki.lustre.org/Special:PasswordReset" target="_blank" class="">
http://wiki.lustre.org/Special:PasswordReset</a>? If that didn’t work then I think that you could email
<a href="mailto:lustre.org@lists.opensfs.org" target="_blank" class="">lustre.org@lists.opensfs.org</a>.<u class=""></u><u class=""></u></p>
<p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<p class="MsoNormal">-Cory<u class=""></u><u class=""></u></p>
<p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<div class="">
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">On 6/24/20, 3:33 PM, "lustre-discuss on behalf of Ms. Megan Larko" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank" class="">lustre-discuss-bounces@lists.lustre.org</a> on behalf of
<a href="mailto:dobsonunit@gmail.com" target="_blank" class="">dobsonunit@gmail.com</a>> wrote:<u class=""></u><u class=""></u></p>
</div>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">On 22 Jun 2020 "guru.novice" wrote:<u class=""></u><u class=""></u></p>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">Hi, all<br class="">
We setup up a cluster use mlx4 and mlx5 driver mixed?all things goes well.<br class="">
Later I find something in wiki<br class="">
<a href="http://wiki.lustre.org/Infiniband_Configuration_Howto" target="_blank" class="">http://wiki.lustre.org/Infiniband_Configuration_Howto</a> and<br class="">
<a href="http://lists.onebuilding.org/pipermail/lustre-devel-lustre.org/2016-May/003842.html" target="_blank" class="">http://lists.onebuilding.org/pipermail/lustre-devel-lustre.org/2016-May/003842.html</a><br class="">
which was<br class="">
last edited on 2016.<br class="">
So do i need to change lnet configuration described in this page ?<br class="">
Or the problem has been resolved in new version (like 2.12.x) ?<br class="">
Anymore where can i find more details ?<br class="">
<br class="">
Any suggestions would be appreciated.<br class="">
Thanks?<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">Hello guru.novice,<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">Lustre 2.12.x has some nice LNet configuration abilities. The old /etc/modprobe.d/ config files have been superceded by /etc/lnet.conf. An install of Lustre 2.12.x provides a sample of this file (with the lines
commented out). Our experience has shown that not all lines are necessary; edit to suit. <u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">The Lustre 2.12.x has Multi-Rail (MR) on by default so Lustre will attempt to automatically find active and viable LNet paths to use. This should have no issue with your mlx4/5 mix environment; we have some mixed
IB and eth that work. To explicitly use MR one may set "Multi-Rail: true" in the "peer" NID section of the /etc/lnet.conf file. But that was not necessary for us. We used a simple /etc/lnet.conf for MR systems:<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">File stub: /etc/lnet.conf<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">net:<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> - net type: o2ib0<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> local NI(s):<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> - interfaces:<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> 0: ib0<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> - net type: o2ib777<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> local NI(s):<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> - interfaces:<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> 0: ib0:1<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">This allowed LNet to use any NID o2ib0 and o2ib777.<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">Whatever is placed in the /etc/lnet.conf file is loaded into the kernel modules used via the Lustre starting mechanism (CentOS uses /usr/lib/systemd/system). Because we are choosing _not_ to use MR on a different
box, we explicitly defined the available routes in /etc/lnet.conf using the lines:<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">route:<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> - net: tcp<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> gateway: 10.10.10.101@o2ib11111<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> - net: tcp<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> gateway: 10.10.10.102@o2ib1111<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">And so on up to 10.10.10.116@o2ib1111<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"> In CentOS7, /usr/lib/systemd/system/lnet.service file is reproduced below. (details: lustre-2.12.4-1 with Mellanox OFED version 4.7-1.0.0.1 and kernel 3.10.957.27.2.el7)<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">File lnet.service:<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">[unit]<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">Description=lnet management<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">Requires=network-online.target<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">After=network-online.target openibd.service rdma.service opa.service<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">ConditionsPathExists=!/proc/sys/lnet/<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">[Service]<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">Type=oneshot<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">RemainAfterExit=true<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">ExecStart=/sbin/modprobe lnet<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">ExecStart=/usr/sbin/lnetctl lnet configure<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">ExecStart=/usr/sbin/lnetctl set discover 0 <--Do NOT use this line if you want MR function<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">ExecStart=/usr/sbin/lnetctl import /etc/lnet.conf <--The file with router, credit and similar info<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">ExecStart=/usr/sbin/lnetctl peer add --nid 10.10.10.[101-116]@o2ib11111 --non_mr <--Omit non_rm if you want to use MR<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">ExecStop=/usr/sbin/lustre_rmmod ptlrpc<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">ExecStop=/usr/sbin/lnetctl lnet unconfigure<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">ExecStop=/usr/sbin/lustre_rmmod libcfs ldiskfs<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">[Install]<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">WantedBy=multi-user.target<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">I hope this info can help you in the right direction.<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">Cheers,<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">megan<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">PS. [I am willing to add/contribute to the
<a href="http://wiki.lustre.org/Infiniband_Configuration_Howto" target="_blank" class="">
http://wiki.lustre.org/Infiniband_Configuration_Howto</a> but I think my account for wiki editing has expired (at least the one I thought I had did not work).<u class=""></u><u class=""></u></p>
</div>
<div class="">
<p class="MsoNormal" style="margin-left:0.5in">Our site had issues with Multi-Rail "not socially distancing appropriately" from other LNet networks so in our particular case we disabled MR. (An entirely different experience.) ]<u class=""></u><u class=""></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>