<div dir="ltr">Hello Folks,<div><br></div><div>To clarify my own issues with working with both Lustre server 2.12.5 and LNet routers at 2.10.4, I have in my notes from October 2020 that I received many, many lines in /var/log/messages reading:</div><div> LNet: 8759:0 (o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for <IP IB addr> 56 seconds </div><div> which was followed by </div><div>Skipped 97 previous similar messages. </div><div><br></div><div>The behavior of the Lustre File System storage was a bit (noticeably) slower when traversing LNets and clients at 2.10.4. Now I will note that the 2.10.4 Lustre clients were built with Mellanox version 4.3-1.0.1 and the Lustre 2.12.5 servers are using Mellanox OFED version 4.7-1.0.0. These were the versions of Mellanox software applied when the boxes were built. I did not investigate the "Timed out tx for <IP IB addr>" I only noticed that it was consistent for me with 2.12.6 Lustre servers and LNet routers at 2.10.4 (with the corresponding Mellanox OFED). I eliminated the obvious performance issue and messages by not using LNet routers with LFS 2.10.4/MOFED and going with an LNet router at Lustre client 2.12.2 or newer where the Lustre client 2.12.2 is using Mellanox OFED 4.5-1.0.1.0.</div><div><br></div><div>That is why I made the comment that Lustre 2.10.x may not play well with newer 2.12.x. It well could be that the differenced in the MOFED stack are more the reason than Lustre software itself. Apologies if I offended. I'm glad other people have had better luck with Lustre 2.10.x and Lustre 2.12.x versions.</div><div><br></div><div>Cheers,</div><div>megan</div><div><br></div><div>P.S. Sorry for delay in response; I was off for a few days.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 2, 2021 at 5:02 AM Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
On Mar 30, 2021, at 11:54, Spitz, Cory James via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>> wrote:<br>
<div>
<blockquote type="cite"><br>
<div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div style="margin:0in;font-size:11pt;font-family:Calibri,sans-serif">
Hello, Megan.<u></u><u></u></div>
<div style="margin:0in;font-size:11pt;font-family:Calibri,sans-serif">
<br>
I was curious why you made this comment:<u></u><u></u></div>
<div style="margin:0in;font-size:11pt;font-family:Calibri,sans-serif">
> A general example is a box with lustre-client 2.10.4 is not going to be completely happy with a new 2.12.x on the lustre network<u></u><u></u></div>
<div style="margin:0in;font-size:11pt;font-family:Calibri,sans-serif">
In general, I think that the two LTS release are very interoperable. What incompatibility are you referring to? Do you have a well-known LU or two to share?</div>
</div>
</div>
</blockquote>
<div><br>
</div>
This could potentially relate to changes with configuring Multi-Rail LNet between those releases?</div>
<div><span style="font-family:Calibri,sans-serif;font-size:11pt"> </span><span style="font-family:Calibri,sans-serif;font-size:11pt"> </span>
<blockquote type="cite">
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div>
<div>
<div style="margin:0in 0in 0in 0.5in;font-size:11pt;font-family:Calibri,sans-serif">
On 3/30/21, 12:14 PM, "lustre-discuss on behalf of Ms. Megan Larko via lustre-discuss" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" style="color:blue;text-decoration:underline" target="_blank">lustre-discuss-bounces@lists.lustre.org</a><span> </span>on
behalf of<span> </span><a href="mailto:lustre-discuss@lists.lustre.org" style="color:blue;text-decoration:underline" target="_blank">lustre-discuss@lists.lustre.org</a>> wrote:<u></u><u></u></div>
</div>
</div>
<div>
<div style="margin:0in 0in 0in 0.5in;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<div>
<div style="margin:0in 0in 0in 0.5in;font-size:11pt;font-family:Calibri,sans-serif">
Hello!<u></u><u></u></div>
<div>
<div style="margin:0in 0in 0in 0.5in;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<div>
<div style="margin:0in 0in 0in 0.5in;font-size:11pt;font-family:Calibri,sans-serif">
I have no direct experience with the DDN AI400X, but as a vendor DDN has some nice value-add to the Lustre systems they build. Having worked with other DDN Lustre hw in my career, interoperability with other Lustre mounts is usually not an issue unless the
current lustre-client software on the client boxes is a very different software version or network stack. A general example is a box with lustre-client 2.10.4 is not going to be completely happy with a new 2.12.x on the lustre network. As far as vendor lock-in,
DDN support in my past experience does have its own value-add to their Lustre storage product so it is not completely vanilla. I have found the enhancements useful. As far as your total admin control of the DDN storage product, that is probably up to the
terms of the service agreement made with purchase. My one experience with DDN on that is contractually DDN maintained the box version level and patches, standard Lustre tunables were fine for local admins. In one case we did stumble upon a bug, I was permitted
to dig around freely but not to change anything; I shared my findings with the DDN team. It worked out well for us.<u></u><u></u></div>
</div>
<div>
<div style="margin:0in 0in 0in 0.5in;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<div>
<div style="margin:0in 0in 0in 0.5in;font-size:11pt;font-family:Calibri,sans-serif">
P.S. I am not in any way employed or compensated by DDN. I'm just sharing my own experience. Smile.<u></u><u></u></div>
</div>
<div>
<div style="margin:0in 0in 0in 0.5in;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<div>
<div style="margin:0in 0in 0in 0.5in;font-size:11pt;font-family:Calibri,sans-serif">
Cheers,<u></u><u></u></div>
</div>
<div>
<div style="margin:0in 0in 0in 0.5in;font-size:11pt;font-family:Calibri,sans-serif">
megan<u></u><u></u></div>
</div>
</div>
</div>
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">lustre-discuss
mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<a href="mailto:lustre-discuss@lists.lustre.org" style="color:blue;text-decoration:underline;font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">lustre-discuss@lists.lustre.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="color:blue;text-decoration:underline;font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></blockquote>
</div>
<br>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<br>
</div>
<br>
</div>
</blockquote></div>