<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div>Looks like your client isn’t set up to use o2ib. What’s your lnet configuration?</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Emyr James <emyr.james@crg.eu><br>
<b>Sent:</b> Monday, November 1, 2021 4:31 PM<br>
<b>To:</b> Patrick Farrell <pfarrell@ddn.com>; lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> Re: [lustre-discuss] Lustre Client on Ubuntu 20.04</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Dear all,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I tried to get it working on centos 7.6 and am <b>almost</b> there...</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I installed Mellanox OFED (MLNX_OFED_LINUX-4.9-4.0.8.0-rhel7.6-x86_64.tgz).</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
It seemed to go through fine - it detected the connectx3 card and updated the firmware.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I then went for a build from src.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I had an issue that the out-of-the-box centos7.6 has kernel 3.10.0-957 but when I installed kernel-devl package it added the headers for 3.10.0-1062.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
In the end I had to get a package from the vault</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
(<a href="https://stackoverflow.com/questions/58304570/no-package-kernel-devel-3-10-0-957-el7-x86-64-available" id="LPNoLPOWALinkPreview">https://stackoverflow.com/questions/58304570/no-package-kernel-devel-3-10-0-957-el7-x86-64-available</a>)<br>
</div>
<div class="x__Entity x__EType_OWALinkPreview x__EId_OWALinkPreview x__EReadonly_1">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
At this point using <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
./configure --disable-server --enable-client --with-o2ib=/usr/src/ofa_kernel/default<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
failed and I had to tweak some files as per <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<a href="https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg15829.htmla" id="LPlnk413937">https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg15829.html</a></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
After that the build seemed to go to completion and I was left with some rpms. I did yum localinstall on these and tried to mount but this is where I'm stuck.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
mount -t lustre 10.3.11.32@o2ib:/scratchB /scratch
<div>mount.lustre: mount 10.3.11.32@o2ib:/scratchB at /scratch failed: No such file or directory</div>
<div>Is the MGS specification correct?</div>
<div>Is the filesystem name correct?</div>
If upgrading, is the copied client log valid? (see upgrade docs)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I tried...</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
lctl ping 10.3.11.32@o2ib<br>
failed to ping 10.3.11.32@o2ib: Input/output error</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
On one of the current compute nodes with Centos6.7 and client 2.7 I get...</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
lctl ping 10.3.11.32@o2ib
<div>12345-0@lo</div>
<div>12345-10.3.11.32@o2ib</div>
12345-10.4.11.32@o2ib1</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
On the new node...</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
ping 10.3.11.32
<div>PING 10.3.11.32 (10.3.11.32) 56(84) bytes of data.</div>
<div>64 bytes from 10.3.11.32: icmp_seq=1 ttl=64 time=2.41 ms</div>
<span>64 bytes from 10.3.11.32: icmp_seq=2 ttl=64 time=0.130 ms</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>So it looks like ipoib works ok.</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>I see this in /var/log/messages...</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Nov 1 21:23:44 glacier01 kernel: LustreError: 6045:0:(ldlm_lib.c:483:client_obd_setup()) can't add initial connection
<div>Nov 1 21:23:44 glacier01 kernel: LustreError: 6045:0:(obd_config.c:558:class_setup()) setup MGC10.3.11.32@o2ib failed (-2)</div>
<div>Nov 1 21:23:44 glacier01 kernel: LustreError: 6045:0:(obd_mount.c:203:lustre_start_simple()) MGC10.3.11.32@o2ib setup error -2</div>
<div>Nov 1 21:23:44 glacier01 kernel: LustreError: 6045:0:(obd_mount.c:1582:lustre_fill_super()) Unable to mount (-2)</div>
Nov 1 21:24:13 glacier01 kernel: LNetError: 6054:0:(lib-move.c:1557:lnet_select_pathway()) no route to 10.3.11.32@o2ib</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:12pt"><span><br>
</span></span><br>
<span style="font-size:12pt">I'm not sure if this is something to do with my lustre client build or if it's to do with the infiniband network on this new compute node.</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Anyone got any ideas?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Many thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_Signature">
<div>
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
Emyr James<br>
Head of IT<br>
CRG - Centre for Genomic Regulation<br>
C/ Dr. Aiguader, 88<br>
Edif. PRBB<br>
08003 Barcelona, Spain</div>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
Phone Ext: #1098</div>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<br>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div id="x_appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Emyr James <emyr.james@crg.eu><br>
<b>Sent:</b> 21 October 2021 23:04<br>
<b>To:</b> Patrick Farrell <pfarrell@ddn.com>; lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> Re: Lustre Client on Ubuntu 20.04</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div>Thanks for the info. Our server side will get upgraded to 2.12.7 LTS in a few months so looking for a more modern OS than redhat 6.7!
<br>
</div>
<div><br>
</div>
<div>Does the 2.14 client work with the 2.12.7 LTS...and even with 2.7?</div>
<div><br>
</div>
<div>Many thanks,</div>
<div><br>
<div id="x_x_Signature">
<div>
<div id="x_x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<div name="x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div name="x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
Emyr James<br>
Head of IT<br>
CRG - Centre for Genomic Regulation<br>
C/ Dr. Aiguader, 88<br>
Edif. PRBB<br>
08003 Barcelona, Spain</div>
<div name="x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
Phone Ext: #1098</div>
<div name="x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<br>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div id="x_x_appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Patrick Farrell <pfarrell@ddn.com><br>
<b>Sent:</b> 21 October 2021 22:57<br>
<b>To:</b> Emyr James <emyr.james@crg.eu>; lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> Re: Lustre Client on Ubuntu 20.04</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Emyr,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
2.10.8 is a fairly old version and I am pretty sure it won't build for Ubuntu 20.04. The most recent kernel it's known to build for is Linux 4.4.x, and Ubuntu 20.04 is
<b>at least</b> 5.4.x.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The only version which officially has support for 20.04 is Lustre 2.14 (and the upcoming 2.15).<br>
<br>
For Ubuntu builds, the instructions here give the basic steps:<br>
<a href="https://wiki.lustre.org/Compiling_Lustre" id="LPNoLPOWALinkPreview">https://wiki.lustre.org/Compiling_Lustre</a><br>
<div class="x_x_x__Entity x_x_x__EType_OWALinkPreview x_x_x__EId_OWALinkPreview x_x_x__EReadonly_1">
</div>
<br>
And the other sections have instructions which will help you with adding in MOFED.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Just bear in mind the comments about version compatibility.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
-Patrick</div>
<div id="x_x_x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Emyr James <emyr.james@crg.eu><br>
<b>Sent:</b> Thursday, October 21, 2021 3:28 PM<br>
<b>To:</b> lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> [lustre-discuss] Lustre Client on Ubuntu 20.04</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Dear all,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
We currently have a lustre 2.7 system running on redhat 6.7.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The 2.10.8 client has been tested on redhat 6.7 and works fine (using Mellanox OFED).</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I'd like to get a 2.10.8 client running on Ubuntu 20.04 which is compatible with Mellanox OFED.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I haven't found any instructions on how to build just the client side on Ubuntu. Does anyone have instructions or can point me to a howto somewhere?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Many thanks.<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_x_x_x_Signature">
<div>
<div id="x_x_x_x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<div name="x_x_x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div name="x_x_x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
Emyr James<br>
Head of IT<br>
CRG - Centre for Genomic Regulation<br>
C/ Dr. Aiguader, 88<br>
Edif. PRBB<br>
08003 Barcelona, Spain</div>
<div name="x_x_x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
Phone Ext: #1098</div>
<div name="x_x_x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>