[lustre-discuss] Lustre Client on Ubuntu 20.04

Mon Nov 1 15:05:31 PDT 2021

Looks like your client isn’t set up to use o2ib. What’s your lnet configuration?
________________________________
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of Emyr James <emyr.james at crg.eu>
Sent: Monday, November 1, 2021 4:31 PM
To: Patrick Farrell <pfarrell at ddn.com>; lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] Lustre Client on Ubuntu 20.04

Dear all,

I tried to get it working on centos 7.6 and am almost there...

I installed Mellanox OFED (MLNX_OFED_LINUX-4.9-4.0.8.0-rhel7.6-x86_64.tgz).
It seemed to go through fine - it detected the connectx3 card and updated the firmware.

I then went for a build from src.

I had an issue that the out-of-the-box centos7.6 has kernel 3.10.0-957 but when I installed kernel-devl package it added the headers for 3.10.0-1062.

In the end I had to get a package from the vault

(https://stackoverflow.com/questions/58304570/no-package-kernel-devel-3-10-0-957-el7-x86-64-available<https://stackoverflow.com/questions/58304570/no-package-kernel-devel-3-10-0-957-el7-x86-64-available>)

At this point using

./configure --disable-server --enable-client  --with-o2ib=/usr/src/ofa_kernel/default

failed and I had to tweak some files as per

https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg15829.html<https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg15829.htmla>

After that the build seemed to go to completion and I was left with some rpms. I did yum localinstall on these and tried to mount but this is where I'm stuck.

mount -t lustre 10.3.11.32 at o2ib:/scratchB /scratch
mount.lustre: mount 10.3.11.32 at o2ib:/scratchB at /scratch failed: No such file or directory
Is the MGS specification correct?
Is the filesystem name correct?
If upgrading, is the copied client log valid? (see upgrade docs)

I tried...

lctl ping 10.3.11.32 at o2ib
failed to ping 10.3.11.32 at o2ib: Input/output error

On one of the current compute nodes with Centos6.7 and client 2.7 I get...

lctl ping 10.3.11.32 at o2ib
12345-0 at lo
12345-10.3.11.32 at o2ib
12345-10.4.11.32 at o2ib1

On the new node...

 ping 10.3.11.32
PING 10.3.11.32 (10.3.11.32) 56(84) bytes of data.
64 bytes from 10.3.11.32: icmp_seq=1 ttl=64 time=2.41 ms
64 bytes from 10.3.11.32: icmp_seq=2 ttl=64 time=0.130 ms

So it looks like ipoib works ok.

I see this in /var/log/messages...

Nov  1 21:23:44 glacier01 kernel: LustreError: 6045:0:(ldlm_lib.c:483:client_obd_setup()) can't add initial connection
Nov  1 21:23:44 glacier01 kernel: LustreError: 6045:0:(obd_config.c:558:class_setup()) setup MGC10.3.11.32 at o2ib failed (-2)
Nov  1 21:23:44 glacier01 kernel: LustreError: 6045:0:(obd_mount.c:203:lustre_start_simple()) MGC10.3.11.32 at o2ib setup error -2
Nov  1 21:23:44 glacier01 kernel: LustreError: 6045:0:(obd_mount.c:1582:lustre_fill_super()) Unable to mount  (-2)
Nov  1 21:24:13 glacier01 kernel: LNetError: 6054:0:(lib-move.c:1557:lnet_select_pathway()) no route to 10.3.11.32 at o2ib

I'm not sure if this is something to do with my lustre client build or if it's to do with the infiniband network on this new compute node.

Anyone got any ideas?

Many thanks,

Emyr James
Head of IT
CRG - Centre for Genomic Regulation
C/ Dr. Aiguader, 88
Edif. PRBB
08003 Barcelona, Spain
Phone Ext: #1098

________________________________
From: Emyr James <emyr.james at crg.eu>
Sent: 21 October 2021 23:04
To: Patrick Farrell <pfarrell at ddn.com>; lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
Subject: Re: Lustre Client on Ubuntu 20.04

Hi,

Thanks for the info. Our server side will get upgraded to 2.12.7 LTS in a few months so looking for a more modern OS than redhat 6.7!

Does the 2.14 client work with the 2.12.7 LTS...and even with 2.7?

Many thanks,

Emyr James
Head of IT
CRG - Centre for Genomic Regulation
C/ Dr. Aiguader, 88
Edif. PRBB
08003 Barcelona, Spain
Phone Ext: #1098

________________________________
From: Patrick Farrell <pfarrell at ddn.com>
Sent: 21 October 2021 22:57
To: Emyr James <emyr.james at crg.eu>; lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
Subject: Re: Lustre Client on Ubuntu 20.04

Emyr,

2.10.8 is a fairly old version and I am pretty sure it won't build for Ubuntu 20.04.  The most recent kernel it's known to build for is Linux 4.4.x, and Ubuntu 20.04 is at least 5.4.x.

The only version which officially has support for 20.04 is Lustre 2.14 (and the upcoming 2.15).

For Ubuntu builds, the instructions here give the basic steps:
https://wiki.lustre.org/Compiling_Lustre<https://wiki.lustre.org/Compiling_Lustre>

And the other sections have instructions which will help you with adding in MOFED.

Just bear in mind the comments about version compatibility.

-Patrick
________________________________
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of Emyr James <emyr.james at crg.eu>
Sent: Thursday, October 21, 2021 3:28 PM
To: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
Subject: [lustre-discuss] Lustre Client on Ubuntu 20.04

Dear all,

We currently have a lustre 2.7 system running on redhat 6.7.
The 2.10.8 client has been tested on redhat 6.7 and works fine (using Mellanox OFED).
I'd like to get a 2.10.8 client running on Ubuntu 20.04 which is compatible with Mellanox OFED.
I haven't found any instructions on how to build just the client side on Ubuntu. Does anyone have instructions or can point me to a howto somewhere?

Many thanks.

Emyr James
Head of IT
CRG - Centre for Genomic Regulation
C/ Dr. Aiguader, 88
Edif. PRBB
08003 Barcelona, Spain
Phone Ext: #1098

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20211101/96fed717/attachment-0001.html>