[lustre-discuss] lustre-discuss Digest, Vol 227, Issue 4
夏天
313680712 at qq.com
Fri Feb 7 17:22:40 PST 2025
hi,
i guess that's kernel version compatible. do you try with kernel 6.8.0-38 ?
夏天
313680712 at qq.com
Original
From:lustre-discuss-request <lustre-discuss-request at lists.lustre.org>
Sent Time:2025-02-8- 02:27
To:lustre-discuss <lustre-discuss at lists.lustre.org>
Subject:lustre-discuss Digest, Vol 227, Issue 4
Send lustre-discuss mailing list submissions to
lustre-discuss at lists.lustre.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
or, via email, send a message with subject or body 'help' to
lustre-discuss-request at lists.lustre.org
You can reach the person managing the list at
lustre-discuss-owner at lists.lustre.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of lustre-discuss digest..."
Today's Topics:
1. Problem with 2.16.1 server on Ubuntu Noble 24.04 (?ke Sandgren)
2. Re: Lnet not going up with InfiniHost III Lx HCA card
(Jesse Stroik)
3. Re: Lustre and ZFS draid (Cameron Harr)
----------------------------------------------------------------------
Message: 1
Date: Fri, 7 Feb 2025 15:43:02 +0000
From: ?ke Sandgren
To: "lustre-discuss at lists.lustre.org"
Subject: [lustre-discuss] Problem with 2.16.1 server on Ubuntu Noble
24.04
Message-ID:
<GVZP280MB0652BE277CCAFED07F805377E8F12 at GVZP280MB0652.SWEP280.PROD.OUTLOOK.COM>
Content-Type: text/plain; charset="iso-8859-1"
Hi!
I'm trying to get 2.16.1 working on Ubuntu 24.04 (kernel 6.8.0-52-generic Ubuntu)
I have everything built according to the procedure I used when doing this with 2.14 on Ubuntu 20.04 and 22.04.
I.e. after a bit of updating ldiskfs patches I have everything built cleanly.
However, when I try to load lnet it fails and I get this back:
===
[Fri Feb 7 16:10:28 2025] BPF: type_id=13 bits_offset=256
[Fri Feb 7 16:10:28 2025] BPF:
[Fri Feb 7 16:10:28 2025] BPF: Invalid name
[Fri Feb 7 16:10:28 2025] BPF:
[Fri Feb 7 16:10:28 2025] failed to validate module [libcfs] BTF: -22
===
Any ideas for what I managed to mess up?
I can't find anything relevant in master branch that might fix this, so did I mess up my kernel build or is this a new problem?
---
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: ake at hpc2n.umu.se ?Mobile: +46 70 7716134 ?Fax: +46 90-580 14
WWW: http://www.hpc2n.umu.se
------------------------------
Message: 2
Date: Fri, 7 Feb 2025 18:24:16 +0000
From: Jesse Stroik
To: Ramiro Alba Queipo ,
"lustre-discuss at lists.lustre.org"
Subject: Re: [lustre-discuss] Lnet not going up with InfiniHost III Lx
HCA card
Message-ID:
<PH0PR06MB77218ABE432182A84CB31CF2DAF12 at PH0PR06MB7721.namprd06.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi Ramiro,
The invalid MR size looks like you're running into a limit with your cards setting up the RDMA (o2ib) LND when bringing up the network. There may be adjustments or workarounds for it possibly including setting map_on_demand=0 as an argument to the lnet module there.
And since you are using older IB hardware on a newer OS, just a heads up: we recently ran into an issue with connectx-3 IB cards after upgrading our operating systems where we found RMDA communication to be unreliable possibly because they often would exceed the amount of connection queue pairs they could create. For us, the workaround was to use the ksocklnd instead of o2iblnd. If you have trouble getting the o2ib lustre network driver to work with this older hardware due to RDMA problems, that could be a workaround although it may not be feasible to implement depending on your networking setup.
Best,
Jesse
________________________________________
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of Ramiro Alba Queipo
Sent: Thursday, February 6, 2025 3:34 AM
To: lustre-discuss at lists.lustre.org
Subject: [lustre-discuss] Lnet not going up with InfiniHost III Lx HCA card
Hi all,
I am testing Ubuntu 24.04 (6.8.0-52-generic) client with Lustre 2.16.1 over Infiniband and using an old Mellanox DDR card (InfiniHost III Lx HCA).
- # ip -br a
options lnet networks=o2ib0(ib0)
- # modprobe lnet
- # lctl network up
LNET configure error 100: Network is down
- # tail -10 /var/log/kernel.log
LNetError: 5071:0:(o2iblnd.c:2866:kiblnd_hdev_get_attr()) Invalid mr size: 0xffffffffffffffff
LNetError: 5071:0:(o2iblnd.c:3103:kiblnd_dev_failover()) Can't get device attributes: -22
LNetError: 5071:0:(o2iblnd.c:3831:kiblnd_startup()) ko2iblnd: Can't initialize device: rc = -22
LNetError: Error -100 starting up LNI o2ib
Lustre 2.15.0 and Ubuntu 20.04 (kernel 5.4.0-198-generic) is working fine with the same hardware
Can anyone give me some advice or idea to make it work?
Thans in advance
Best regards
--
Ramiro Alba
Centre Tecnol?gic de Tranfer?ncia de Calor
http://www.cttc.upc.edu
Escola T?cnica Superior d'Enginyeries
Industrial i Aeron?utica de Terrassa
Colom 11, E-08222, Terrassa, Barcelona, Spain
Tel: (+34) 93 739 8928
------------------------------
Message: 3
Date: Fri, 7 Feb 2025 10:26:49 -0800
From: Cameron Harr
To: lustre-discuss at lists.lustre.org
Subject: Re: [lustre-discuss] Lustre and ZFS draid
Message-ID: <80ce762b-3921-42a4-a3ce-1d89209528d7 at llnl.gov>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
We've been using draid in production since 2020 and I think were
generally happy with it. We have quite a few Lustre clusters and on the
majority of them, we run 90-drive JBODs with 1 OST/OSS node, 1 OST/pool
and 1 pool/JBOD. We use a draid2:8d:90c:2s config and let the
distributed spares rebuild (~2-4 hours) before replacing the 16TB
physical disk, which then rebuilds within a day or so.
An important note with this configuration is that we also include NVMe
in the pool as special allocation devices configured to store small
blocks up to 16K. We probably have much more NVMe space than we need due
to large NVMe drives (zpool list -v shows each mirror capacity is still
low), but we're happy with performance.
NAME STATE READ WRITE CKSUM
asp8 ONLINE 0 0 0
draid2:8d:90c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
L88 ONLINE 0 0 0
L89 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
N6 ONLINE 0 0 0
N7 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
N8 ONLINE 0 0 0
N9 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
N10 ONLINE 0 0 0
N11 ONLINE 0 0 0
spares
draid2-0-0 AVAIL
draid2-0-1 AVAIL
On our newest systems, we have some 106-drive JBODs with 20TB drives and
in order to reduce the chance of multiple disk failures in a single
draid device, we reconfigured the pools to have 2 draid devices per
pool, though still one OST per pool and one OST per OSS. In this config
we only have one distributed spare per draid. Due to significant write
performance reasons we also (reluctantly) started spanning pools across
2 JBODs. An additional difference is we had much less NVMe capacity on
these systems with just one small pair of NVMe drives per enclosure, so
we configure them as special devices for pool metadata rather than for
small block storage. The config for one of those pools looks like the
following:
NAME STATE READ WRITE CKSUM
merced239 ONLINE 0 0 0
draid2:11d:53c:1s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L2 ONLINE 0 0 0
L4 ONLINE 0 0 0
...
L100 ONLINE 0 0 0
L102 ONLINE 0 0 0
L104 ONLINE 0 0 0
draid2:11d:53c:1s-1 ONLINE 0 0 0
U1 ONLINE 0 0 0
U3 ONLINE 0 0 0
U5 ONLINE 0 0 0
...
U101 ONLINE 0 0 0
U103 ONLINE 0 0 0
U105 ONLINE 0 0 0
special
mirror-2 ONLINE 0 0 0
N2 ONLINE 0 0 0
N3 ONLINE 0 0 0
spares
draid2-0-0 AVAIL
draid2-1-0 AVAIL
Hope this helps,
Cameron
On 2/6/25 11:29 AM, Nehring, Shane R [ITS] wrote:
> Hello All,
>
> I didn't want to hijack the other thread today about draid, but I have been
> meaning to ask questions about it and folks' experience with it in the context
> of Lustre. Most of my questions come from not having a chance to really play
> around with draid much.
>
> Have you been generally satisfied with performance of a single draid vdev vs
> either multiple pools/osts per node or single osts on a pool spanning multiple
> raidz(2) vdev members? Is random io comparable to a span of raidz2 vdevs? I know
> one of the pain points (more from a space usage perspective as I understand it)
> is the fixed stripe width and how that impacts small files, but does small file
> io perform particularly badly on draid vs a span raidz2?
>
> I've got hardware on order (a couple 60 bay jbods and heads) that's going to
> replace some of the older OSTs in our current volume and I'm leaning toward a
> single draid pool OST per OSS. I plan to do some benchmarking of the pools in
> various configurations, but it's hard to generate a benchmark that's actually
> representative of real world usage.
>
> If you've got any insights or anecdotes regarding your experience with draid and
> Lustre I'd love to hear them!
>
> Thanks,
> Shane
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
------------------------------
Subject: Digest Footer
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
------------------------------
End of lustre-discuss Digest, Vol 227, Issue 4
**********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250208/407dbccc/attachment-0001.htm>
More information about the lustre-discuss
mailing list