[lustre-discuss] nothing provides ksym with ZFS and Lustre 2.14 from DDN
Peter Jones
pjones at whamcloud.com
Thu Aug 8 06:30:34 PDT 2024
Rob
I think that you’re going to need to re-engage with your vendor to get guidance here. This forum is for discussion relating to community releases of Lustre and that is a DDN version of Lustre (which is not supported/tested in conjunction with ZFS)
Peter
From: Rob Kudyba <rk3199 at columbia.edu>
Date: Thursday, August 8, 2024 at 6:25 AM
To: Peter Jones <pjones at whamcloud.com>
Cc: "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] nothing provides ksym with ZFS and Lustre 2.14 from DDN
Thanks Peter but we're using ZFS for pools with the NVME's and for ZFS snapshots. We also use Bright Computing 9.2 now called Nvidia Base Manager. In their manual for RDMA they mention this:
"Drivers To Use For NFS over RDMA Must Be From The Parent Distribution
The use of the RDMA protocol (section 3.6) to provide NFS, by installing updated cluster manager OFED
drivers (section 7.6 of the Installation Manual) is currently not supported. This is because these drivers are
packaged by Bright Computing from the vendor (Mellanox or Qlogic) releases, and the vendor releases
themselves do not support NFS over RDMA.
The option can be selected, but NFS will fall back to using the default NFS TCP/IP protocol.
When using NFS over RDMA, ibnet, the IP network used for InfiniBand, should be set. Section 3.6.3
explains how that can be done."
Well we used the Bright-provided OFED when using the already-installer version 2.14, specifically lustre-client-2.14.0_ddn136. Could that be the cause of the ksym errors?
On Wed, Aug 7, 2024 at 4:38 PM Peter Jones <pjones at whamcloud.com<mailto:pjones at whamcloud.com>> wrote:
Rob
If you’re wanting to run Lustre on ZFS I would recommend that you use the latest community LTS release – 2.15.5.
Peter
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org>> on behalf of Rob Kudyba via lustre-discuss <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>>
Reply-To: Rob Kudyba <rk3199 at columbia.edu<mailto:rk3199 at columbia.edu>>
Date: Wednesday, August 7, 2024 at 12:56 PM
To: "lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>" <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>>
Subject: [lustre-discuss] nothing provides ksym with ZFS and Lustre 2.14 from DDN
Do we need to rebuild the Lustre packages with options for ZFS? Or is there a missing package(s)?
yum install ./kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64.rpm ./lustre-client-2.14.0_ddn136-1.el9.x86_64.rpm ./lustre-client-devel-2.14.0_ddn136-1.el9.x86_64.rpm --nobest
Updating Subscription Management repositories.
CUIT_EL9_RPMS 32 kB/s | 2.1 kB 00:00
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) 34 kB/s | 2.4 kB 00:00
Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs) 29 kB/s | 2.1 kB 00:00
Red Hat Enterprise Linux 9 for x86_64 - Supplementary (RPMs) 31 kB/s | 2.1 kB 00:00
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) 42 kB/s | 2.8 kB 00:00
Red Hat CodeReady Linux Builder for RHEL 9 x86_64 (RPMs) 44 kB/s | 2.8 kB 00:00
Error:
Problem 1: conflicting requests
- nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_query_port) = 0x85125bb5 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_listen) = 0x1d60b437 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_notify) = 0xa75102da needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_reject) = 0x10d4f713 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
Problem 2: package lustre-client-devel-2.14.0_ddn136-1.el9.x86_64 from @commandline requires kmod-lustre-client = 2.14.0_ddn136, but none of the providers can be installed
- conflicting requests
- nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_query_port) = 0x85125bb5 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_listen) = 0x1d60b437 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_notify) = 0xa75102da needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_reject) = 0x10d4f713 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
Problem 3: package lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline requires kmod-lustre-client = 2.14.0_ddn136, but none of the providers can be installed
- conflicting requests
- nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_query_port) = 0x85125bb5 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_listen) = 0x1d60b437 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_notify) = 0xa75102da needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_reject) = 0x10d4f713 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
- nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
(try to add '--skip-broken' to skip uninstallable packages)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20240808/933d3c3b/attachment-0001.htm>
More information about the lustre-discuss
mailing list