<div dir="ltr">Thanks Rick we were going to go that route but ran across this (ironic being that they are now Nvidia) section of the Bright Computing (now Nvidia Base Manager) <a href="https://support.brightcomputing.com/manuals/9.2/admin-manual.pdf#page.124">admin manual</a>:<div>\u201c<b>Drivers To Use For NFS over RDMA Must Be From The Parent Distribution</b><br>The use of the RDMA protocol to provide NFS, by installing updated cluster manager OFED<br>drivers is currently <b>not</b> supported. This is because these drivers are packaged by Bright Computing from the vendor (Mellanox or Qlogic) releases, and the vendor releases themselves do not support NFS over RDMA.</div><div><br>The option can be selected, but NFS will fall back to using the default NFS TCP/IP protocol. When using NFS over RDMA, ibnet, the IP network used for InfiniBand, should be set." <br></div><div><br></div><div>So it MAY have worked to do a new build, but now we know we'd likly to use MOFED from Nvidia. </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 9, 2024 at 3:15\u202fAM Mohr, Rick <<a href="mailto:mohrrf@ornl.gov" target="_blank">mohrrf@ornl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>
Rob,<br>
<br>
Those errors appear to be IB/OFED related and not ZFS related. This can happen when you install a version of OFED that is different from the version that the Lustre packages were compiled against.<br>
<br>
-Rick<br>
<br>
<br>
<br>
\ufeffOn 8/8/24, 9:28 AM, "lustre-discuss on behalf of Rob Kudyba via lustre-discuss" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a> <mailto:<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a> <mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>>> wrote:<br>
<br>
<br>
Thanks Peter but we're using ZFS for pools with the NVME's and for ZFS snapshots. We also use Bright Computing 9.2 now called Nvidia Base Manager. In their manual for RDMA they mention this:"Drivers To Use For NFS over RDMA Must Be From The Parent Distribution<br>
The use of the RDMA protocol (section 3.6) to provide NFS, by installing updated cluster manager OFED<br>
drivers (section 7.6 of the Installation Manual) is currently not supported. This is because these drivers are<br>
packaged by Bright Computing from the vendor (Mellanox or Qlogic) releases, and the vendor releases<br>
themselves do not support NFS over RDMA.<br>
The option can be selected, but NFS will fall back to using the default NFS TCP/IP protocol.<br>
When using NFS over RDMA, ibnet, the IP network used for InfiniBand, should be set. Section 3.6.3<br>
explains how that can be done."<br>
<br>
<br>
<br>
<br>
<br>
<br>
Well we used the Bright-provided OFED when using the already-installer version 2.14, specifically lustre-client-2.14.0_ddn136. Could that be the cause of the ksym errors?<br>
<br>
<br>
<br>
<br>
On Wed, Aug 7, 2024 at 4:38 PM Peter Jones <<a href="mailto:pjones@whamcloud.com" target="_blank">pjones@whamcloud.com</a> <mailto:<a href="mailto:pjones@whamcloud.com" target="_blank">pjones@whamcloud.com</a>> <mailto:<a href="mailto:pjones@whamcloud.com" target="_blank">pjones@whamcloud.com</a> <mailto:<a href="mailto:pjones@whamcloud.com" target="_blank">pjones@whamcloud.com</a>>>> wrote:<br>
<br>
<br>
Rob <br>
<br>
<br>
If you\u2019re wanting to run Lustre on ZFS I would recommend that you use the latest community LTS release \u2013 2.15.5. <br>
<br>
<br>
Peter <br>
<br>
<br>
From: lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a> <mailto:<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> <_blank>> on behalf of Rob Kudyba via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a> <mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>> <_blank>><br>
Reply-To: Rob Kudyba <<a href="mailto:rk3199@columbia.edu" target="_blank">rk3199@columbia.edu</a> <mailto:<a href="mailto:rk3199@columbia.edu" target="_blank">rk3199@columbia.edu</a>> <_blank>><br>
Date: Wednesday, August 7, 2024 at 12:56 PM<br>
To: "<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a> <mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>> <_blank>" <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a> <mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>> <_blank>><br>
Subject: [lustre-discuss] nothing provides ksym with ZFS and Lustre 2.14 from DDN <br>
<br>
<br>
<br>
<br>
<br>
<br>
Do we need to rebuild the Lustre packages with options for ZFS? Or is there a missing package(s)? <br>
<br>
<br>
yum install ./kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64.rpm ./lustre-client-2.14.0_ddn136-1.el9.x86_64.rpm ./lustre-client-devel-2.14.0_ddn136-1.el9.x86_64.rpm --nobest <br>
Updating Subscription Management repositories. <br>
CUIT_EL9_RPMS 32 kB/s | 2.1 kB 00:00 <br>
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) 34 kB/s | 2.4 kB 00:00 <br>
Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs) 29 kB/s | 2.1 kB 00:00 <br>
Red Hat Enterprise Linux 9 for x86_64 - Supplementary (RPMs) 31 kB/s | 2.1 kB 00:00 <br>
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) 42 kB/s | 2.8 kB 00:00 <br>
Red Hat CodeReady Linux Builder for RHEL 9 x86_64 (RPMs) 44 kB/s | 2.8 kB 00:00 <br>
Error: <br>
Problem 1: conflicting requests <br>
- nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_query_port) = 0x85125bb5 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_listen) = 0x1d60b437 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_notify) = 0xa75102da needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_reject) = 0x10d4f713 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
Problem 2: package lustre-client-devel-2.14.0_ddn136-1.el9.x86_64 from @commandline requires kmod-lustre-client = 2.14.0_ddn136, but none of the providers can be installed <br>
- conflicting requests <br>
- nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_query_port) = 0x85125bb5 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_listen) = 0x1d60b437 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_notify) = 0xa75102da needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_reject) = 0x10d4f713 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
Problem 3: package lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline requires kmod-lustre-client = 2.14.0_ddn136, but none of the providers can be installed <br>
- conflicting requests <br>
- nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_query_port) = 0x85125bb5 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_listen) = 0x1d60b437 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_notify) = 0xa75102da needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_reject) = 0x10d4f713 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
- nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline <br>
(try to add '--skip-broken' to skip uninstallable packages) <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</blockquote></div>