<div dir="ltr"><div dir="ltr">Thanks Peter but we're using ZFS for pools with the NVME's and for ZFS snapshots. We also use Bright Computing 9.2 now called Nvidia Base Manager. In their manual for RDMA they mention this:<div>"<b>Drivers To Use For NFS over RDMA Must Be From The Parent Distribution</b><br>The use of the RDMA protocol (section 3.6) to provide NFS, by installing updated cluster manager OFED<br>drivers (section 7.6 of the Installation Manual) is currently not supported. This is because these drivers are<br>packaged by Bright Computing from the vendor (Mellanox or Qlogic) releases, and the vendor releases<br>themselves do not support NFS over RDMA.<br>The option can be selected, but NFS will fall back to using the default NFS TCP/IP protocol.<br>When using NFS over RDMA, ibnet, the IP network used for InfiniBand, should be set. Section 3.6.3<br>explains how that can be done."<br></div><div><br></div><div>Well we used the Bright-provided OFED when using the already-installer version 2.14, specifically lustre-client-2.14.0_ddn136. Could that be the cause of the ksym errors?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 7, 2024 at 4:38\u202fPM Peter Jones <<a href="mailto:pjones@whamcloud.com">pjones@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div class="msg4654326749902067623">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_4654326749902067623WordSection1">
<p class="MsoNormal"><span style="font-size:11pt">Rob<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">If you\u2019re wanting to run Lustre on ZFS I would recommend that you use the latest community LTS release \u2013 2.15.5.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Peter<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<div style="border-width:1pt medium medium;border-style:solid none none;border-color:rgb(181,196,223) currentcolor currentcolor;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri,sans-serif;color:black">From:
</span></b><span style="font-family:Calibri,sans-serif;color:black">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Rob Kudyba via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Reply-To: </b>Rob Kudyba <<a href="mailto:rk3199@columbia.edu" target="_blank">rk3199@columbia.edu</a>><br>
<b>Date: </b>Wednesday, August 7, 2024 at 12:56 PM<br>
<b>To: </b>"<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Subject: </b>[lustre-discuss] nothing provides ksym with ZFS and Lustre 2.14 from DDN<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Do we need to rebuild the Lustre packages with options for ZFS? Or is there a missing package(s)?<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">yum install ./kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64.rpm ./lustre-client-2.14.0_ddn136-1.el9.x86_64.rpm ./lustre-client-devel-2.14.0_ddn136-1.el9.x86_64.rpm --nobest
<br>
Updating Subscription Management repositories. <br>
CUIT_EL9_RPMS 32 kB/s | 2.1 kB 00:00 <br>
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) 34 kB/s | 2.4 kB 00:00 <br>
Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs) 29 kB/s | 2.1 kB 00:00 <br>
Red Hat Enterprise Linux 9 for x86_64 - Supplementary (RPMs) 31 kB/s | 2.1 kB 00:00
<br>
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) 42 kB/s | 2.8 kB 00:00 <br>
Red Hat CodeReady Linux Builder for RHEL 9 x86_64 (RPMs) 44 kB/s | 2.8 kB 00:00 <br>
Error: <br>
Problem 1: conflicting requests <br>
- nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_query_port) = 0x85125bb5 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_listen) = 0x1d60b437 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_notify) = 0xa75102da needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_reject) = 0x10d4f713 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
Problem 2: package lustre-client-devel-2.14.0_ddn136-1.el9.x86_64 from @commandline requires kmod-lustre-client = 2.14.0_ddn136, but none of the providers can be installed
<br>
- conflicting requests <br>
- nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_query_port) = 0x85125bb5 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_listen) = 0x1d60b437 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_notify) = 0xa75102da needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_reject) = 0x10d4f713 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
Problem 3: package lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline requires kmod-lustre-client = 2.14.0_ddn136, but none of the providers can be installed
<br>
- conflicting requests <br>
- nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_query_port) = 0x85125bb5 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_listen) = 0x1d60b437 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_notify) = 0xa75102da needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_reject) = 0x10d4f713 needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
- nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline
<br>
(try to add '--skip-broken' to skip uninstallable packages)</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div></div>