[lustre-discuss] [EXTERNAL] Re: Lustre 2.15.1 server with ZFS nothing provides ksym

Tue Jul 11 09:47:07 PDT 2023

Jon, 

Also, I had similar errors when I missed some of the dependencies in a recent build under RL. I think it was enabling the EPEL repo and installing additional rpms like kernel-devel-$(uname -r) kernel-headers-$(uname -r) and maybe even platform-python-devel. 

Hope that helps, 
-Ian Schroeder-Anderson

On 7/11/23, 10:15 AM, "lustre-discuss on behalf of Bertschinger, Thomas Andrew Hjorth via lustre-discuss" <lustre-discuss-bounces at lists.lustre.org <mailto:lustre-discuss-bounces at lists.lustre.org> on behalf of lustre-discuss at lists.lustre.org <mailto:lustre-discuss at lists.lustre.org>> wrote:

Hello Jon,

I've ran into this issue in the past when trying to install Lustre with ZFS RPMs I built myself. I was able to resolve the problem by including the flag "--with-spec=redhat" when configuring ZFS: 

./configure --with-spec=redhat

see: https://urldefense.com/v3/__https://openzfs.github.io/openzfs-docs/Developer <https://urldefense.com/v3/__https://openzfs.github.io/openzfs-docs/Developer>*20Resources/Custom*20Packages.html?highlight=redhat*kabi-tracking-kmod__;JSUj!!Bt8fGhp8LhKGRg!AAC8AEKPpyuPPt232ylwWXMaAr7OYwltQKBNCZFgwUb_olA-IkbjYgWbvCJpO_ScwrIVo3_0NNwSMDlWc03mn0FfPNA$ 

If you weren't already doing that, does adding that configure flag fix things for you?

- Thomas Bertschinger
________________________________________
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org <mailto:lustre-discuss-bounces at lists.lustre.org>> on behalf of Jon Marshall via lustre-discuss <lustre-discuss at lists.lustre.org <mailto:lustre-discuss at lists.lustre.org>>
Sent: Tuesday, July 11, 2023 9:35 AM
To: lustre-discuss at lists.lustre.org <mailto:lustre-discuss at lists.lustre.org>
Subject: [lustre-discuss] Lustre 2.15.1 server with ZFS nothing provides ksym

Hi,

I'm having a bit of a nightmare trying to build server packages for 2.15.1 - I feel like I've tried quite a few different approaches and am getting stumped, it is most likely that I am missing something incredibly obvious so I'd appreciate any pointers. I'd like to point out that I am by no means an expert in any of this though I have had about 4 years maintaining various Lustre builds - I am hugely grateful for all the work you guys do on Lustre, and I hope I don't come across as anything other than frustrated by my inability to get this version to install!

To start, we are running Rocky 8.6, with the 4.18.0-372.9.1.el8 kernel. Initially, as with previous Lustre builds I've done, I've installed the Whamcloud provided el8_lustre kernel, along with headers, then installed the Mellanox OFED stack from their repos, making sure to use the version that appears to be being used for 2.15.1 builds (in this case 5.6-https://urldefense.com/v3/__http://2.0.9.0__;!!Bt8fGhp8LhKGRg!AAC8AEKPpyuPPt232ylwWXMaAr7OYwltQKBNCZFgwUb_olA-IkbjYgWbvCJpO_ScwrIVo3_0NNwSMDlWc03muNGm7L0$ <https://urldefense.com/v3/__http://2.0.9.0__;!!Bt8fGhp8LhKGRg!AAC8AEKPpyuPPt232ylwWXMaAr7OYwltQKBNCZFgwUb_olA-IkbjYgWbvCJpO_ScwrIVo3_0NNwSMDlWc03muNGm7L0$> - incidentally I can't find the specific OFEDs used in any compatibility matrices or changelogs, where this information used to be provided, so I've back formulated from the Whamcloud repos).

In the past, I've then simply yum/dnf installed the lustre server packages. With this release however, I immediately run in to "Nothing provides _ksym" errors, all of which appear to be for ZFS symbols. A quick check on the Whamcloud Jira throws up this issue, which is marked closed but references this issue, which says that the issue is fixed but not for 2.15.1, instead for 2.15.2.

I'm intending to build these servers with kickstart and puppet, and I'd much rather use the official repos rather than compile it myself but this is not a hard requirement. A quick check on the Whamcloud repos and it appears that 2.15.2 only supports 8.7, rather than 8.6. This is a bit of a problem as I'd like to keep the same kernel version as the rest of the machines we're running where possible, but again, not a hard requirement. I spun up a new build for 8.7 on the same hardware, updated the Mellanox repos to point to the new correct version and immediately got _ksym errors but now it appears they're for the OFEDs instead.

In the meantime, I've seen an email in the mailing list suggesting that the symbols are in fact provided by the package kmod-zfs, which is not provided by the OpenZFS repos, but that can be built manually, so I thought I'd have another crack at getting 2.15.1 working. I download the tar, built ZFS and installed the resulting rpms, making sure to install devel and debugsource and debuginfo. I attempted to build lustre against this and it all appears to go ok - I get some rpms out! However, installing them results in the exact same ksym errors. The thing is the ksyms appear to be present by name in /proc/kallsyms, just not matching exactly.

The main point I guess is that LU-16059 appears to have been closed erroneously, as on a fresh install the issue is 100% reproducible. I also note that the packages hosted here have timestamps from 2022-08-10 but the issue was created and closed after this. I'm happy to re-open the bug and provide as much detail as necessary but thought I'd check to see if anyone else has experienced this issue or if I am indeed missing something trivial.

Thanks in advance
Jon

Jon Marshall

High Performance Computing Specialist

IT and Scientific Computing Team

Cancer Research UK Cambridge Institute

Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org <mailto:lustre-discuss at lists.lustre.org>
https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Bt8fGhp8LhKGRg!AAC8AEKPpyuPPt232ylwWXMaAr7OYwltQKBNCZFgwUb_olA-IkbjYgWbvCJpO_ScwrIVo3_0NNwSMDlWc03mnaUX25E$ <https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Bt8fGhp8LhKGRg!AAC8AEKPpyuPPt232ylwWXMaAr7OYwltQKBNCZFgwUb_olA-IkbjYgWbvCJpO_ScwrIVo3_0NNwSMDlWc03mnaUX25E$> 