[lustre-discuss] ksym errors on kmod-lustre RPM after 2.10.4 build against MOFED 4.3

Matt Rásó-Barnett matt at rasobarnett.com
Tue Aug 7 15:42:23 PDT 2018


Hello,

I've been trying to build a fresh set of RPMs to reproduce the publicly 
available build of Lustre v2_10_4 against RHEL7.5 
kernel-3.10.0-862.2.3.el7 with MOFED 4.3-3.0.2.1:
https://downloads.whamcloud.com/public/lustre/lustre-2.10.4-ib/MOFED-4.3-3.0.2.1/

To do this I've been closely following the instructions at 
http://wiki.lustre.org/Compiling_Lustre

The build process itself is successful, however I am running into an 
error when I try to install the lustre RPMs, and specifically get the 
following:

[root at build ~]# yum --enablerepo=lustre-server install lustre 
kmod-lustre kmod-lustre-osd-ldiskfs lustre-osd-ldiskfs-mount
Loaded plugins: product-id, search-disabled-repos, subscription-manager
...
Resolving Dependencies
--> Running transaction check
---> Package kmod-lustre.x86_64 0:2.10.4-1.el7 will be installed
--> Processing Dependency: ksym(ib_get_dma_mr) = 0x9add141e for package: kmod-lustre-2.10.4-1.el7.x86_64
--> Processing Dependency: ksym(backport_dependency_symbol) = 0xb43a926b for package: kmod-lustre-2.10.4-1.el7.x86_64
--> Processing Dependency: ksym(__rdma_create_id) = 0x4f20f8d2 for package: kmod-lustre-2.10.4-1.el7.x86_64
--> Processing Dependency: ksym(__rdma_accept) = 0x2962f069 for package: kmod-lustre-2.10.4-1.el7.x86_64
---> Package kmod-lustre-osd-ldiskfs.x86_64 0:2.10.4-1.el7 will be installed
---> Package lustre.x86_64 0:2.10.4-1.el7 will be installed
---> Package lustre-osd-ldiskfs-mount.x86_64 0:2.10.4-1.el7 will be installed
--> Running transaction check
---> Package kmod-lustre.x86_64 0:2.10.4-1.el7 will be installed
--> Processing Dependency: ksym(ib_get_dma_mr) = 0x9add141e for package: kmod-lustre-2.10.4-1.el7.x86_64
--> Processing Dependency: ksym(__rdma_create_id) = 0x4f20f8d2 for package: kmod-lustre-2.10.4-1.el7.x86_64
--> Processing Dependency: ksym(__rdma_accept) = 0x2962f069 for package: kmod-lustre-2.10.4-1.el7.x86_64
---> Package kmod-mlnx-ofa_kernel.x86_64 0:4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5 will be installed
--> Finished Dependency Resolution
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)
           Requires: ksym(__rdma_create_id) = 0x4f20f8d2
           Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)
               ksym(__rdma_create_id) = 0x9b63e879
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)
           Requires: ksym(ib_get_dma_mr) = 0x9add141e
           Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)
               ksym(ib_get_dma_mr) = 0x87f4f63d
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)
           Requires: ksym(__rdma_accept) = 0x2962f069
           Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)
               ksym(__rdma_accept) = 0x530ea333
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)
           Requires: ksym(ib_get_dma_mr) = 0x9add141e
           Available: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)
               ksym(ib_get_dma_mr) = 0x87f4f63d
**********************************************************************
yum can be configured to try to resolve such errors by temporarily enabling
disabled repos and searching for missing dependencies.
To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf
**********************************************************************

Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)
           Requires: ksym(__rdma_create_id) = 0x4f20f8d2
           Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)
               ksym(__rdma_create_id) = 0x9b63e879
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)
           Requires: ksym(ib_get_dma_mr) = 0x9add141e
           Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)
               ksym(ib_get_dma_mr) = 0x87f4f63d
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)
           Requires: ksym(__rdma_accept) = 0x2962f069
           Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)
               ksym(__rdma_accept) = 0x530ea333
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)
           Requires: ksym(ib_get_dma_mr) = 0x9add141e
           Available: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)
               ksym(ib_get_dma_mr) = 0x87f4f63d
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

My build process has roughly been:

* Download lustre-src and checkout tag v2_10_4
* Build patched kernel RPMs against kernel-3.10.0-862.2.3.el7 as 
described in lustre wiki:

  rpmbuild -ba --with firmware --with baseonly --without kabichk 
  --without perf --define "buildid _lustre" --target x86_64 
  SPECS/kernel.spec

* Install patched kernel RPMs and reboot into this kernel (also remove 
all other kernels on the system)
* Download MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64.tgz
* Add kernel support for the above patched kernel with:

  ./mlnx_add_kernel_support.sh --make-tgz -m 
  /root/MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64

* Take produced RPMs from the above and install them with:

  yum --enablerepo=mlnx-ofed install 
  mlnx-ofed-all-3.10.0-862.2.3.el7_lustre.x86_64

* Then I build Lustre RPMs with:

  ./configure --enable-server 
  --with-linux=/usr/src/kernels/3.10.0-862.2.3.el7_lustre.x86_64 
  --with-o2ib=/usr/src/ofa_kernel/default

  make rpms

However when trying to install the RPMs produced I get the ksym errors 
as above.

One other thing - I've installed the lustre rpms outside of yum with:

  rpm -ivh --nodeps lustre-2.10.4-1.el7.x86_64.rpm 
  kmod-lustre-2.10.4-1.el7.x86_64.rpm 
  kmod-lustre-osd-ldiskfs-2.10.4-1.el7.x86_64.rpm 
  lustre-osd-ldiskfs-mount-2.10.4-1.el7.x86_64.rpm

which gets me past the ksym error above. With these installed I'm able 
to load the lnet/ko2iblnd/lustre modules successfully (with no errors in 
dmesg), and I'm able to setup the infiniband LNET NID successfully, so 
it feels like the build has been successful but perhaps I've made an 
error somewhere that affects the generated RPM dependencies perhaps? Or 
there is an error to do with kABI compatibility that I don't understand?

Does anyone have any advice on whether the above build steps, 
particularly when building with MOFED, look like how this is normally 
done? Any ideas on what I'm doing wrong here?

Many thanks,
Matt

-- 
Matt Rásó-Barnett
Research Computing Platforms
University Information Services
High Performance Computing Service
University of Cambridge


More information about the lustre-discuss mailing list