<div dir="ltr"><div>Had the same issue, it was because I still had the kernel modules loaded while building the RPMs.</div><div><br></div><div>Uninstalled kmod-lustre and re-compiled fixed the ksym issues.</div><div><br></div><div>Also, there's a page allocation issue with MoFED 4.3, use MoFED 4.4.</div><div><br></div><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="font-size:12.8px">-- </span><br style="font-size:12.8px"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><strong><font face="arial narrow, sans-serif" size="2">Gin Tan<br></font></strong></div><div><font face="arial narrow, sans-serif" size="2">MASSIVE support and consulting services<br></font></div><div><font face="arial narrow, sans-serif" size="2"><br></font></div><div dir="ltr"><b><font face="arial narrow, sans-serif" size="2">Monash eResearch Centre</font></b></div><div dir="ltr"><font face="arial narrow, sans-serif" size="2">Monash University</font></div><div dir="ltr"><font face="arial narrow, sans-serif" size="2"><br></font></div><div dir="ltr"><font face="arial narrow, sans-serif" size="2">Room G63B, 15 Innovation Walk</font></div><div dir="ltr"><font face="arial narrow, sans-serif" size="2">Clayton Campus</font></div><div dir="ltr"><font face="arial narrow, sans-serif" size="2">Wellington Road</font></div><div dir="ltr"><font face="arial narrow, sans-serif" size="2">Clayton VIC 3800</font></div><div dir="ltr"><font face="arial narrow, sans-serif" size="2">Australia</font></div><div dir="ltr"><font face="arial narrow, sans-serif" size="2"><br></font></div><div dir="ltr"><font face="arial narrow, sans-serif" size="2">T: <font color="#000000">+61 3 9902 0245</font> </font></div><div><font face="arial narrow, sans-serif" size="2">E:<font color="#000000"> <a href="mailto:gin.tan@monash.edu" target="_blank">gin.tan@monash.edu</a></font></font></div><div><font face="arial narrow, sans-serif" size="2">Z: </font><a href="https://monash.zoom.us/my/gintan" target="_blank"><span style="white-space:nowrap;font-size:13px;color:rgb(35,35,51);font-family:"Open Sans",Helvetica,Arial;letter-spacing:normal">https://monash.zoom.us/my/</span><span style="white-space:nowrap;font-size:13px;color:rgb(35,35,51);font-family:"Open Sans",Helvetica,Arial;letter-spacing:normal">gintan</span></a></div><div><a href="http://www.monash.edu.au/eresearch" target="_blank"><font face="arial narrow, sans-serif" size="2">www.monash.edu.au/eresearch</font></a></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, 9 Aug 2018 at 06:21, <<a href="mailto:lustre-discuss-request@lists.lustre.org">lustre-discuss-request@lists.lustre.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send lustre-discuss mailing list submissions to<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-discuss-owner@lists.lustre.org" target="_blank">lustre-discuss-owner@lists.lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of lustre-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. ksym errors on kmod-lustre RPM after 2.10.4 build against<br>
MOFED 4.3 (Matt R?s?-Barnett)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 7 Aug 2018 23:42:23 +0100<br>
From: Matt R?s?-Barnett <<a href="mailto:matt@rasobarnett.com" target="_blank">matt@rasobarnett.com</a>><br>
To: "<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
Subject: [lustre-discuss] ksym errors on kmod-lustre RPM after 2.10.4<br>
build against MOFED 4.3<br>
Message-ID: <20180807224223.rhags26mq2vlp6tt@yoshimo.localdomain><br>
Content-Type: text/plain; charset=iso-8859-1; format=flowed<br>
<br>
Hello,<br>
<br>
I've been trying to build a fresh set of RPMs to reproduce the publicly <br>
available build of Lustre v2_10_4 against RHEL7.5 <br>
kernel-3.10.0-862.2.3.el7 with MOFED 4.3-3.0.2.1:<br>
<a href="https://downloads.whamcloud.com/public/lustre/lustre-2.10.4-ib/MOFED-4.3-3.0.2.1/" rel="noreferrer" target="_blank">https://downloads.whamcloud.com/public/lustre/lustre-2.10.4-ib/MOFED-4.3-3.0.2.1/</a><br>
<br>
To do this I've been closely following the instructions at <br>
<a href="http://wiki.lustre.org/Compiling_Lustre" rel="noreferrer" target="_blank">http://wiki.lustre.org/Compiling_Lustre</a><br>
<br>
The build process itself is successful, however I am running into an <br>
error when I try to install the lustre RPMs, and specifically get the <br>
following:<br>
<br>
[root@build ~]# yum --enablerepo=lustre-server install lustre <br>
kmod-lustre kmod-lustre-osd-ldiskfs lustre-osd-ldiskfs-mount<br>
Loaded plugins: product-id, search-disabled-repos, subscription-manager<br>
...<br>
Resolving Dependencies<br>
--> Running transaction check<br>
---> Package kmod-lustre.x86_64 0:2.10.4-1.el7 will be installed<br>
--> Processing Dependency: ksym(ib_get_dma_mr) = 0x9add141e for package: kmod-lustre-2.10.4-1.el7.x86_64<br>
--> Processing Dependency: ksym(backport_dependency_symbol) = 0xb43a926b for package: kmod-lustre-2.10.4-1.el7.x86_64<br>
--> Processing Dependency: ksym(__rdma_create_id) = 0x4f20f8d2 for package: kmod-lustre-2.10.4-1.el7.x86_64<br>
--> Processing Dependency: ksym(__rdma_accept) = 0x2962f069 for package: kmod-lustre-2.10.4-1.el7.x86_64<br>
---> Package kmod-lustre-osd-ldiskfs.x86_64 0:2.10.4-1.el7 will be installed<br>
---> Package lustre.x86_64 0:2.10.4-1.el7 will be installed<br>
---> Package lustre-osd-ldiskfs-mount.x86_64 0:2.10.4-1.el7 will be installed<br>
--> Running transaction check<br>
---> Package kmod-lustre.x86_64 0:2.10.4-1.el7 will be installed<br>
--> Processing Dependency: ksym(ib_get_dma_mr) = 0x9add141e for package: kmod-lustre-2.10.4-1.el7.x86_64<br>
--> Processing Dependency: ksym(__rdma_create_id) = 0x4f20f8d2 for package: kmod-lustre-2.10.4-1.el7.x86_64<br>
--> Processing Dependency: ksym(__rdma_accept) = 0x2962f069 for package: kmod-lustre-2.10.4-1.el7.x86_64<br>
---> Package kmod-mlnx-ofa_kernel.x86_64 0:4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5 will be installed<br>
--> Finished Dependency Resolution<br>
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)<br>
Requires: ksym(__rdma_create_id) = 0x4f20f8d2<br>
Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)<br>
ksym(__rdma_create_id) = 0x9b63e879<br>
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)<br>
Requires: ksym(ib_get_dma_mr) = 0x9add141e<br>
Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)<br>
ksym(ib_get_dma_mr) = 0x87f4f63d<br>
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)<br>
Requires: ksym(__rdma_accept) = 0x2962f069<br>
Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)<br>
ksym(__rdma_accept) = 0x530ea333<br>
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)<br>
Requires: ksym(ib_get_dma_mr) = 0x9add141e<br>
Available: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)<br>
ksym(ib_get_dma_mr) = 0x87f4f63d<br>
**********************************************************************<br>
yum can be configured to try to resolve such errors by temporarily enabling<br>
disabled repos and searching for missing dependencies.<br>
To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf<br>
**********************************************************************<br>
<br>
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)<br>
Requires: ksym(__rdma_create_id) = 0x4f20f8d2<br>
Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)<br>
ksym(__rdma_create_id) = 0x9b63e879<br>
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)<br>
Requires: ksym(ib_get_dma_mr) = 0x9add141e<br>
Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)<br>
ksym(ib_get_dma_mr) = 0x87f4f63d<br>
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)<br>
Requires: ksym(__rdma_accept) = 0x2962f069<br>
Installing: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)<br>
ksym(__rdma_accept) = 0x530ea333<br>
Error: Package: kmod-lustre-2.10.4-1.el7.x86_64 (lustre-server)<br>
Requires: ksym(ib_get_dma_mr) = 0x9add141e<br>
Available: kmod-mlnx-ofa_kernel-4.3-OFED.4.3.3.0.2.1.gcf60532.rhel7u5.x86_64 (mlnx-ofed)<br>
ksym(ib_get_dma_mr) = 0x87f4f63d<br>
You could try using --skip-broken to work around the problem<br>
You could try running: rpm -Va --nofiles --nodigest<br>
<br>
My build process has roughly been:<br>
<br>
* Download lustre-src and checkout tag v2_10_4<br>
* Build patched kernel RPMs against kernel-3.10.0-862.2.3.el7 as <br>
described in lustre wiki:<br>
<br>
rpmbuild -ba --with firmware --with baseonly --without kabichk <br>
--without perf --define "buildid _lustre" --target x86_64 <br>
SPECS/kernel.spec<br>
<br>
* Install patched kernel RPMs and reboot into this kernel (also remove <br>
all other kernels on the system)<br>
* Download MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64.tgz<br>
* Add kernel support for the above patched kernel with:<br>
<br>
./mlnx_add_kernel_support.sh --make-tgz -m <br>
/root/MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64<br>
<br>
* Take produced RPMs from the above and install them with:<br>
<br>
yum --enablerepo=mlnx-ofed install <br>
mlnx-ofed-all-3.10.0-862.2.3.el7_lustre.x86_64<br>
<br>
* Then I build Lustre RPMs with:<br>
<br>
./configure --enable-server <br>
--with-linux=/usr/src/kernels/3.10.0-862.2.3.el7_lustre.x86_64 <br>
--with-o2ib=/usr/src/ofa_kernel/default<br>
<br>
make rpms<br>
<br>
However when trying to install the RPMs produced I get the ksym errors <br>
as above.<br>
<br>
One other thing - I've installed the lustre rpms outside of yum with:<br>
<br>
rpm -ivh --nodeps lustre-2.10.4-1.el7.x86_64.rpm <br>
kmod-lustre-2.10.4-1.el7.x86_64.rpm <br>
kmod-lustre-osd-ldiskfs-2.10.4-1.el7.x86_64.rpm <br>
lustre-osd-ldiskfs-mount-2.10.4-1.el7.x86_64.rpm<br>
<br>
which gets me past the ksym error above. With these installed I'm able <br>
to load the lnet/ko2iblnd/lustre modules successfully (with no errors in <br>
dmesg), and I'm able to setup the infiniband LNET NID successfully, so <br>
it feels like the build has been successful but perhaps I've made an <br>
error somewhere that affects the generated RPM dependencies perhaps? Or <br>
there is an error to do with kABI compatibility that I don't understand?<br>
<br>
Does anyone have any advice on whether the above build steps, <br>
particularly when building with MOFED, look like how this is normally <br>
done? Any ideas on what I'm doing wrong here?<br>
<br>
Many thanks,<br>
Matt<br>
<br>
-- <br>
Matt R?s?-Barnett<br>
Research Computing Platforms<br>
University Information Services<br>
High Performance Computing Service<br>
University of Cambridge<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
<br>
------------------------------<br>
<br>
End of lustre-discuss Digest, Vol 149, Issue 5<br>
**********************************************<br>
</blockquote></div>