<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.xelementtoproof
{mso-style-name:x_elementtoproof;}
span.contentpasted0
{mso-style-name:contentpasted0;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Jon<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Have you tried building 2.15.3 against Rocky 8.6 locally? While it is true that Whamcloud focuses testing on the last RHEL/Rocky minor release, many sites still use the same Lustre software with down-rev OS versions….<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Peter<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Jon Marshall via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Reply-To: </b>Jon Marshall <Jon.Marshall@cruk.cam.ac.uk><br>
<b>Date: </b>Tuesday, July 11, 2023 at 8:38 AM<br>
<b>To: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[lustre-discuss] Lustre 2.15.1 server with ZFS nothing provides ksym<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="xelementtoproof"><span style="font-size:12.0pt;color:black;background:white">Hi,</span></span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">I'm having a bit of a nightmare trying to build server packages for 2.15.1 - I feel like I've tried quite a few different approaches and am getting stumped, it is most
likely that I am missing something incredibly obvious so I'd appreciate any pointers. I'd like to point out that I am by no means an expert in any of this though I have had about 4 years maintaining various Lustre builds - I am hugely grateful for all the
work you guys do on Lustre, and I hope I don't come across as anything other than frustrated by my inability to get this version to install!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">To start, we are running Rocky 8.6, with the 4.18.0-372.9.1.el8 kernel. Initially, as with previous Lustre builds I've done, I've installed the Whamcloud provided el8_lustre
kernel, along with headers, then installed the Mellanox OFED stack from their repos, making sure to use the version that appears to be being used for 2.15.1 builds (in this case 5.6-2.0.9.0 - incidentally I can't find the specific OFEDs used in any compatibility
matrices or changelogs, where this information used to be provided, so I've back formulated from the Whamcloud repos). <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">In the past, I've then simply yum/dnf installed the lustre server packages. With this release however, I immediately run in to "Nothing provides _ksym" errors, all of which
appear to be for ZFS symbols. A quick check on the Whamcloud Jira throws up<span class="contentpasted0"> </span><a href="https://jira.whamcloud.com/browse/LU-16109" target="_blank" title="https://jira.whamcloud.com/browse/LU-16109">this</a><span class="contentpasted0"> </span>issue,
which is marked closed but references<span class="contentpasted0"> </span><a href="https://jira.whamcloud.com/browse/LU-16059" target="_blank" title="https://jira.whamcloud.com/browse/LU-16059">this</a><span class="contentpasted0"> </span>issue, which says
that the issue is fixed but not for 2.15.1, instead for 2.15.2.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">I'm intending to build these servers with kickstart and puppet, and I'd much rather use the official repos rather than compile it myself but this is not a hard requirement.
A quick check on the Whamcloud repos and it appears that 2.15.2 only supports 8.7, rather than 8.6. This is a bit of a problem as I'd like to keep the same kernel version as the rest of the machines we're running where possible, but again, not a hard requirement.
I spun up a new build for 8.7 on the same hardware, updated the Mellanox repos to point to the new correct version and immediately got _ksym errors but now it appears they're for the OFEDs instead.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">In the meantime, I've seen an email in the mailing list suggesting that the symbols are in fact provided by the package kmod-zfs, which is not provided by the OpenZFS repos,
but that can be built manually, so I thought I'd have another crack at getting 2.15.1 working. I download the tar, built ZFS and installed the resulting rpms, making sure to install devel and debugsource and debuginfo. I attempted to build lustre against this
and it all appears to go ok - I get some rpms out! However, installing them results in the exact same ksym errors. The thing is the ksyms appear to be present by name in /proc/kallsyms, just not matching exactly.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">The main point I guess is that LU-16059 appears to have been closed erroneously, as on a fresh install the issue is 100% reproducible. I also note that the packages hosted<span class="contentpasted0"> </span><a href="https://downloads.whamcloud.com/public/lustre/lustre-2.15.1/el8.6/server/RPMS/x86_64/" target="_blank" title="https://downloads.whamcloud.com/public/lustre/lustre-2.15.1/el8.6/server/RPMS/x86_64/">here</a> have
timestamps from 2022-08-10 but the issue was created and closed after this. I'm happy to re-open the bug and provide as much detail as necessary but thought I'd check to see if anyone else has experienced this issue or if I am indeed missing something trivial.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">Thanks in advance<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span class="xelementtoproof"><span style="font-size:12.0pt;color:black;background:white">Jon</span></span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div id="Signature">
<div>
<div>
<p><span style="color:#595959">Jon Marshall</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:10.0pt;color:#595959">High Performance Computing Specialist</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:10.0pt;color:#595959"> </span><span style="color:black"><o:p></o:p></span></p>
<p><b><span style="color:#595959">IT and Scientific Computing Team</span></b><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:10.0pt;color:#595959"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:10.0pt;color:#595959">Cancer Research UK Cambridge Institute</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:10.0pt;color:#595959">Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="color:black"><a href="http://www.cruk.cam.ac.uk/" target="_blank"><span style="font-size:10.0pt;color:#0070C0">Web</span></a></span><span style="font-size:10.0pt;color:#0070C0"> |
</span><span style="color:black"><a href="http://www.facebook.com/cancerresearchuk" target="_blank"><span style="font-size:10.0pt;color:#0070C0">Facebook</span></a></span><span style="font-size:10.0pt;color:#0070C0"> |
</span><span style="color:black"><a href="http://twitter.com/CR_UK" target="_blank"><span style="font-size:10.0pt;color:#0070C0">Twitter</span></a><o:p></o:p></span></p>
<p><span style="font-size:10.0pt;color:#0070C0"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span style="color:black"><a href="http://www.cruk.cam.ac.uk/" target="_blank"><span style="color:black;text-decoration:none"><span style="font-size:10.5pt;font-family:"Times New Roman",serif;color:blue"><img border="0" width="32" height="32" style="width:.3333in;height:.3333in" id="_x0000_i1025" src="cid:image001.jpg@01D4CCEA.783A5790" alt="Description: CRI Logo"></span></span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>