<div dir="ltr"><div dir="ltr">I've managed to solve this after checking a few nodes in the cluster and discovered this particular node must have had a partial update resulting in a mismatch between the kernel version (locked at base release) and some of the kernel support files which appeared to be a slightly later release causing the DKMS to not generate the required files.</div><div dir="ltr"><br></div><div dir="ltr">Normally I disable kernel updates in YUM so everything is at the same release version and just update packages until I'm ready for a major update cycle.<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>bad node:</div><div><br></div><div># yum list installed | grep kernel<br>abrt-addon-kerneloops.x86_64 2.1.11-60.el7.centos @anaconda<br>kernel.x86_64 3.10.0-1160.el7 @anaconda<br>kernel-debug-devel.x86_64 3.10.0-1160.15.2.el7 @updates<br>kernel-devel.x86_64 3.10.0-1160.15.2.el7 @updates<br>kernel-headers.x86_64 3.10.0-1160.15.2.el7 @updates<br>kernel-tools.x86_64 3.10.0-1160.15.2.el7 @updates<br>kernel-tools-libs.x86_64 3.10.0-1160.15.2.el7 @updates<br>#<br></div><div><br></div><div>Working node:</div><div># yum list installed | grep kernel<br>abrt-addon-kerneloops.x86_64 2.1.11-60.el7.centos @anaconda<br>kernel.x86_64 3.10.0-1160.el7 @anaconda<br>kernel-debug-devel.x86_64 3.10.0-1160.31.1.el7 @updates<br>kernel-devel.x86_64 3.10.0-1160.el7 @/kernel-devel-3.10.0-1160.el7.x86_64<br>kernel-headers.x86_64 3.10.0-1160.el7 @anaconda<br>kernel-tools.x86_64 3.10.0-1160.el7 @anaconda<br>kernel-tools-libs.x86_64 3.10.0-1160.el7 @anaconda<br>#<br></div><div><br></div><div>After I removed the extraneous release packages and the lustre packages, I then updated the kernel and re-installed the kernel-headers and kernel-devel code then installed the (minimal) lustre client:</div><div><br></div><div># yum list installed|grep lustre<br>kmod-lustre-client.x86_64 2.12.7-1.el7 @/kmod-lustre-client-2.12.7-1.el7.x86_64<br>lustre-client.x86_64 2.12.7-1.el7 @/lustre-client-2.12.7-1.el7.x86_64<br>lustre-client-dkms.noarch 2.12.7-1.el7 @/lustre-client-dkms-2.12.7-1.el7.noarch<br>#<br></div><div><br></div><div>And all good, every mounts and works first go as expected :)</div><div><br></div><div><br></div><div><br></div><div>Sid Young</div><div><div>Translational Research Institute</div><div>Brisbane</div></div></div></div></div></div></div></div></div></div></div><br></div><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">---------- Forwarded message ----------</span><br></div></div></div>From: Sid Young <<a href="mailto:sid.young@gmail.com" target="_blank">sid.young@gmail.com</a>><br>To: lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>Cc: <br>Bcc: <br>Date: Mon, 8 Nov 2021 11:15:59 +1000<br>Subject: [lustre-discuss] upgrade 2.12.6 to 2.12.7 - no lnet after reboot?<br><div dir="ltr"><div>I was running 2.12.6 on a HP DL385 running standard Centos 7.9 (3.10.0-1160.el7.x86_64) for around 6 months and decided to plan and start an upgrade cycle to 2.12.7, so I downloaded and installed the 2.12.7 centos release from whamcloud using the 7.9.2009 release RPMS</div><div><br></div><div># cat /etc/centos-release</div>CentOS Linux release 7.9.2009 (Core)<br><div><br></div><div>I have tried on the a node and I now have the following error after I rebooted:</div><div><br></div># modprobe -v lnet<br>modprobe: FATAL: Module lnet not found.<br><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>I suspect its not built against the kernel as there are 3 releases showing and no errors during the yum install process:</div><div><br></div><div># ls -la
/usr/lib/modules<br>drwxr-xr-x. 3 root root 4096 Mar 18 2021 3.10.0-1160.2.1.el7.x86_64<br>drwxr-xr-x 3 root root 4096 Nov 8 10:32 3.10.0-1160.25.1.el7.x86_64<br>drwxr-xr-x. 7 root root 4096 Nov 8 11:02 3.10.0-1160.el7.x86_64<br>#<br></div><div><br></div><div>Anyone upgraded this way? Any obvious gottas I've missed?</div><div><br></div><div>Sid Young</div><div>Translational Research Institute</div><div>Brisbane</div><div><br></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div></div>