<div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
2. Tools to check a lustre (Sid Young)<br> 4. Re: Tools to check a lustre (Dennis Nelson)<br>
<br><br></blockquote><div>My key issue is why /home locks solid when you try to use it but /lustre is OK . The backend is ZFS used to manage the disks presented from the HP D8000 JBOD</div><div>I'm at a loss after 6 months of 100% operation why this is suddenly occurring. If I do repeated "dd" tasks on lustre it works fine, start one on /home and it locks solid.</div><div><br></div><div>I have started a ZFS scrub on two of the zfs pools. at 47T each it will take most of today to resolve, but that should rule out the actual storage (which is showing "NORMAL/ONLINE" and no errors.</div><div><br></div><div>I'm seeing a lot of these in /var/log/messages</div><div>kernel: LustreError: 6578:0:(events.c:200:client_bulk_callback()) event type 1, status -5, desc ffff89cdf3b9dc00<br></div><div>A google search returned this: <a href="https://wiki.lustre.org/Lustre_Resiliency:_Understanding_Lustre_Message_Loss_and_Tuning_for_Resiliency">https://wiki.lustre.org/Lustre_Resiliency:_Understanding_Lustre_Message_Loss_and_Tuning_for_Resiliency</a></div><div><br></div><div>Could it be a network issue? - the nodes are running the Centos7.9 drivers... the Mellanox one did not seam to make any difference when I originally tried it 6 months ago.</div><div><br></div><div>Any help appreciated :)</div><div><br></div><div>Sid</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br><div dir="ltr"><div><br></div></div>---------- Forwarded message ----------<br>From: Sid Young <<a href="mailto:sid.young@gmail.com" target="_blank">sid.young@gmail.com</a>><br>To: lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>Cc: <br>Bcc: <br>Date: Mon, 11 Oct 2021 16:07:56 +1000<br>Subject: [lustre-discuss] Tools to check a lustre<br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>I'm having trouble diagnosing where the problem lies in my Lustre installation, clients are 2.12.6 and I have a /home and /lustre filesystems using Lustre. </div><div><br></div><div>/home has 4 OSTs and /lustre is made up of 6 OSTs. lfs df shows all OSTs as ACTIVE.</div><div><br></div><div>The /lustre file system appears fine, I can <b>ls </b>into every directory.</div><div><br></div><div>When people log into the login node, it appears to lockup. I have shut down everything and remounted the OSTs and MDTs etc in order with no errors reporting but I'm getting the lockup issue soon after a few people log in.</div><div>The backend network is 100G Ethernet using ConnectX5 cards and the OS is Cento 7.9, everything was installed as RPMs and updates are disabled in yum.conf </div><div><br></div><div>Two questions to start with:</div><div>Is there a command line tool to check each OST individually? </div><div>Apart from /var/log/messages, is there a lustre specific log I can monitor on the login node to see errors when I hit /home... </div><div><br></div><div><br></div><div><br></div><div>Sid Young</div></div></div></div></div></div></div></div></div></div>
<br><br><div lang="FR" style="overflow-wrap: break-word;"><div class="gmail-m_1130481326328403398WordSection1"><div><div><div><div><div><div><div><div><div><div><div><div><div><br>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br><br><br>---------- Forwarded message ----------<br>From: Dennis Nelson <<a href="mailto:dnelson@ddn.com" target="_blank">dnelson@ddn.com</a>><br>To: Sid Young <<a href="mailto:sid.young@gmail.com" target="_blank">sid.young@gmail.com</a>><br><br>Date: Mon, 11 Oct 2021 12:20:25 +0000<br>Subject: Re: [lustre-discuss] Tools to check a lustre<br>
<div dir="auto">
Have you tried lfs check servers on the login node?<br></div></blockquote><div><br></div><div>Yes - one of the first things I did and this is what it always reports:</div><div><br></div><div>]# lfs check servers<br>home-OST0000-osc-ffff89adb7e5e000 active.<br>home-OST0001-osc-ffff89adb7e5e000 active.<br>home-OST0002-osc-ffff89adb7e5e000 active.<br>home-OST0003-osc-ffff89adb7e5e000 active.<br>lustre-OST0000-osc-ffff89cdd14a2000 active.<br>lustre-OST0001-osc-ffff89cdd14a2000 active.<br>lustre-OST0002-osc-ffff89cdd14a2000 active.<br>lustre-OST0003-osc-ffff89cdd14a2000 active.<br>lustre-OST0004-osc-ffff89cdd14a2000 active.<br>lustre-OST0005-osc-ffff89cdd14a2000 active.<br>home-MDT0000-mdc-ffff89adb7e5e000 active.<br>lustre-MDT0000-mdc-ffff89cdd14a2000 active.<br>[root@tri-minihub-01 ~]#<br></div><div><br></div></div></div>