[lustre-discuss] How to activate an OST on a client ?

[Lixin Liu]

We had the same problem after we upgraded Lustre servers from 2.12.8 to 2.15.3.
Clients were running 2.15.3 on CentOS 7. Random OST dropped out frequently on
busy login nodes (almost daily), but less so on compute nodes. “lctl” command
cannot active OSTs and reboot we the only way to clear the problem.

In June, we upgraded all client OS to AlmaLinux 9.3 and Lustre version to 2.15.4 on
both servers and clients (missed 2.15.5 release by about 2 weeks). After the upgrade,
we no longer have this problem.

In our case, I wonder this was OmniPath related. Servers on AlamLinux 8 was using
in kernel driver, but CentOS 7 clients are using driver from Intel/Cornelis release.
Alma 9 clients are now also using in kernel driver.

Cheers,

Lixin.

From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of Cameron Harr via lustre-discuss <lustre-discuss at lists.lustre.org>
Reply-To: Cameron Harr <harr1 at llnl.gov>
Date: Wednesday, August 28, 2024 at 8:19 AM
To: "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] How to activate an OST on a client ?


There's also an "lctl --device <dev> activate" that I've used in the past though I don't know what conditions need to be for it to work.
On 8/27/24 07:46, Andreas Dilger via lustre-discuss wrote:
Hi Jan,
There is "lctl --device XXXX recover" that will trigger a reconnect to the named OST device (per "lctl dl" output), but not sure if that will help.


Cheers, Andreas


On Aug 22, 2024, at 06:36, Haarst, Jan van via lustre-discuss <lustre-discuss at lists.lustre.org><mailto:lustre-discuss at lists.lustre.org> wrote:
Hi,

Probably the wording of the subject doesn’t actually cover the issue, what we see is this :
We have a client behind a router (linking tcp to Omnipath) that shows an inactive OST (all on 2.15.5).
Other clients that go through the router do not have this issue.

One client had the same issue, although it showed a different OST as inactive.
After a reboot, all was well again on that machine.

The clients can lctl ping the OSSs.

So although we have a workaround (reboot the client), it would be nice to:

  1.  Fix the issue without a reboot
  2.  Fix the underlying issue.

It might be unrelated, but we also see another routing issue every now and then:
The router stops routing request toward a certain OSS, and this can be fixed by deleting the peer_nid of the OSS from the router.

I am probably missing informative logs, but I’m more than happy to try to generate them, if somebody has a pointer to how.

We are a bit stumped right now.

With kind regards,

--
Jan van Haarst
HPC Administrator
For Anunna/HPC questions, please use https://support.wur.nl<https://urldefense.us/v3/__https:/support.wur.nl__;!!G2kpM7uM-TzIFchu!1YPSOGUFPvipdg8HUxDkmcB7rvfUxuSATnKZq-9LFTP16TrMxtlrPe7m3ccX4BmKFoLsVnaKiIL3u4pxK2GT6mMjyuAoAg$> (with HPC as service)
Aanwezig: maandag, dinsdag, donderdag & vrijdag
Facilitair Bedrijf, onderdeel van Wageningen University & Research
Afdeling Informatie Technologie
Postbus 59, 6700 AB, Wageningen
Gebouw 116, Akkermaalsbos 12, 6700 WB, Wageningen
http://www.wur.nl/nl/Disclaimer.htm<https://urldefense.us/v3/__http:/www.wur.nl/nl/Disclaimer.htm__;!!G2kpM7uM-TzIFchu!1YPSOGUFPvipdg8HUxDkmcB7rvfUxuSATnKZq-9LFTP16TrMxtlrPe7m3ccX4BmKFoLsVnaKiIL3u4pxK2GT6mP2LXgG1Q$>

_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<https://urldefense.us/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!G2kpM7uM-TzIFchu!1YPSOGUFPvipdg8HUxDkmcB7rvfUxuSATnKZq-9LFTP16TrMxtlrPe7m3ccX4BmKFoLsVnaKiIL3u4pxK2GT6mNJQIy33g$>



_______________________________________________

lustre-discuss mailing list

lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>

https://urldefense.us/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!G2kpM7uM-TzIFchu!1YPSOGUFPvipdg8HUxDkmcB7rvfUxuSATnKZq-9LFTP16TrMxtlrPe7m3ccX4BmKFoLsVnaKiIL3u4pxK2GT6mNJQIy33g$<https://urldefense.us/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!G2kpM7uM-TzIFchu!1YPSOGUFPvipdg8HUxDkmcB7rvfUxuSATnKZq-9LFTP16TrMxtlrPe7m3ccX4BmKFoLsVnaKiIL3u4pxK2GT6mNJQIy33g$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20240828/dd243266/attachment-0001.htm>