[lustre-discuss] Safest means to delete a previously-used OST index?

[Ellis Wilson]

Yea, I think not reusing the index ends up being the sanest solution anyhow, as client state of the old OSTs that are brought back online doesn't always get updated AFAICT.  It seems like maximum index is quite high (either 4M or 64K — I can't tell if the flag used at 64K will play havoc with things once you reach that number).  Also, I realized I can remove the log file via lctl llog_remove, so that's not a blocker regardless.

Thanks guys!

ellis


________________________________
From: Stephane Thiell <sthiell at stanford.edu>
Sent: Wednesday, November 5, 2025 3:35 PM
To: Ellis Wilson <elliswilson at microsoft.com>
Cc: Andreas Dilger <adilger at ddn.com>; lustre-discuss <lustre-discuss at lists.lustre.org>
Subject: [EXTERNAL] Re: [lustre-discuss] Safest means to delete a previously-used OST index?

Hello,

There is indeed no reason for the old OST config to be kept around on the MGS. It’s something I originally forgot when submitting the del_ost patch, sorry about that.

As Andreas said, we have some long lived production Lustre filesystems (almost 10 years in production) with COTS hardware that we lifecycle when they reach ~5 years (mostly for warranty reasons).

We are never reusing the OST indexes for the following reasons:

- We remove/add OSTs several times a year, but their configuration evolves: we have either fewer or more of them depending on the new hardware. Thus, reusing OST indexes is not a good fit. By always appending new OSTs, we don’t have to worry about that. It’s like having a sliding window of active OST indexes. This has become a habit for us and tools like `lfs` handle that well.

- We want to be able to lifecycle OSTs while all clients remain mounted. If we wanted to reuse OST indexes, we would need something like what Andreas described in his comment on LU-16475 (from April 2024). That would allow already-mounted clients to reuse OST indexes, which doesn’t work today.

Best,

Stéphane

On Nov 2, 2025, at 10:51 PM, Andreas Dilger <adilger at ddn.com> wrote:

Ellis,
OST removal is something that happens relatively rarely in most filesystems, though the filesystem at Stanford is possibly one of the longer-lived Lustre filesystems in use.  The process for that filesystem is to delete old OSTs and not re-use the index.

I'm not aware of any reason that the old configuration file is kept around after the del_ost command is run, except that it has never been an issue in the past.  It would seem relatively straight forward to imlpement this additional step as part of del_ost.

There is an existing ioctl(OBD_IOC_LLOG_REMOVE) that can be used to delete a specific llog, so it should be possible to add this into the del_ost command to clean up the old config log.

On Oct 31, 2025, at 13:13, Ellis Wilson via lustre-discuss <lustre-discuss at lists.lustre.org> wrote:

Hi all,

With lctl del_ost the process of deleting an OST that you partially or fully added, but don't want any longer is nearly there.  However, we have need for a more complete solution, such that a subsequent addition of an OST at the same index works without the need to issue --replace with the mkfs command.

I've found that if I remount the MGT mount in a non-lustre mode (e.g., simply mount /dev/<dev> /mnt/mgt), I can locate the OST in question under CONFIGS/<name>.  By deleting that file, the OST is accepted in the next time around with a simple non-replacement mkfs invocation.  However, this strikes me as very sketchy.

Two questions:

  1.
Why keep the configuration around for a node that is no longer in any of the llogs (post lctl del_ost)?  Does it serve any purpose?  Would it be better for that del_ost command to also remove that file?
  2.
If I do this and then remount the MGT "normally" and re-add an OST with the same index, are there issues I'm going to hit (ignoring in-memory stale context on the client-side alluded to in LU-16475)?

This is discussed thoroughly in https://jira.whamcloud.com/browse/LU-16475<https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-16475__;!!G92We9drHetJ8EofZw!fyg2GZk6-KRgQiIr4RqTpS1qoQdjZwRrsOJ7y9bIo_NF_hwHCfJ-0RhjYz-IjNqYtn5q9trReBm8d4R8$> (which of course I found after my exploration), but the above two questions aren't completely addressed.

Thanks for any advice in advance!

ellis
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!G92We9drHetJ8EofZw!fyg2GZk6-KRgQiIr4RqTpS1qoQdjZwRrsOJ7y9bIo_NF_hwHCfJ-0RhjYz-IjNqYtn5q9trReGYi7agD$>

Cheers, Andreas
—
Andreas Dilger
Lustre Principal Architect
Whamcloud/DDN





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20251124/35aaa7f1/attachment-0001.htm>