<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Yea, I think not reusing the index ends up being the sanest solution anyhow, as client state of the old OSTs that are brought back online doesn't always get updated AFAICT. It seems like maximum index is quite high (either 4M or 64K — I can't tell if the flag
used at 64K will play havoc with things once you reach that number). Also, I realized I can remove the log file via lctl llog_remove, so that's not a blocker regardless.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thanks guys!</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
ellis</div>
<div id="appendonsend"></div>
<div><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>From:</b> Stephane Thiell <sthiell@stanford.edu><br>
<b>Sent:</b> Wednesday, November 5, 2025 3:35 PM<br>
<b>To:</b> Ellis Wilson <elliswilson@microsoft.com><br>
<b>Cc:</b> Andreas Dilger <adilger@ddn.com>; lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> [EXTERNAL] Re: [lustre-discuss] Safest means to delete a previously-used OST index?
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>Hello, </div>
<div><br>
</div>
<div>There is indeed no reason for the old OST config to be kept around on the MGS. It’s something I originally forgot when submitting the del_ost patch, sorry about that.</div>
<div><br>
</div>
<div>As Andreas said, we have some long lived production Lustre filesystems (almost 10 years in production) with COTS hardware that we lifecycle when they reach ~5 years (mostly for warranty reasons).</div>
<div><br>
</div>
<div>We are never reusing the OST indexes for the following reasons:</div>
<div><br>
</div>
<div>- We remove/add OSTs several times a year, but their configuration evolves: we have either fewer or more of them depending on the new hardware. Thus, reusing OST indexes is not a good fit. By always appending new OSTs, we don’t have to worry about that.
It’s like having a sliding window of active OST indexes. This has become a habit for us and tools like `lfs` handle that well.</div>
<div><br>
</div>
<div>- We want to be able to lifecycle OSTs while all clients remain mounted. If we wanted to reuse OST indexes, we would need something like what Andreas described in his comment on LU-16475 (from April 2024). That would allow already-mounted clients to reuse
OST indexes, which doesn’t work today.</div>
<div><br>
</div>
<div>Best,</div>
<div><br>
</div>
<div>Stéphane</div>
<div><br>
</div>
<blockquote>
<div>On Nov 2, 2025, at 10:51 PM, Andreas Dilger <adilger@ddn.com> wrote:</div>
<div><br>
</div>
<div>Ellis,</div>
<div>OST removal is something that happens relatively rarely in most filesystems, though the filesystem at Stanford is possibly one of the longer-lived Lustre filesystems in use. The process for that filesystem is to delete old OSTs and not re-use the index.</div>
<div><br>
</div>
<div>I'm not aware of any reason that the old configuration file is kept around after the del_ost command is run, except that it has never been an issue in the past. It would seem relatively straight forward to imlpement this additional step as part of del_ost.</div>
<div><br>
</div>
<div>There is an existing ioctl(OBD_IOC_LLOG_REMOVE) that can be used to delete a specific llog, so it should be possible to add this into the del_ost command to clean up the old config log.</div>
<div><br>
</div>
<blockquote>
<div>On Oct 31, 2025, at 13:13, Ellis Wilson via lustre-discuss <lustre-discuss@lists.lustre.org> wrote:</div>
<div><br>
</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
Hi all,</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
With lctl del_ost the process of deleting an OST that you partially or fully added, but don't want any longer is nearly there. However, we have need for a more complete solution, such that a subsequent addition of an OST at the same index works without the
need to issue --replace with the mkfs command.</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
I've found that if I remount the MGT mount in a non-lustre mode (e.g., simply mount /dev/<dev> /mnt/mgt), I can locate the OST in question under CONFIGS/<name>. By deleting that file, the OST is accepted in the next time around with a simple non-replacement
mkfs invocation. However, this strikes me as very sketchy.</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
Two questions:</div>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-position: initial; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":false,"orderedStyleType":1}" start="1">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<div role="presentation">Why keep the configuration around for a node that is no longer in any of the llogs (post lctl del_ost)? Does it serve any purpose? Would it be better for that del_ost command to also remove that file?</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<div role="presentation">If I do this and then remount the MGT "normally" and re-add an OST with the same index, are there issues I'm going to hit (ignoring in-memory stale context on the client-side alluded to in LU-16475)?</div>
</li></ol>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
This is discussed thoroughly in <a class="OWAAutoLink" id="OWA8053a42a-6c52-c148-2737-7e6431325286" data-auth="NotApplicable" originalsrc="https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-16475__;!!G92We9drHetJ8EofZw!fyg2GZk6-KRgQiIr4RqTpS1qoQdjZwRrsOJ7y9bIo_NF_hwHCfJ-0RhjYz-IjNqYtn5q9trReBm8d4R8$" href="https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-16475__;!!G92We9drHetJ8EofZw!fyg2GZk6-KRgQiIr4RqTpS1qoQdjZwRrsOJ7y9bIo_NF_hwHCfJ-0RhjYz-IjNqYtn5q9trReBm8d4R8$">
https://jira.whamcloud.com/browse/LU-16475</a> (which of course I found after my exploration), but the above two questions aren't completely addressed.</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
Thanks for any advice in advance!</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
ellis</div>
<div><span style="font-family: Helvetica; font-size: 14px;">_______________________________________________</span><br>
<span style="font-family: Helvetica; font-size: 14px;">lustre-discuss mailing list</span><br>
<span style="font-family: Helvetica; font-size: 14px;"><a class="OWAAutoLink" id="OWA3dd55a4a-9e71-697a-60fd-0a5e8905905b" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a></span><br>
<span style="font-family: Helvetica; font-size: 14px;"><a class="OWAAutoLink" id="OWA640ea585-fcc8-b77f-06e5-4ec30fb69678" data-auth="NotApplicable" originalsrc="https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!G92We9drHetJ8EofZw!fyg2GZk6-KRgQiIr4RqTpS1qoQdjZwRrsOJ7y9bIo_NF_hwHCfJ-0RhjYz-IjNqYtn5q9trReGYi7agD$" href="https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!G92We9drHetJ8EofZw!fyg2GZk6-KRgQiIr4RqTpS1qoQdjZwRrsOJ7y9bIo_NF_hwHCfJ-0RhjYz-IjNqYtn5q9trReGYi7agD$">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span></div>
</blockquote>
<div><br>
</div>
<div style="direction: ltr; text-indent: 0px;">Cheers, Andreas</div>
<div style="direction: ltr; text-indent: 0px;">—</div>
<div style="direction: ltr; text-indent: 0px;">Andreas Dilger</div>
<div style="direction: ltr; text-indent: 0px;">Lustre Principal Architect</div>
<div style="direction: ltr; text-indent: 0px;">Whamcloud/DDN</div>
<div style="direction: ltr;"><br>
</div>
<div><br>
<br>
</div>
<div><br>
</div>
</blockquote>
<div><br>
</div>
</body>
</html>