<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Hello,
<div><br>
</div>
<div>There is indeed no reason for the old OST config to be kept around on the MGS. It’s something I originally forgot when submitting the del_ost patch, sorry about that.</div>
<div><br>
</div>
<div>As Andreas said, we have some long lived production Lustre filesystems (almost 10 years in production) with COTS hardware that we lifecycle when they reach ~5 years (mostly for warranty reasons).</div>
<div><br>
</div>
<div>We are never reusing the OST indexes for the following reasons:</div>
<div><br>
</div>
<div>- We remove/add OSTs several times a year, but their configuration evolves: we have either fewer or more of them depending on the new hardware. Thus, reusing OST indexes is not a good fit. By always appending new OSTs, we don’t have to worry about that.
It’s like having a sliding window of active OST indexes. This has become a habit for us and tools like `lfs` handle that well.</div>
<div><br>
</div>
<div>- We want to be able to lifecycle OSTs while all clients remain mounted. If we wanted to reuse OST indexes, we would need something like what Andreas described in his comment on LU-16475 (from April 2024). That would allow already-mounted clients to reuse
OST indexes, which doesn’t work today.</div>
<div><br>
</div>
<div>Best,</div>
<div><br>
</div>
<div>Stéphane</div>
<div>
<div><br>
<blockquote type="cite">
<div>On Nov 2, 2025, at 10:51 PM, Andreas Dilger <adilger@ddn.com> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>Ellis,</div>
<div>OST removal is something that happens relatively rarely in most filesystems, though the filesystem at Stanford is possibly one of the longer-lived Lustre filesystems in use. The process for that filesystem is to delete old OSTs and not re-use the index.</div>
<div><br>
</div>
<div>I'm not aware of any reason that the old configuration file is kept around after the del_ost command is run, except that it has never been an issue in the past. It would seem relatively straight forward to imlpement this additional step as part of del_ost.</div>
<div><br>
</div>
<div>There is an existing ioctl(OBD_IOC_LLOG_REMOVE) that can be used to delete a specific llog, so it should be possible to add this into the del_ost command to clean up the old config log.</div>
<br>
<div>
<blockquote type="cite">
<div>On Oct 31, 2025, at 13:13, Ellis Wilson via lustre-discuss <lustre-discuss@lists.lustre.org> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
Hi all,</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
With lctl del_ost the process of deleting an OST that you partially or fully added, but don't want any longer is nearly there. However, we have need for a more complete solution, such that a subsequent addition of an OST at the same index works without the
need to issue --replace with the mkfs command.</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
I've found that if I remount the MGT mount in a non-lustre mode (e.g., simply mount /dev/<dev> /mnt/mgt), I can locate the OST in question under CONFIGS/<name>. By deleting that file, the OST is accepted in the next time around with a simple non-replacement
mkfs invocation. However, this strikes me as very sketchy.</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
Two questions:</div>
<ol data-editing-info="{"applyListStyleFromLevel":false,"orderedStyleType":1}" start="1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<div role="presentation" class="elementToProof">Why keep the configuration around for a node that is no longer in any of the llogs (post lctl del_ost)? Does it serve any purpose? Would it be better for that del_ost command to also remove that file?</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<div role="presentation" class="elementToProof">If I do this and then remount the MGT "normally" and re-add an OST with the same index, are there issues I'm going to hit (ignoring in-memory stale context on the client-side alluded to in LU-16475)?</div>
</li></ol>
<div style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
This is discussed thoroughly in<span class="Apple-converted-space"> </span><a href="https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-16475__;!!G92We9drHetJ8EofZw!fyg2GZk6-KRgQiIr4RqTpS1qoQdjZwRrsOJ7y9bIo_NF_hwHCfJ-0RhjYz-IjNqYtn5q9trReBm8d4R8$">https://jira.whamcloud.com/browse/LU-16475</a> (which
of course I found after my exploration), but the above two questions aren't completely addressed.</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
Thanks for any advice in advance!</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
ellis</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">lustre-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<a href="mailto:lustre-discuss@lists.lustre.org" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">lustre-discuss@lists.lustre.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<a href="https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!G92We9drHetJ8EofZw!fyg2GZk6-KRgQiIr4RqTpS1qoQdjZwRrsOJ7y9bIo_NF_hwHCfJ-0RhjYz-IjNqYtn5q9trReGYi7agD$" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></div>
</blockquote>
</div>
<br>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>Cheers, Andreas</div>
<div>—</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud/DDN</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>