<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
lctl del_ost was added in the upcoming Lustre 2.16 to remove a specific OST while the filesystem is online. What it does is actually launching the llog_cancel commands for the specified OST, hence with reduced risk of user errors (like what happened here).
We have used it several times in production. lctl del_ost is user-space and it is easy to backport it to 2.15 or even 2.12 if you are familiar with how to build Lustre. Do it yourself or ask your favorite storage vendor if it’s not already done. :)
<div class="">In any case, do not panic, as Andreas said, you can always recover from this mistake by doing a full writeconf as documented (which will regenerate all config files).
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">-- </div>
<div class="">Stéphane Thiell</div>
<div class="">Stanford University</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Feb 2, 2023, at 12:57 AM, Andreas Dilger via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">You
should follow the documented process, that's why it is documented. All targets need to be unmounted to make it work properly.</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
<blockquote type="cite" class="">
<div class="">On Feb 2, 2023, at 01:08, BALVERS Martin <<a href="mailto:Martin.BALVERS@danone.com" class="">Martin.BALVERS@danone.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Hi Andreas,<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""> </o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Thank you for answering.<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""> </o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Can I just run the ‘tunefs.lustre --writeconf /dev/ost_device’ on the affected server (serving OST0002 in my case) on a live filesystem? So unmount OST0002, issue writeconf command and mount OST0002 again?<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Or should I follow the procedure as described in<span class="Apple-converted-space"> </span><a href="https://doc.lustre.org/lustre_manual.xhtml#lustremaint.regenerateConfigLogs" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">https://doc.lustre.org/lustre_manual.xhtml#lustremaint.regenerateConfigLogs</a>,
and take the whole filesystem offline and run the writeconf command on all servers?<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""> </o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
To be clear, my situation is this.<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
I had a server with OST0003 that needed to be removed. That worked, but in the process I deleted the add_uuid and attach indexes for OST0002. OST0002 is the one I need to keep.<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""> </o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Regards,<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Martin<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""> </o:p></div>
<div class="">
<div class="" style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;">
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<b class="">From:</b><span class="Apple-converted-space"> </span>Andreas Dilger <<a href="mailto:adilger@whamcloud.com" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">adilger@whamcloud.com</a>><span class="Apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, February 1, 2023 18:16<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>BALVERS Martin <<a href="mailto:Martin.BALVERS@danone.com" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">Martin.BALVERS@danone.com</a>><br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">lustre-discuss@lists.lustre.org</a><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [lustre-discuss] Mistake while removing an OST<o:p class=""></o:p></div>
</div>
</div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""> </o:p></div>
<table class="MsoNormalTable" border="0" cellpadding="0">
<tbody class="">
<tr class="">
<td width="640" valign="top" class="" style="width: 480pt; background-color: rgb(255, 251, 224); padding: 0.75pt;">
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-align: center;">
<span class="" style="font-size: 10pt;">** Caution - this is an external email **</span><o:p class=""></o:p></div>
</td>
</tr>
</tbody>
</table>
<div class="">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
You should just be able to run the "writeconf" process to regenerate the config logs. The removed OST will not re-register with the MGS, but all of the other servers will, so it should be fine. <o:p class=""></o:p></p>
<div class="">
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Cheers, Andreas<o:p class=""></o:p></div>
</div>
<div class="">
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<br class="">
<br class="">
<o:p class=""></o:p></div>
<blockquote class="" style="margin-top: 5pt; margin-bottom: 5pt;">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
On Feb 1, 2023, at 03:48, BALVERS Martin via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">lustre-discuss@lists.lustre.org</a>> wrote:<o:p class=""></o:p></p>
</blockquote>
</div>
<blockquote class="" style="margin-top: 5pt; margin-bottom: 5pt;">
<div class="">
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Hi,<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
I have a defective OSS with a single OST that I was tying to remove from the lustre filesystem completely (2.15.1). I was following<a href="https://urldefense.com/v3/__https:/doc.lustre.org/lustre_manual.xhtml*lustremaint.remove_ost__;Iw!!OUGTln_Lrg!RiLzALkCuFehXiToUr1EEaIJH6J8AnCRNE6js19QvVd68PYBhvNQV9-Teh1p6lltu5zpR_tjJEwAvRuZ5wWrTA$" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">https://doc.lustre.org/lustre_manual.xhtml#lustremaint.remove_ost</a><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
I had drained the OST and was now using the commands with llog_cancel to remove the config. This is where it went wrong. I first deleted attach, setup, add_osc indexes for the ‘client’ and then needed to also delete those for MDT0000 and MDT0001, but I accidentally
removed two more indexes from the ‘client’.<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Now I have incomplete client llogs for one OST, I am missing the add_uuid and attach lines for OST0002.<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class="">[root@mds ~]# lctl --device MGS llog_print lustre-client</span><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class="">- { index: 34, event: add_uuid, nid:<span class="Apple-converted-space"> </span><a href="mailto:192.168.2.3@tcp(0x20000c0a80203)" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">192.168.2.3@tcp(0x20000c0a80203)</a>,
node:<span class="Apple-converted-space"> </span><a href="mailto:192.168.2.3@tcp" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">192.168.2.3@tcp</a><span class="Apple-converted-space"> </span>}</span><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class="">- { index: 35, event: attach, device: lustre-OST0001-osc, type: osc, UUID: lustre-clilov_UUID }</span><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class="">- { index: 36, event: setup, device: lustre-OST0001-osc, UUID: lustre-OST0001_UUID, node:<span class="Apple-converted-space"> </span><a href="mailto:192.168.2.3@tcp" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">192.168.2.3@tcp</a><span class="Apple-converted-space"> </span>}</span><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class="">- { index: 37, event: add_osc, device: lustre-clilov, ost: lustre-OST0001_UUID, index: 1, gen: 1 }</span><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class=""> </span><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class="">- { index: 42, event: setup, device: lustre-OST0002-osc, UUID: lustre-OST0002_UUID, node:<span class="Apple-converted-space"> </span><a href="mailto:192.168.2.4@tcp" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">192.168.2.4@tcp</a><span class="Apple-converted-space"> </span>}</span><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class="">- { index: 43, event: add_osc, device: lustre-clilov, ost: lustre-OST0002_UUID, index: 2, gen: 1 }</span><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="FR" class=""> </span><o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Is there a way to recover from this ?<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
I hope someone can help.<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Regards,<o:p class=""></o:p></div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Martin Balvers<o:p class=""></o:p></div>
<div class="">
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées
et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et
des attachements qu'il contient sont strictement interdits.<span class="Apple-converted-space"> </span><br class="">
<br class="">
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented
are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.<o:p class=""></o:p></div>
</div>
<div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">lustre-discuss@lists.lustre.org</a><br class="">
<a href="https://urldefense.com/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!OUGTln_Lrg!RiLzALkCuFehXiToUr1EEaIJH6J8AnCRNE6js19QvVd68PYBhvNQV9-Teh1p6lltu5zpR_tjJEwAvRuLSxlvXw$" class="" style="color: rgb(5, 99, 193); text-decoration: underline;">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div dir="auto" class="" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" class="" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" class="" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" class="" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" class="" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" class="" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div class="">Cheers, Andreas</div>
<div class="">--</div>
<div class="">Andreas Dilger</div>
<div class="">Lustre Principal Architect</div>
<div class="">Whamcloud</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">lustre-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="mailto:lustre-discuss@lists.lustre.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">lustre-discuss@lists.lustre.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</body>
</html>