[lustre-discuss] Mistake while removing an OST

[Andreas Dilger]

You should follow the documented process, that's why it is documented.  All targets need to be unmounted to make it work properly.

On Feb 2, 2023, at 01:08, BALVERS Martin <Martin.BALVERS at danone.com<mailto:Martin.BALVERS at danone.com>> wrote:

Hi Andreas,

Thank you for answering.

Can I just run the ‘tunefs.lustre --writeconf /dev/ost_device’ on the affected server (serving OST0002 in my case) on a live filesystem? So unmount OST0002, issue writeconf command and mount OST0002 again?
Or should I follow the procedure as described in https://doc.lustre.org/lustre_manual.xhtml#lustremaint.regenerateConfigLogs, and take the whole filesystem offline and run the writeconf command on all servers?

To be clear, my situation is this.
I had a server with OST0003 that needed to be removed. That worked, but in the process I deleted the add_uuid and attach indexes for OST0002. OST0002 is the one I need to keep.

Regards,
Martin

From: Andreas Dilger <adilger at whamcloud.com<mailto:adilger at whamcloud.com>>
Sent: Wednesday, February 1, 2023 18:16
To: BALVERS Martin <Martin.BALVERS at danone.com<mailto:Martin.BALVERS at danone.com>>
Cc: lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] Mistake while removing an OST

** Caution - this is an external email **
You should just be able to run the "writeconf" process to regenerate the config logs. The removed OST will not re-register with the MGS, but all of the other servers will, so it should be fine.
Cheers, Andreas


On Feb 1, 2023, at 03:48, BALVERS Martin via lustre-discuss <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>> wrote:

Hi,

I have a defective OSS with a single OST that I was tying to remove from the lustre filesystem completely (2.15.1). I was followinghttps://doc.lustre.org/lustre_manual.xhtml#lustremaint.remove_ost<https://urldefense.com/v3/__https:/doc.lustre.org/lustre_manual.xhtml*lustremaint.remove_ost__;Iw!!OUGTln_Lrg!RiLzALkCuFehXiToUr1EEaIJH6J8AnCRNE6js19QvVd68PYBhvNQV9-Teh1p6lltu5zpR_tjJEwAvRuZ5wWrTA$>
I had drained the OST and was now using the commands with llog_cancel to remove the config. This is where it went wrong. I first deleted attach, setup, add_osc indexes for the ‘client’ and then needed to also delete those for MDT0000 and MDT0001, but I accidentally removed two more indexes from the ‘client’.
Now I have incomplete client llogs for one OST, I am missing the add_uuid and attach lines for OST0002.

[root at mds ~]# lctl --device MGS llog_print lustre-client
- { index: 34, event: add_uuid, nid: 192.168.2.3 at tcp(0x20000c0a80203)<mailto:192.168.2.3 at tcp(0x20000c0a80203)>, node: 192.168.2.3 at tcp<mailto:192.168.2.3 at tcp> }
- { index: 35, event: attach, device: lustre-OST0001-osc, type: osc, UUID: lustre-clilov_UUID }
- { index: 36, event: setup, device: lustre-OST0001-osc, UUID: lustre-OST0001_UUID, node: 192.168.2.3 at tcp<mailto:192.168.2.3 at tcp> }
- { index: 37, event: add_osc, device: lustre-clilov, ost: lustre-OST0001_UUID, index: 1, gen: 1 }

- { index: 42, event: setup, device: lustre-OST0002-osc, UUID: lustre-OST0002_UUID, node: 192.168.2.4 at tcp<mailto:192.168.2.4 at tcp> }
- { index: 43, event: add_osc, device: lustre-clilov, ost: lustre-OST0002_UUID, index: 2, gen: 1 }

Is there a way to recover from this ?

I hope someone can help.

Regards,
Martin Balvers
Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits.

This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<https://urldefense.com/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!OUGTln_Lrg!RiLzALkCuFehXiToUr1EEaIJH6J8AnCRNE6js19QvVd68PYBhvNQV9-Teh1p6lltu5zpR_tjJEwAvRuLSxlvXw$>

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20230202/c87154da/attachment-0001.htm>