[Lustre-discuss] how do I deactivate a very wonky OST

[Thomas Roth]

Brian,

the command to deactivate we are using is
> lctl set_param osc.FS_Name-ID-osc*.active=0
as in lctl set_param osc.lustre-OST0122-osc*.active=0

(lctl --device has failed me a number of times, probably got the wrong device-number there.)

If your killer OST was once part of you Lustre, to "unregister" it for good would ask for extreme
measures (total shutdown, total writeconf everywhere, restart) - did you do that? If not, you should
be able to start all your servers, omitting the bad OST, the 'lctl set_param' it on the MDT and all
clients. Then Lustre should be usable, except for the file remnants of the objects on the missing OST.

At this point I'd run
> lfs find  --obd FS-Name-#{ost}_UUID /mountpoint
writing the output to some files, which are later used to delete/unlink all these remnants.


Cheers,
Thomas

On 01/15/2015 12:11 AM, Andrus, Brian Contractor wrote:
> Thanks Sean,
> 
> Right now neither help me as I had to bring the entire system up from scratch and NOT mount the bad OST.
> So, now OST5 is not listed anywhere. The only knows it is missing.
> Doing 'lctl dl' only show the OSTs that have been brought up.
> If I try to bring it up, it registers, the MDS becomes aware, the OSS kernel panics and the MDS starts making everyone wait for it to come back.
> 
> Part of 'lfs df':
> OST0005             : Resource temporarily unavailable
> 
> Hassle is I cannot really do an 'lfs find' for the files on the bad OST because the OST is not registered... stuck in a loop here...
> 
> If I could find a way to tag it as offline even though the MDS doesn't see it yet, that may help.
> 
> 
> Brian Andrus
> ITACS/Research Computing
> Naval Postgraduate School
> Monterey, California
> voice: 831-656-6238
> 
> 
> 
> 
> From: Sean Brisbane [mailto:s.brisbane1 at physics.ox.ac.uk]
> Sent: Wednesday, January 14, 2015 3:04 PM
> To: Andrus, Brian Contractor; lustre-discuss at lists.lustre.org
> Subject: RE: how do I deactivate a very wonky OST
> 
> This caught me out in a recent upgrade:
> 
> cat /proc/fs/lustre/lov/{yourmdt}/target_obd
> 
> rather than
> 
> "lctl dl"
> 
> Shows the state of the OST.
> 
> Cheers,
> Sean
> ________________________________
> From: lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org> [lustre-discuss-bounces at lists.lustre.org] on behalf of Andrus, Brian Contractor [bdandrus at nps.edu]
> Sent: 13 January 2015 17:28
> To: lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
> Subject: [Lustre-discuss] how do I deactivate a very wonky OST
> All,
> 
> We are still trying to move forward getting our filesystem at least partially up with a failed OST.
> 
> Currently the OST will kernel panic any device that mounts it. That seems to be a constant.
> 
> So, the plan is to bring the system up without that OST and find what data will be lost.
> Now, I am trying to deactivate the OST on the MGS, but it seems to have no effect.
> Running lctl --device 14 deactivate does not change anything. The OST still shows 'UP'
> 
> Is there a way to force lustre to deactivate an OST altogether when it is showing 'UP' and the OST is not going to be happily mounted?
> 
> I can mount the filesystem, but many actions hang (ls, df, etc).
> 
> Brian Andrus
> ITACS/Research Computing
> Naval Postgraduate School
> Monterey, California
> voice: 831-656-6238
> 
> 
> 
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 


-- 
--------------------------------------------------------------------
Thomas Roth
Department: Informationstechnologie
Location: SB3 1.262
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1
64291 Darmstadt
www.gsi.de

Gesellschaft mit beschränkter Haftung
Sitz der Gesellschaft: Darmstadt
Handelsregister: Amtsgericht Darmstadt, HRB 1528

Geschäftsführung: Professor Dr. Dr. h.c. Horst Stöcker,
Dr.-Ing. Jürgen Henschel

Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt