[Lustre-discuss] Lots of "No ctxt" after OST crush

[Andreas Dilger]

The missing logdile problem is easily fixed - delete the CATALOGS file  
on the MDT and restart. There is a bug just opened to handle this  
better, but it isn't fixed yet.

Cheers, Andreas

On 2010-04-26, at 7:00, Thomas Roth <t.roth at gsi.de> wrote:

> Hi all,
>
> one of our OSTs crushed - actually we ran into Bug 17052
> (https://bugzilla.lustre.org/show_bug.cgi?id=17052). The OST fscks
> without errors, mouting and aborting recovery also works.
>
> However, the MDT doesn't accept it anymore (I've attached the entire  
> log
> line of the event below):
> LustreError:... (llog_lvfs.c: ...:llog_lvfs_create()) error looking up
> logfile ...: rc -2
>
> It would seem the Logfile was lost somewhere in the process (rc -2).
> The MDT then deactivates this OST.
>
> The clients can see the files on this OST, files can be read and  
> deleted
> - as expected.
>
> So my idea was to leave the OST as it is now and try and move the file
> off it to the other OSTs, eventually reformatting it sometime.
>
> However, the MDT log now has a lot of
> Apr 26 14:45:34 lxmds3 kernel: LustreError:
> 3804:0:(llog_obd.c:226:llog_add()) No ctxt
> Apr 26 14:45:34 lxmds3 kernel: LustreError:
> 3804:0:(llog_obd.c:226:llog_add()) Skipped 4058 previous similar  
> messages
>
> These "No ctxt" appear immedeately after the MDT refuses the said OST,
> so I assume a connection.
> My question: Do these messages mean any further trouble? Could  
> something
> be building up and finally blwo up the MDT/the file system? Should I  
> try
> to do something with this Log-less OST instead, and would would that  
> be?
>
> This is Lustre 1.6.7.2 running under Debian Etch 64bit, Kernel 2.6.22.
>
> Regards,
> Thomas
>
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --- 
> --------------------------------------------------------------------
>
> MDT-Log of OST-Remount-Attempt:
>
> Apr 26 13:58:00 lxmds3 kernel: Lustre: gsilust-OST00b3-osc: Connection
> restored to service gsilust-OST00b3 using nid 10.12.119.138 at tcp.
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13532:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile
> 0x2dd861c:0xe95a1032: rc -2
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13532:0:(llog_cat.c:172:llog_cat_id2handle()) error opening log id
> 0x2dd861c:e95a1032: rc -2
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13532:0:(llog_obd.c:279:cat_cancel_cb()) Cannot find handle for log
> 0x2dd861c
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13531:0:(llog_obd.c:350:llog_obd_origin_setup()) llog_process with
> cat_cancel_cb failed: -2
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13531:0:(llog_obd.c:194:llog_setup()) obd gsilust-OST00b3-osc ctxt 2
> lop_setup=ffffffff88354370 failed -2
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13531:0:(osc_request.c:3724:osc_llog_init()) failed  
> LLOG_MDS_OST_ORIG_CTXT
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13531:0:(osc_request.c:3740:osc_llog_init()) osc 'gsilust-OST00b3-osc'
> tgt 'gsilust-MDT0000' cnt 1 catid ffff8103aa285ce0 rc=-2
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13531:0:(osc_request.c:3742:osc_llog_init()) logid 0x50fb811:0xf129dc6
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13531:0:(lov_log.c:243:lov_llog_init()) error osc_llog_init idx 179  
> osc
> 'gsilust-OST00b3-osc' tgt 'gsilust-MDT0000' (rc=-2)
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13531:0:(mds_log.c:219:mds_llog_init()) lov_llog_init err -2
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13531:0:(llog_obd.c:439:llog_cat_initialize()) rc: -2
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13531:0:(mds_lov.c:918:__mds_lov_synchronize()) gsilust-OST00b3_UUID
> failed at update_mds: -2
> Apr 26 13:58:00 lxmds3 kernel: LustreError:
> 13531:0:(mds_lov.c:960:__mds_lov_synchronize()) gsilust-OST00b3_UUID
> sync failed -2, deactivating
>
>
> -- 
> --------------------------------------------------------------------
> Thomas Roth
> Department: Informationstechnologie
> Location: SB3 1.262
> Phone: +49-6159-71 1453  Fax: +49-6159-71 2986
>
> GSI Helmholtzzentrum für Schwerionenforschung GmbH
> Planckstraße 1
> D-64291 Darmstadt
> www.gsi.de
>
> Gesellschaft mit beschränkter Haftung
> Sitz der Gesellschaft: Darmstadt
> Handelsregister: Amtsgericht Darmstadt, HRB 1528
>
> Geschäftsführer: Professor Dr. Horst Stöcker (wissenschaftlich)
> Geschäftsführer: Christiane Neumann (kaufmännisch)
>
> Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph,
> Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss