[lustre-discuss] MDT quota problem / MDS crash 2.5.3

Guido Laubender laubender at zib.de
Tue Jul 12 13:33:10 PDT 2016


Hallo Thomas,

(wir kennen uns noch aus meiner Frankfurter Zeit am CSC, vielleicht 
erinnerst Du Dich...)

Bei uns waren vor kurzem die Inode-Quoten nicht korrekt; durch 
Deaktivieren und anschließendes Aktivieren der Quoten-Unterstützung 
(mittels 'tune2fs -O ^quota' und anschließendem 'tunefs.lustre --quota') 
auf dem MDT konnten wir es wieder reparieren.

Vielleicht hilft das bei Euch auch...

Grüße
Guido

--
Guido Laubender      phone: +49 30 84185 214
ZIB                  fax:   +49 30 84185 311
Takustr. 7
D-14195 Berlin

On Tue, 12 Jul 2016, Thomas Roth wrote:

> Hi all,
>
> we are running Lustre 2.5.3 on our servers. OSTs are on ZFS, MDS is on 
> ldiskfs.
>
> After a MDS crash and e2fsck 1.42.9.wc1 on the partition, the MDS mounts but 
> causes high-frequency log 
> entries
>
> Jul 12 16:06:38 lxmds12 kernel: VFS: find_free_dqentry(): Data block full but 
> it shouldn't.
> Jul 12 16:06:38 lxmds12 kernel: VFS: Error -5 occurred while creating quota.
>
> interspersed with
>
> Jul 12 16:06:38 lxmds12 kernel: LustreError: 
> 13159:0:(qsd_handler.c:1155:qsd_op_adjust()) nyx-MDT0000: 
> fail to locate lqe for id:6763, type:0
> Jul 12 16:06:38 lxmds12 kernel: LustreError: 
> 13159:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 4973 
> previous similar messages
>
> or
>
> Jul 12 15:59:26 lxmds12 kernel: LustreError: 
> 13414:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed 
> to read disk usage, rc:-3 qsd:nyx-MDT0000 qtype:usr id:7408 enforced:0 
> granted:0 pending:0 waiting:0 
> req:0 usage:0 qunit:0 qtune:0 edquot:0
> Jul 12 15:59:26 lxmds12 kernel: LustreError: 
> 13414:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 
> 5166 previous similar messages
>
>
> According to our experience from the last few days, this will eventually 
> bring all Lustre operations 
> to a halt.
>
>
> Both the web and the e2fsck-messages ([QUOTA WARNING] Usage inconsistent for 
> ID 7989:actual 
> (278528000, 738675) != expected (222507008, 531071)) hint towards quota 
> issues.
>
> Therefore, we have 'switched off' quota by "lctl conf_param 
> fsname.quota.ost|mdt=u|g|ug|none", 
> restarted, umounted and 'switched on' quota again, restarted, unmounted.
>
> -> The VSF-Errors still appear.
>
> Is there anything else we could do?
> Mount the MDT as ldiskfs and do nasty things on the disk?
> Is there any command that recalculates / rewrites the quota files on the MDT?
>
>
>
> (As long as Lustre is still accessible, 'lfs quota' gives results for both 
> users and groups, but at 
> least the file count is entirely wrong (all of my own Lustre files amount to 
> exactle 0 files).
>
> And the update of the usage numbers does not work either - I managed to copy 
> a 1GB-file  and still had 
> the same kbytes used...)
>
>
> Regards,
> Thomas
>
> -- 
> --------------------------------------------------------------------
> Thomas Roth
> Department: Informationstechnologie
> Location: SB3 1.250
> Phone: +49-6159-71 1453  Fax: +49-6159-71 2986
>
> GSI Helmholtzzentrum für Schwerionenforschung GmbH
> Planckstraße 1
> 64291 Darmstadt
> www.gsi.de
>
> Gesellschaft mit beschränkter Haftung
> Sitz der Gesellschaft: Darmstadt
> Handelsregister: Amtsgericht Darmstadt, HRB 1528
>
> Geschäftsführung: Ursula Weyrich
> Professor Dr. Karlheinz Langanke
> Jörg Blaurock
>
> Vorsitzende des Aufsichtsrates: St Dr. Georg Schütte
> Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>


More information about the lustre-discuss mailing list