[lustre-discuss] MDT quota problem / MDS crash 2.5.3

[Guido Laubender]

English version (I'm sorry for my previous mail in German - but should 
have been a personal mail to Thomas only :( ):

We were recently able to fix wrong Lustre inode quotas by disabling and 
re-enabling quota support on the MDT by 'tune2fs -O ^quota /dev/mdt' and 
'tunefs.lustre --quota /dev/mdt'.

Maybe it helps here as well.


On Tue, 12 Jul 2016, Guido Laubender wrote:

> Bei uns waren vor kurzem die Inode-Quoten nicht korrekt; durch Deaktivieren 
> und anschließendes Aktivieren der Quoten-Unterstützung (mittels 'tune2fs -O 
> ^quota' und anschließendem 'tunefs.lustre --quota') auf dem MDT konnten wir 
> es wieder reparieren.
>
> Vielleicht hilft das bei Euch auch...
>
> On Tue, 12 Jul 2016, Thomas Roth wrote:
>
>> Hi all,
>> 
>> we are running Lustre 2.5.3 on our servers. OSTs are on ZFS, MDS is on 
>> ldiskfs.
>> 
>> After a MDS crash and e2fsck 1.42.9.wc1 on the partition, the MDS mounts 
>> but causes high-frequency log entries
>> 
>> Jul 12 16:06:38 lxmds12 kernel: VFS: find_free_dqentry(): Data block full 
>> but it shouldn't.
>> Jul 12 16:06:38 lxmds12 kernel: VFS: Error -5 occurred while creating 
>> quota.
>> 
>> interspersed with
>> 
>> Jul 12 16:06:38 lxmds12 kernel: LustreError: 
>> 13159:0:(qsd_handler.c:1155:qsd_op_adjust()) nyx-MDT0000: fail to locate 
>> lqe for id:6763, type:0
>> Jul 12 16:06:38 lxmds12 kernel: LustreError: 
>> 13159:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 4973 previous similar 
>> messages
>> 
>> or
>> 
>> Jul 12 15:59:26 lxmds12 kernel: LustreError: 
>> 13414:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk 
>> usage, rc:-3 qsd:nyx-MDT0000 qtype:usr id:7408 enforced:0 granted:0 
>> pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
>> Jul 12 15:59:26 lxmds12 kernel: LustreError: 
>> 13414:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 5166 previous similar 
>> messages
>> 
>> 
>> According to our experience from the last few days, this will eventually 
>> bring all Lustre operations to a halt.
>> 
>> 
>> Both the web and the e2fsck-messages ([QUOTA WARNING] Usage inconsistent 
>> for ID 7989:actual (278528000, 738675) != expected (222507008, 531071)) 
>> hint towards quota issues.
>> 
>> Therefore, we have 'switched off' quota by "lctl conf_param 
>> fsname.quota.ost|mdt=u|g|ug|none", restarted, umounted and 'switched on' 
>> quota again, restarted, unmounted.
>> 
>> -> The VSF-Errors still appear.
>> 
>> Is there anything else we could do?
>> Mount the MDT as ldiskfs and do nasty things on the disk?
>> Is there any command that recalculates / rewrites the quota files on the 
>> MDT?
>> 
>> 
>> 
>> (As long as Lustre is still accessible, 'lfs quota' gives results for both 
>> users and groups, but at least the file count is entirely wrong (all of my 
>> own Lustre files amount to exactle 0 files).
>> 
>> And the update of the usage numbers does not work either - I managed to 
>> copy a 1GB-file  and still had the same kbytes used...)
>> 
>> 
>> Regards,
>> Thomas
>> 
>> -- 
>> --------------------------------------------------------------------
>> Thomas Roth
>> Department: Informationstechnologie
>> Location: SB3 1.250
>> Phone: +49-6159-71 1453  Fax: +49-6159-71 2986
>> 
>> GSI Helmholtzzentrum für Schwerionenforschung GmbH
>> Planckstraße 1
>> 64291 Darmstadt
>> www.gsi.de
>> 
>> Gesellschaft mit beschränkter Haftung
>> Sitz der Gesellschaft: Darmstadt
>> Handelsregister: Amtsgericht Darmstadt, HRB 1528
>> 
>> Geschäftsführung: Ursula Weyrich
>> Professor Dr. Karlheinz Langanke
>> Jörg Blaurock
>> 
>> Vorsitzende des Aufsichtsrates: St Dr. Georg Schütte
>> Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt
>> 
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>