[lustre-discuss] MDT quota problem / MDS crash 2.5.3

[Thomas Roth]

Hi all,

we are running Lustre 2.5.3 on our servers. OSTs are on ZFS, MDS is on ldiskfs.

After a MDS crash and e2fsck 1.42.9.wc1 on the partition, the MDS mounts but causes high-frequency log 
entries

Jul 12 16:06:38 lxmds12 kernel: VFS: find_free_dqentry(): Data block full but it shouldn't.
Jul 12 16:06:38 lxmds12 kernel: VFS: Error -5 occurred while creating quota.

interspersed with

Jul 12 16:06:38 lxmds12 kernel: LustreError: 13159:0:(qsd_handler.c:1155:qsd_op_adjust()) nyx-MDT0000: 
fail to locate lqe for id:6763, type:0
Jul 12 16:06:38 lxmds12 kernel: LustreError: 13159:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 4973 
previous similar messages

or

Jul 12 15:59:26 lxmds12 kernel: LustreError: 13414:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed 
to read disk usage, rc:-3 qsd:nyx-MDT0000 qtype:usr id:7408 enforced:0 granted:0 pending:0 waiting:0 
req:0 usage:0 qunit:0 qtune:0 edquot:0
Jul 12 15:59:26 lxmds12 kernel: LustreError: 13414:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 
5166 previous similar messages


According to our experience from the last few days, this will eventually bring all Lustre operations 
to a halt.


Both the web and the e2fsck-messages ([QUOTA WARNING] Usage inconsistent for ID 7989:actual 
(278528000, 738675) != expected (222507008, 531071)) hint towards quota issues.

Therefore, we have 'switched off' quota by "lctl conf_param fsname.quota.ost|mdt=u|g|ug|none", 
restarted, umounted and 'switched on' quota again, restarted, unmounted.

-> The VSF-Errors still appear.

Is there anything else we could do?
Mount the MDT as ldiskfs and do nasty things on the disk?
Is there any command that recalculates / rewrites the quota files on the MDT?



(As long as Lustre is still accessible, 'lfs quota' gives results for both users and groups, but at 
least the file count is entirely wrong (all of my own Lustre files amount to exactle 0 files).

And the update of the usage numbers does not work either - I managed to copy a 1GB-file  and still had 
the same kbytes used...)


Regards,
Thomas

-- 
--------------------------------------------------------------------
Thomas Roth
Department: Informationstechnologie
Location: SB3 1.250
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1
64291 Darmstadt
www.gsi.de

Gesellschaft mit beschränkter Haftung
Sitz der Gesellschaft: Darmstadt
Handelsregister: Amtsgericht Darmstadt, HRB 1528

Geschäftsführung: Ursula Weyrich
Professor Dr. Karlheinz Langanke
Jörg Blaurock

Vorsitzende des Aufsichtsrates: St Dr. Georg Schütte
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt