[lustre-discuss] MDT quota problem / MDS crash 2.5.3

[Thomas Roth]

Hi Guido,

thanks for the tip, that was successful, with the exact same commands,

   tune2fs -O ^quota /dev/mdt     (took about ~3min)
   tunefs.lustre --quota /dev/mdt (took about ~30min with ~200M used inodes)


A subsequent
   lctl lfsck_start -M nyx-MDT0000
ran for less than 45 min and seems to have cleaned up the mdt mess.

Cheers,
Thomas


On 07/12/2016 11:18 PM, Guido Laubender wrote:
>
> English version (I'm sorry for my previous mail in German - but should have been a personal mail to Thomas only :( ):
>
> We were recently able to fix wrong Lustre inode quotas by disabling and re-enabling quota support on the MDT by 'tune2fs -O ^quota /dev/mdt' and
> 'tunefs.lustre --quota /dev/mdt'.
>
> Maybe it helps here as well.
>
>
> On Tue, 12 Jul 2016, Guido Laubender wrote:
>
>> Bei uns waren vor kurzem die Inode-Quoten nicht korrekt; durch Deaktivieren und anschließendes Aktivieren der Quoten-Unterstützung (mittels 'tune2fs
>> -O ^quota' und anschließendem 'tunefs.lustre --quota') auf dem MDT konnten wir es wieder reparieren.
>>
>> Vielleicht hilft das bei Euch auch...
>>
>> On Tue, 12 Jul 2016, Thomas Roth wrote:
>>
>>> Hi all,
>>>
>>> we are running Lustre 2.5.3 on our servers. OSTs are on ZFS, MDS is on ldiskfs.
>>>
>>> After a MDS crash and e2fsck 1.42.9.wc1 on the partition, the MDS mounts but causes high-frequency log entries
>>>
>>> Jul 12 16:06:38 lxmds12 kernel: VFS: find_free_dqentry(): Data block full but it shouldn't.
>>> Jul 12 16:06:38 lxmds12 kernel: VFS: Error -5 occurred while creating quota.
>>>
>>> interspersed with
>>>
>>> Jul 12 16:06:38 lxmds12 kernel: LustreError: 13159:0:(qsd_handler.c:1155:qsd_op_adjust()) nyx-MDT0000: fail to locate lqe for id:6763, type:0
>>> Jul 12 16:06:38 lxmds12 kernel: LustreError: 13159:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 4973 previous similar messages
>>>
>>> or
>>>
>>> Jul 12 15:59:26 lxmds12 kernel: LustreError: 13414:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:nyx-MDT0000
>>> qtype:usr id:7408 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
>>> Jul 12 15:59:26 lxmds12 kernel: LustreError: 13414:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 5166 previous similar messages
>>>
>>>
>>> According to our experience from the last few days, this will eventually bring all Lustre operations to a halt.
>>>
>>>
>>> Both the web and the e2fsck-messages ([QUOTA WARNING] Usage inconsistent for ID 7989:actual (278528000, 738675) != expected (222507008, 531071))
>>> hint towards quota issues.
>>>
>>> Therefore, we have 'switched off' quota by "lctl conf_param fsname.quota.ost|mdt=u|g|ug|none", restarted, umounted and 'switched on' quota again,
>>> restarted, unmounted.
>>>
>>> -> The VSF-Errors still appear.
>>>
>>> Is there anything else we could do?
>>> Mount the MDT as ldiskfs and do nasty things on the disk?
>>> Is there any command that recalculates / rewrites the quota files on the MDT?
>>>
>>>
>>>
>>> (As long as Lustre is still accessible, 'lfs quota' gives results for both users and groups, but at least the file count is entirely wrong (all of
>>> my own Lustre files amount to exactle 0 files).
>>>
>>> And the update of the usage numbers does not work either - I managed to copy a 1GB-file  and still had the same kbytes used...)
>>>
>>>
>>> Regards,
>>> Thomas
>>>
>>> --
>>> --------------------------------------------------------------------
>>> Thomas Roth
>>> Department: Informationstechnologie
>>> Location: SB3 1.250
>>> Phone: +49-6159-71 1453  Fax: +49-6159-71 2986
>>>
>>> GSI Helmholtzzentrum für Schwerionenforschung GmbH
>>> Planckstraße 1
>>> 64291 Darmstadt
>>> www.gsi.de
>>>
>>> Gesellschaft mit beschränkter Haftung
>>> Sitz der Gesellschaft: Darmstadt
>>> Handelsregister: Amtsgericht Darmstadt, HRB 1528
>>>
>>> Geschäftsführung: Ursula Weyrich
>>> Professor Dr. Karlheinz Langanke
>>> Jörg Blaurock
>>>
>>> Vorsitzende des Aufsichtsrates: St Dr. Georg Schütte
>>> Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt
>>>
>>> _______________________________________________
>>> lustre-discuss mailing list
>>> lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>

-- 
--------------------------------------------------------------------
Thomas Roth
Department: HPC
Location: SB3 1.262
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1
64291 Darmstadt
www.gsi.de

Gesellschaft mit beschränkter Haftung
Sitz der Gesellschaft: Darmstadt
Handelsregister: Amtsgericht Darmstadt, HRB 1528

Geschäftsführung: Professor Dr. Karlheinz Langanke
Ursula Weyrich
Jörg Blaurock

Vorsitzender des Aufsichtsrates: St Dr. Georg Schütte
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt