[lustre-discuss] lfsck repair quota

Wed Apr 17 22:49:50 PDT 2019

On Apr 16, 2019, at 13:53, Fernando Pérez <fperez at icm.csic.es> wrote:
> 
>> You can try...
>> unmount the filesystem and the run a force e2fsck on all OSTs and MDT.
>> 
>> e2fsck -f -p /dev/ost...
>> 
>> Regargs.
> 
> Thank you Mahmoud.
> 
> The problem is that e2fsck in the MDT runs very very slowly…10 inodes per second for more than 100 million inodes. 
> 
>>> Are there a lot of inodes moved to lost+found by the fsck, which contribute to the occupied quota now?
> 
> Thank you Martin.
> 
> I can’t finish the e2fsck in the MDT.
> 
>> Correct, LFSCK will repair the OST object ownership, but does not *directly* repair quota
>> files. That said, if an object is owned by the wrong owner for some reason, changing the
>> ownership _should_ transfer the quota usage properly to the new owner, so in that regard
>> the quota usage should be indirectly repaired by an LFSCK run.
> 
> 
> Thank you Andreas.
> 
> I can confirm that lfsck layout does not repair the wrong quotes.
> 
> I’m running lfsck namespace in the MDT to try other way.

Note that "namespace" checking only verifes/repairs the MDT consistency.
Running "layout" checking is what verifies the MDT-OST consistency.

That said, it looks like you are running "layout" checking based on the output below.

>> Note the "tune2fs -O ^quota" to repair the quota accounting is a bit of a "cargo cult" behaviour.
>> 
>> Running "e2fsck -fp" is the proper way to repair the quota files, since it not only recalculates the quota accounting using the same code as "tune2fs -O quota" does, but it also ensures that the files themselves are valid in the first place.  They should take about the same time, except in the case your filesystem is corrupted, in which case you'd want e2fsck to repair the filesystem anyway.
> 
> I ran this in the lfsck MDT and lfsck OSTs. After do that the quotes for all users were more corrupted than before (for example for a user with 27 TB  the quota reports 900 MB).
> 
> Then I ran the e2fsck in the lfsck OSTs. I tried to do the same in the lfsck MDT but it ran very very slow….and I canceled it.
> 
> I suppose that the correct way to repair quota is to run the e2fsck in the MDT, but the problem is the system downtime.
> 
> I copy the lfsck stats at the end of this message:
> 
>> lctl get_param -n mdd.lustre-MDT0000.lfsck_layout
>> name: lfsck_layout
>> magic: 0xb17371b9
>> version: 2
>> status: partial
>> flags:
>> param:
>> last_completed_time: 1555393553
>> time_since_last_completed: 50098 seconds
>> latest_start_time: 1555339300
>> time_since_latest_start: 104351 seconds
>> last_checkpoint_time: 1555393553
>> time_since_last_checkpoint: 50098 seconds
>> latest_start_position: 77
>> last_checkpoint_position: 102983584
>> first_failure_position: 21342296
>> success_count: 1
>> repaired_dangling: 2470024
>> repaired_unmatched_pair: 7487028
>> repaired_multiple_referenced: 0
>> repaired_orphan: 0
>> repaired_inconsistent_owner: 18950288

This is a lot of repairs that would affect quota.

>> repaired_others: 0
>> skipped: 0
>> failed_phase1: 1622485
>> failed_phase2: 0
>> checked_phase1: 119083557
>> checked_phase2: 0
>> run_time_phase1: 31777 seconds
>> run_time_phase2: 22423 seconds
>> average_speed_phase1: 3747 items/sec
>> average_speed_phase2: 0 objs/sec
>> real-time_speed_phase1: N/A
>> real-time_speed_phase2: N/A
>> current_position: N/A
> 
> 
>> lctl lfsck_query -t namespace -M lustre-MDT0000
>> namespace_mdts_init: 0
>> namespace_mdts_scanning-phase1: 0
>> namespace_mdts_scanning-phase2: 1
>> namespace_mdts_completed: 0
>> namespace_mdts_failed: 0
>> namespace_mdts_stopped: 0
>> namespace_mdts_paused: 0
>> namespace_mdts_crashed: 0
>> namespace_mdts_partial: 0
>> namespace_mdts_co-failed: 0
>> namespace_mdts_co-stopped: 0
>> namespace_mdts_co-paused: 0
>> namespace_mdts_unknown: 0
>> namespace_osts_init: 0
>> namespace_osts_scanning-phase1: 0
>> namespace_osts_scanning-phase2: 0
>> namespace_osts_completed: 0
>> namespace_osts_failed: 0
>> namespace_osts_stopped: 0
>> namespace_osts_paused: 0
>> namespace_osts_crashed: 0
>> namespace_osts_partial: 0
>> namespace_osts_co-failed: 0
>> namespace_osts_co-stopped: 0
>> namespace_osts_co-paused: 0
>> namespace_osts_unknown: 0
>> namespace_repaired: 4550034
> 
> 
> 
> =============================================
> Fernando Pérez
> Institut de Ciències del Mar (CMIMA-CSIC)
> Departament Oceanografía Física i Tecnològica
> Passeig Marítim de la Barceloneta,37-49
> 08003 Barcelona
> Phone:  (+34) 93 230 96 35
> =============================================
> 
>> El 16 abr 2019, a las 20:43, Martin Hecht <hecht at hlrs.de> escribió:
>> 
>> Are there a lot of inodes moved to lost+found by the fsck, which contribute to the occupied quota now?
>> 
>> ----- Ursprüngliche Mail -----
>> Von: Fernando Pérez <fperez at icm.csic.es>
>> An: lustre-discuss at lists.lustre.org
>> Gesendet: Tue, 16 Apr 2019 16:24:13 +0200 (CEST)
>> Betreff: Re: [lustre-discuss] lfsck repair quota
>> 
>> Thank you Rick.
>> 
>> I followed these steps for the ldiskfs OSTs and MDT, but the quotes for all users is more corrupted than before.
>> 
>> I tried to run e2fsck in ldiskfs OSTs MDT, but the problem was the MDT e2fsck ran very slow ( 10 inodes per second for more than 100 million inodes).
>> 
>> According to the lustre wiki I though that the lfsck could repair corrupted quotes:
>> 
>> http://wiki.lustre.org/Lustre_Quota_Troubleshooting
>> 
>> Regards.
>> 
>> ============================================
>> Fernando Pérez
>> Institut de Ciències del Mar (CSIC)
>> Departament Oceanografía Física i Tecnològica
>> Passeig Marítim de la Barceloneta,37-49
>> 08003 Barcelona
>> Phone:  (+34) 93 230 96 35
>> ============================================
>> 
>>> El 16 abr 2019, a las 15:34, Mohr Jr, Richard Frank (Rick Mohr) <rmohr at utk.edu> escribió:
>>> 
>>> 
>>>> On Apr 15, 2019, at 10:54 AM, Fernando Perez <fperez at icm.csic.es> wrote:
>>>> 
>>>> Could anyone confirm me that the correct way to repair wrong quotes in a ldiskfs mdt is lctl lfsck_start -t layout -A?
>>> 
>>> As far as I know, lfsck doesn’t repair quota info. It only fixes internal consistency within Lustre.
>>> 
>>> Whenever I have had to repair quotas, I just follow the procedure you did (unmount everything, run “tune2fs -O ^quota <dev>”, run “tune2fs -O quota <dev>”, and then remount).  But all my systems used ldiskfs, so I don’t know if the ZFS OSTs introduce any sort of complication.  (Actually, I am not even sure if/how you can regenerate quota info for ZFS.)
>>> 
>>> --
>>> Rick Mohr
>>> Senior HPC System Administrator
>>> National Institute for Computational Sciences
>>> http://www.nics.tennessee.edu
>>> 
>> 
> 

Cheers, Andreas
---
Andreas Dilger
CTO Whamcloud