[lustre-discuss] Corrupted quota file

Dilger, Andreas andreas.dilger at intel.com
Fri Sep 30 08:46:20 PDT 2016


I would recommend to install a more modern e2fsprogs.
e2fsck-1.42.13.wc5 is the current version.

Cheers, Andreas
-- 
Andreas Dilger

Lustre Principal Architect
Intel High Performance Data Division

On 2016/09/30, 08:16, "lustre-discuss on behalf of Ramiro Alba" <lustre-discuss-bounces at lists.lustre.org on behalf of raq at cttc.upc.edu> wrote:

    Hi all,
    
    I am using Lustre 2.5.3 under CentOS 6.5 both servers and clients and
    the lustre file system is using 6 OSTs of 30TB size with 2 OSSs.
    
    Last days I've been having problems with OSS server rebooting without a
    known reason after running fine for several hours (betwen 3 to 24), and
    I also noticed that quotas were not showing the correct usage.
    
    At the last OSS server reboot issue, I deciced to run e2fsck both on MDT 
    and
    OSTs, and those bellow were the errors at command output:
    
    ------------------
    MDT:
    ------------------
    e2fsck -f /dev/mapper/vglustre-MDT
    
    e2fsck 1.42.9.wc1 (24-Feb-2014)
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (1045 >= 
    10) in user quota file. Quota file is probably corrupted.
    Please run e2fsck ( to fix it.
    [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (16322 >= 
    10) in user quota file. Quota file is probably corrupted.
    Please run e2fsck ( to fix it.
    [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (2940928 
     >= 10) in user quota file. Quota file is probably corrupted.
    Please run e2fsck ( to fix it.
    jffstg-MDT0000: 1218855/47185920 files (0.2% non-contiguous), 
    6191259/23592960 blocks
    
    ------------------
    OSSs:
    ------------------
    [root at jffoss1 system]# e2fsck -fp /dev/mapper/ost0; e2fsck -fp 
    /dev/mapper/ost1; e2fsck -fp /dev/mapper/ost2
    
    MMP interval is 10 seconds and total wait time is 42 seconds. Please 
    wait...
    jffstg-OST0000: recovering journal
    [QUOTA WARNING] Usage inconsistent for ID 0:actual (1766737936384, 348 
    != expected (0, 0)
    [QUOTA WARNING] Usage inconsistent for ID 1042:actual (507613184, 3613) 
    != expected (40960, 1)
    [QUOTA WARNING] Usage inconsistent for ID 1031:actual (1041013137408, 
    527 != expected (42952433664, 119)
    [QUOTA WARNING] Usage inconsistent for ID 1304:actual (562630971392, 
    5127) != expected (0, 0)
    [QUOTA WARNING] Usage inconsistent for ID 1338:actual (1634377928704, 
    7847) != expected (2098733056, 12)
    [QUOTA WARNING] Usage inconsistent for ID 1037:actual (948502528, 1224) 
    != expected (2150400, 0)
    [QUOTA WARNING] Usage inconsistent for ID 1367:actual (1078501474304, 
    18130) != expected (20640858112, 55)
    [QUOTA WARNING] Usage inconsistent for ID 1030:actual (84762128384, 
    3973) != expected (26271744, 2)
    [QUOTA WARNING] Usage inconsistent for ID 1041:actual (258749169664, 267 
    != expected (109670400, 11)
    [QUOTA WARNING] Usage inconsistent for ID 1056:actual (389121245184, 
    1333) != expected (9640079360, 21)
    [QUOTA WARNING] Usage inconsistent for ID 1363:actual (1562884812800, 
    28993) != expected (41607168, 44)
    [QUOTA WARNING] Usage inconsistent for ID 1015:actual (263251644416, 177 
    != expected (1230139392,
    [QUOTA WARNING] Usage inconsistent for ID 1027:actual (121355857920, 
    4042) != expected (918818816, 5)
    [QUOTA WARNING] Usage inconsistent for ID 1000:actual (44268187648, 165) 
    != expected (0, 0)
    jffstg-OST0000: Update quota info for quota type 0.
    [ERROR] quotaio_tree.c:241:find_free_dqentry:: find_free_dqentry(): Data 
    block full unexpectedly.
    [ERROR] quotaio_tree.c:241:find_free_dqentry:: find_free_dqentry(): Data 
    block full unexpectedly.
    [QUOTA WARNING] Usage inconsistent for ID 0:actual (1766738993152, 3492) 
    != expected (0, 0)
    [QUOTA WARNING] Usage inconsistent for ID 100:actual (14970901614592, 
    187889) != expected (0, 0)
    jffstg-OST0000: Update quota info for quota type 1.
    jffstg-OST0000: 191438/30519552 files (22.0% non-contiguous), 
    4088922050/7812984832 blocks
    
    The other OSTs showed similar errors.
    
    I read on some email threads about the issue, and there seems to be 2 
    ways:
    
    1) e2fsck
    
          e2fsck -f -y $DEVICE
    
    2) tune2fs
    
         tune2fs -O ^quota $DEVICE
         tune2fs -O quota $DEVICE
    
    Those commands should be run on the MDT and every OST having e2fsck 
    errors.
    
    Which one do you advice me to apply?
    
    Best regards
    
    --- PD:
    I've made a test on a MDT backup and using e2fsck -f -y $DEVICE and 
    could no solve the error,
    but using tune2fs command, cleaned the errors and do not show when 
    passing e2fsck. The only
    think is that it emits the message:
      Warning: the quota feature is still under development
    See https://ext4.wiki.kernel.org/index.php/Quota for more information
    
    
    -- 
    Ramiro Alba
    
    Centre Tecnològic de Tranferència de Calor
    http://www.cttc.upc.edu
    
    Escola Tècnica Superior d'Enginyeries
    Industrial i Aeronàutica de Terrassa
    Colom 11, E-08222, Terrassa, Barcelona, Spain
    Tel: (+34) 93 739 8928
    
    -- 
    Aquest missatge ha estat analitzat per MailScanner
    a la cerca de virus i d'altres continguts perillosos,
    i es considera que est� net.
    
    



More information about the lustre-discuss mailing list