[lustre-discuss] Corrupted quota file

Fri Sep 30 09:41:47 PDT 2016

Yes. You should always use the latest and greatest for e2fsprogs - they
are not tied to the latest version at the time of the Lustre release you
are using

On 9/30/16, 9:00 AM, "lustre-discuss on behalf of Ramiro Alba"
<lustre-discuss-bounces at lists.lustre.org on behalf of raq at cttc.upc.edu>
wrote:

>Andreas,
>
>I am using lustre 2.5.3 so I installed the recommended e2fstools for
>that version at that moment
>Does it means that higher e2fsprogs versions are suitable for lower
>lustre versions?
>
>Best regards
>
>On 2016-09-30 17:46, Dilger, Andreas wrote:
>
>> I would recommend to install a more modern e2fsprogs.
>> e2fsck-1.42.13.wc5 is the current version.
>> 
>> Cheers, Andreas
>> --
>> Andreas Dilger
>> 
>> Lustre Principal Architect
>> Intel High Performance Data Division
>> 
>> On 2016/09/30, 08:16, "lustre-discuss on behalf of Ramiro Alba"
>> <lustre-discuss-bounces at lists.lustre.org on behalf of raq at cttc.upc.edu>
>> wrote:
>> 
>> Hi all,
>> 
>> I am using Lustre 2.5.3 under CentOS 6.5 both servers and clients and
>> the lustre file system is using 6 OSTs of 30TB size with 2 OSSs.
>> 
>> Last days I've been having problems with OSS server rebooting without a
>> known reason after running fine for several hours (betwen 3 to 24), and
>> I also noticed that quotas were not showing the correct usage.
>> 
>> At the last OSS server reboot issue, I deciced to run e2fsck both on
>> MDT
>> and
>> OSTs, and those bellow were the errors at command output:
>> 
>> ------------------
>> MDT:
>> ------------------
>> e2fsck -f /dev/mapper/vglustre-MDT
>> 
>> e2fsck 1.42.9.wc1 (24-Feb-2014)
>> Pass 1: Checking inodes, blocks, and sizes
>> Pass 2: Checking directory structure
>> Pass 3: Checking directory connectivity
>> Pass 4: Checking reference counts
>> Pass 5: Checking group summary information
>> [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (1045 >=
>> 10) in user quota file. Quota file is probably corrupted.
>> Please run e2fsck ( to fix it.
>> [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (16322
>> >=
>> 10) in user quota file. Quota file is probably corrupted.
>> Please run e2fsck ( to fix it.
>> [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (2940928
>>> = 10) in user quota file. Quota file is probably corrupted.
>> Please run e2fsck ( to fix it.
>> jffstg-MDT0000: 1218855/47185920 files (0.2% non-contiguous),
>> 6191259/23592960 blocks
>> 
>> ------------------
>> OSSs:
>> ------------------
>> [root at jffoss1 system]# e2fsck -fp /dev/mapper/ost0; e2fsck -fp
>> /dev/mapper/ost1; e2fsck -fp /dev/mapper/ost2
>> 
>> MMP interval is 10 seconds and total wait time is 42 seconds. Please
>> wait...
>> jffstg-OST0000: recovering journal
>> [QUOTA WARNING] Usage inconsistent for ID 0:actual (1766737936384, 348
>> != expected (0, 0)
>> [QUOTA WARNING] Usage inconsistent for ID 1042:actual (507613184, 3613)
>> != expected (40960, 1)
>> [QUOTA WARNING] Usage inconsistent for ID 1031:actual (1041013137408,
>> 527 != expected (42952433664, 119)
>> [QUOTA WARNING] Usage inconsistent for ID 1304:actual (562630971392,
>> 5127) != expected (0, 0)
>> [QUOTA WARNING] Usage inconsistent for ID 1338:actual (1634377928704,
>> 7847) != expected (2098733056, 12)
>> [QUOTA WARNING] Usage inconsistent for ID 1037:actual (948502528, 1224)
>> != expected (2150400, 0)
>> [QUOTA WARNING] Usage inconsistent for ID 1367:actual (1078501474304,
>> 18130) != expected (20640858112, 55)
>> [QUOTA WARNING] Usage inconsistent for ID 1030:actual (84762128384,
>> 3973) != expected (26271744, 2)
>> [QUOTA WARNING] Usage inconsistent for ID 1041:actual (258749169664,
>> 267
>> != expected (109670400, 11)
>> [QUOTA WARNING] Usage inconsistent for ID 1056:actual (389121245184,
>> 1333) != expected (9640079360, 21)
>> [QUOTA WARNING] Usage inconsistent for ID 1363:actual (1562884812800,
>> 28993) != expected (41607168, 44)
>> [QUOTA WARNING] Usage inconsistent for ID 1015:actual (263251644416,
>> 177
>> != expected (1230139392,
>> [QUOTA WARNING] Usage inconsistent for ID 1027:actual (121355857920,
>> 4042) != expected (918818816, 5)
>> [QUOTA WARNING] Usage inconsistent for ID 1000:actual (44268187648,
>> 165)
>> != expected (0, 0)
>> jffstg-OST0000: Update quota info for quota type 0.
>> [ERROR] quotaio_tree.c:241:find_free_dqentry:: find_free_dqentry():
>> Data
>> block full unexpectedly.
>> [ERROR] quotaio_tree.c:241:find_free_dqentry:: find_free_dqentry():
>> Data
>> block full unexpectedly.
>> [QUOTA WARNING] Usage inconsistent for ID 0:actual (1766738993152,
>> 3492)
>> != expected (0, 0)
>> [QUOTA WARNING] Usage inconsistent for ID 100:actual (14970901614592,
>> 187889) != expected (0, 0)
>> jffstg-OST0000: Update quota info for quota type 1.
>> jffstg-OST0000: 191438/30519552 files (22.0% non-contiguous),
>> 4088922050/7812984832 blocks
>> 
>> The other OSTs showed similar errors.
>> 
>> I read on some email threads about the issue, and there seems to be 2
>> ways:
>> 
>> 1) e2fsck
>> 
>> e2fsck -f -y $DEVICE
>> 
>> 2) tune2fs
>> 
>> tune2fs -O ^quota $DEVICE
>> tune2fs -O quota $DEVICE
>> 
>> Those commands should be run on the MDT and every OST having e2fsck
>> errors.
>> 
>> Which one do you advice me to apply?
>> 
>> Best regards
>> 
>> --- PD:
>> I've made a test on a MDT backup and using e2fsck -f -y $DEVICE and
>> could no solve the error,
>> but using tune2fs command, cleaned the errors and do not show when
>> passing e2fsck. The only
>> think is that it emits the message:
>> Warning: the quota feature is still under development
>> See https://ext4.wiki.kernel.org/index.php/Quota for more information
>> 
>> --
>> Ramiro Alba
>> 
>> Centre Tecnològic de Tranferència de Calor
>> http://www.cttc.upc.edu
>> 
>> Escola Tècnica Superior d'Enginyeries
>> Industrial i Aeronàutica de Terrassa
>> Colom 11, E-08222, Terrassa, Barcelona, Spain
>> Tel: (+34) 93 739 8928
>> 
>> --
>> Aquest missatge ha estat analitzat per MailScanner
>> a la cerca de virus i d'altres continguts perillosos,
>> i es considera que est� net.
>
>
>-- 
>Ramiro Alba
>
>Centre Tecnològic de Tranferència de Calor
>http://www.cttc.upc.edu
>
>Escola Tècnica Superior d'Enginyeries
>Industrial i Aeronàutica de Terrassa
>Colom 11, E-08222, Terrassa, Barcelona, Spain
>Tel: (+34) 93 739 8928
>
>-- 
>Aquest missatge ha estat analitzat per MailScanner
>a la cerca de virus i d'altres continguts perillosos,
>i es considera que est� net.
>