[lustre-discuss] Corrupted quota file

Fri Sep 30 09:00:47 PDT 2016

Andreas,

I am using lustre 2.5.3 so I installed the recommended e2fstools for 
that version at that moment
Does it means that higher e2fsprogs versions are suitable for lower 
lustre versions?

Best regards

On 2016-09-30 17:46, Dilger, Andreas wrote:

> I would recommend to install a more modern e2fsprogs.
> e2fsck-1.42.13.wc5 is the current version.
> 
> Cheers, Andreas
> --
> Andreas Dilger
> 
> Lustre Principal Architect
> Intel High Performance Data Division
> 
> On 2016/09/30, 08:16, "lustre-discuss on behalf of Ramiro Alba" 
> <lustre-discuss-bounces at lists.lustre.org on behalf of raq at cttc.upc.edu> 
> wrote:
> 
> Hi all,
> 
> I am using Lustre 2.5.3 under CentOS 6.5 both servers and clients and
> the lustre file system is using 6 OSTs of 30TB size with 2 OSSs.
> 
> Last days I've been having problems with OSS server rebooting without a
> known reason after running fine for several hours (betwen 3 to 24), and
> I also noticed that quotas were not showing the correct usage.
> 
> At the last OSS server reboot issue, I deciced to run e2fsck both on 
> MDT
> and
> OSTs, and those bellow were the errors at command output:
> 
> ------------------
> MDT:
> ------------------
> e2fsck -f /dev/mapper/vglustre-MDT
> 
> e2fsck 1.42.9.wc1 (24-Feb-2014)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (1045 >=
> 10) in user quota file. Quota file is probably corrupted.
> Please run e2fsck ( to fix it.
> [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (16322 
> >=
> 10) in user quota file. Quota file is probably corrupted.
> Please run e2fsck ( to fix it.
> [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (2940928
>> = 10) in user quota file. Quota file is probably corrupted.
> Please run e2fsck ( to fix it.
> jffstg-MDT0000: 1218855/47185920 files (0.2% non-contiguous),
> 6191259/23592960 blocks
> 
> ------------------
> OSSs:
> ------------------
> [root at jffoss1 system]# e2fsck -fp /dev/mapper/ost0; e2fsck -fp
> /dev/mapper/ost1; e2fsck -fp /dev/mapper/ost2
> 
> MMP interval is 10 seconds and total wait time is 42 seconds. Please
> wait...
> jffstg-OST0000: recovering journal
> [QUOTA WARNING] Usage inconsistent for ID 0:actual (1766737936384, 348
> != expected (0, 0)
> [QUOTA WARNING] Usage inconsistent for ID 1042:actual (507613184, 3613)
> != expected (40960, 1)
> [QUOTA WARNING] Usage inconsistent for ID 1031:actual (1041013137408,
> 527 != expected (42952433664, 119)
> [QUOTA WARNING] Usage inconsistent for ID 1304:actual (562630971392,
> 5127) != expected (0, 0)
> [QUOTA WARNING] Usage inconsistent for ID 1338:actual (1634377928704,
> 7847) != expected (2098733056, 12)
> [QUOTA WARNING] Usage inconsistent for ID 1037:actual (948502528, 1224)
> != expected (2150400, 0)
> [QUOTA WARNING] Usage inconsistent for ID 1367:actual (1078501474304,
> 18130) != expected (20640858112, 55)
> [QUOTA WARNING] Usage inconsistent for ID 1030:actual (84762128384,
> 3973) != expected (26271744, 2)
> [QUOTA WARNING] Usage inconsistent for ID 1041:actual (258749169664, 
> 267
> != expected (109670400, 11)
> [QUOTA WARNING] Usage inconsistent for ID 1056:actual (389121245184,
> 1333) != expected (9640079360, 21)
> [QUOTA WARNING] Usage inconsistent for ID 1363:actual (1562884812800,
> 28993) != expected (41607168, 44)
> [QUOTA WARNING] Usage inconsistent for ID 1015:actual (263251644416, 
> 177
> != expected (1230139392,
> [QUOTA WARNING] Usage inconsistent for ID 1027:actual (121355857920,
> 4042) != expected (918818816, 5)
> [QUOTA WARNING] Usage inconsistent for ID 1000:actual (44268187648, 
> 165)
> != expected (0, 0)
> jffstg-OST0000: Update quota info for quota type 0.
> [ERROR] quotaio_tree.c:241:find_free_dqentry:: find_free_dqentry(): 
> Data
> block full unexpectedly.
> [ERROR] quotaio_tree.c:241:find_free_dqentry:: find_free_dqentry(): 
> Data
> block full unexpectedly.
> [QUOTA WARNING] Usage inconsistent for ID 0:actual (1766738993152, 
> 3492)
> != expected (0, 0)
> [QUOTA WARNING] Usage inconsistent for ID 100:actual (14970901614592,
> 187889) != expected (0, 0)
> jffstg-OST0000: Update quota info for quota type 1.
> jffstg-OST0000: 191438/30519552 files (22.0% non-contiguous),
> 4088922050/7812984832 blocks
> 
> The other OSTs showed similar errors.
> 
> I read on some email threads about the issue, and there seems to be 2
> ways:
> 
> 1) e2fsck
> 
> e2fsck -f -y $DEVICE
> 
> 2) tune2fs
> 
> tune2fs -O ^quota $DEVICE
> tune2fs -O quota $DEVICE
> 
> Those commands should be run on the MDT and every OST having e2fsck
> errors.
> 
> Which one do you advice me to apply?
> 
> Best regards
> 
> --- PD:
> I've made a test on a MDT backup and using e2fsck -f -y $DEVICE and
> could no solve the error,
> but using tune2fs command, cleaned the errors and do not show when
> passing e2fsck. The only
> think is that it emits the message:
> Warning: the quota feature is still under development
> See https://ext4.wiki.kernel.org/index.php/Quota for more information
> 
> --
> Ramiro Alba
> 
> Centre TecnolÃ²gic de TranferÃ¨ncia de Calor
> http://www.cttc.upc.edu
> 
> Escola TÃ¨cnica Superior d'Enginyeries
> Industrial i AeronÃ utica de Terrassa
> Colom 11, E-08222, Terrassa, Barcelona, Spain
> Tel: (+34) 93 739 8928
> 
> --
> Aquest missatge ha estat analitzat per MailScanner
> a la cerca de virus i d'altres continguts perillosos,
> i es considera que estï¿½ net.

-- 
Ramiro Alba

Centre TecnolÃ²gic de TranferÃ¨ncia de Calor
http://www.cttc.upc.edu

Escola TÃ¨cnica Superior d'Enginyeries
Industrial i AeronÃ utica de Terrassa
Colom 11, E-08222, Terrassa, Barcelona, Spain
Tel: (+34) 93 739 8928

-- 
Aquest missatge ha estat analitzat per MailScanner
a la cerca de virus i d'altres continguts perillosos,
i es considera que està net.