[lustre-discuss] Corrupted quota file

[Ramiro Alba]

Hi all,

I am using Lustre 2.5.3 under CentOS 6.5 both servers and clients and
the lustre file system is using 6 OSTs of 30TB size with 2 OSSs.

Last days I've been having problems with OSS server rebooting without a
known reason after running fine for several hours (betwen 3 to 24), and
I also noticed that quotas were not showing the correct usage.

At the last OSS server reboot issue, I deciced to run e2fsck both on MDT 
and
OSTs, and those bellow were the errors at command output:

------------------
MDT:
------------------
e2fsck -f /dev/mapper/vglustre-MDT

e2fsck 1.42.9.wc1 (24-Feb-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
[ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (1045 >= 
10) in user quota file. Quota file is probably corrupted.
Please run e2fsck ( to fix it.
[ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (16322 >= 
10) in user quota file. Quota file is probably corrupted.
Please run e2fsck ( to fix it.
[ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (2940928 
 >= 10) in user quota file. Quota file is probably corrupted.
Please run e2fsck ( to fix it.
jffstg-MDT0000: 1218855/47185920 files (0.2% non-contiguous), 
6191259/23592960 blocks

------------------
OSSs:
------------------
[root at jffoss1 system]# e2fsck -fp /dev/mapper/ost0; e2fsck -fp 
/dev/mapper/ost1; e2fsck -fp /dev/mapper/ost2

MMP interval is 10 seconds and total wait time is 42 seconds. Please 
wait...
jffstg-OST0000: recovering journal
[QUOTA WARNING] Usage inconsistent for ID 0:actual (1766737936384, 348 
!= expected (0, 0)
[QUOTA WARNING] Usage inconsistent for ID 1042:actual (507613184, 3613) 
!= expected (40960, 1)
[QUOTA WARNING] Usage inconsistent for ID 1031:actual (1041013137408, 
527 != expected (42952433664, 119)
[QUOTA WARNING] Usage inconsistent for ID 1304:actual (562630971392, 
5127) != expected (0, 0)
[QUOTA WARNING] Usage inconsistent for ID 1338:actual (1634377928704, 
7847) != expected (2098733056, 12)
[QUOTA WARNING] Usage inconsistent for ID 1037:actual (948502528, 1224) 
!= expected (2150400, 0)
[QUOTA WARNING] Usage inconsistent for ID 1367:actual (1078501474304, 
18130) != expected (20640858112, 55)
[QUOTA WARNING] Usage inconsistent for ID 1030:actual (84762128384, 
3973) != expected (26271744, 2)
[QUOTA WARNING] Usage inconsistent for ID 1041:actual (258749169664, 267 
!= expected (109670400, 11)
[QUOTA WARNING] Usage inconsistent for ID 1056:actual (389121245184, 
1333) != expected (9640079360, 21)
[QUOTA WARNING] Usage inconsistent for ID 1363:actual (1562884812800, 
28993) != expected (41607168, 44)
[QUOTA WARNING] Usage inconsistent for ID 1015:actual (263251644416, 177 
!= expected (1230139392,
[QUOTA WARNING] Usage inconsistent for ID 1027:actual (121355857920, 
4042) != expected (918818816, 5)
[QUOTA WARNING] Usage inconsistent for ID 1000:actual (44268187648, 165) 
!= expected (0, 0)
jffstg-OST0000: Update quota info for quota type 0.
[ERROR] quotaio_tree.c:241:find_free_dqentry:: find_free_dqentry(): Data 
block full unexpectedly.
[ERROR] quotaio_tree.c:241:find_free_dqentry:: find_free_dqentry(): Data 
block full unexpectedly.
[QUOTA WARNING] Usage inconsistent for ID 0:actual (1766738993152, 3492) 
!= expected (0, 0)
[QUOTA WARNING] Usage inconsistent for ID 100:actual (14970901614592, 
187889) != expected (0, 0)
jffstg-OST0000: Update quota info for quota type 1.
jffstg-OST0000: 191438/30519552 files (22.0% non-contiguous), 
4088922050/7812984832 blocks

The other OSTs showed similar errors.

I read on some email threads about the issue, and there seems to be 2 
ways:

1) e2fsck

      e2fsck -f -y $DEVICE

2) tune2fs

     tune2fs -O ^quota $DEVICE
     tune2fs -O quota $DEVICE

Those commands should be run on the MDT and every OST having e2fsck 
errors.

Which one do you advice me to apply?

Best regards

--- PD:
I've made a test on a MDT backup and using e2fsck -f -y $DEVICE and 
could no solve the error,
but using tune2fs command, cleaned the errors and do not show when 
passing e2fsck. The only
think is that it emits the message:
  Warning: the quota feature is still under development
See https://ext4.wiki.kernel.org/index.php/Quota for more information


-- 
Ramiro Alba

Centre Tecnològic de Tranferència de Calor
http://www.cttc.upc.edu

Escola Tècnica Superior d'Enginyeries
Industrial i Aeronà utica de Terrassa
Colom 11, E-08222, Terrassa, Barcelona, Spain
Tel: (+34) 93 739 8928

-- 
Aquest missatge ha estat analitzat per MailScanner
a la cerca de virus i d'altres continguts perillosos,
i es considera que està net.