[lustre-discuss] du / quota reporting with DOM enabled

[Jake Grimmett]

Dear All,

du (and quota) report that files are using double the apparent-size, we 
suspect this is due to DOM being enabled.

This is a new cluster, running 2.16-1 with 3 x MDT and 7 x OST, using 
ZFS for both MDT and OST

looking at a couple of files created with dd:
fmb384_gollum> du -sh *
734M	big_file
390G	huge_file

compare to:
fmb384_gollum> du -sh --apparent-size *
319M	big_file
170G	huge_file

DOM is enabled:
root at dino-s1 11:22 [~]: lfs getstripe /scratch/gollum/huge_file
/scratch/gollum/huge_file
   lcm_layout_gen:    3
   lcm_mirror_count:  1
   lcm_entry_count:   2
     lcme_id:             1
     lcme_mirror_id:      0
     lcme_flags:          init
     lcme_extent.e_start: 0
     lcme_extent.e_end:   1048576
       lmm_stripe_count:  0
       lmm_stripe_size:   1048576
       lmm_pattern:       mdt
       lmm_layout_gen:    0
       lmm_stripe_offset: 0

     lcme_id:             2
     lcme_mirror_id:      0
     lcme_flags:          init
     lcme_extent.e_start: 1048576
     lcme_extent.e_end:   EOF
       lmm_stripe_count:  1
       lmm_stripe_size:   1048576
       lmm_pattern:       raid0
       lmm_layout_gen:    0
       lmm_stripe_offset: 3
       lmm_objects:
       -   0: { l_ost_idx:   3, l_fid: [0x380000402:0xa8e72:0x0] }

Whereas folder without DOM:
# lfs setstripe /scratch/jog/noDOM
# dd if=/dev/zero of=noDOM_100M_file bs=1M count=100
# du -h noDOM_100M_file
92M	noDOM_100M_file

compare to:
# du -h --apparent-size noDOM_100M_file
100M	noDOM_100M_file

quota agrees:
root at dino-s1 14:13 [scratch]: lfs quota -h -u jog
Disk quotas for usr jog (uid 1022):
       Filesystem    used   quota   limit   grace   files   quota 
limit   grace
         /scratch  92.05M      0k      0k       -       4       0 
0

Quota agrees with du, unless we use apparent-size:
fmb384_gollum> du -h /scratch/gollum/*
229M	/scratch/gollum/DOM_100M_file

fmb384_gollum> du -h --apparent-size /scratch/gollum/*
100M	/scratch/gollum/DOM_100M_file

fmb384_gollum> lfs quota -h -u gollum /scratch
Disk quotas for usr gollum (uid 1084):
      Filesystem    used   quota   limit   grace   files   quota   limit 
   grace
        /scratch  228.3M    1.5T      2T       -       5       0       0 
       -

Any thoughts gratefully received...

Jake

-- 
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge CB2 0QH, UK.