[lustre-discuss] Size difference between du and quota

[Stearman, Marc]

And just to add one more item to the list:

If you are running ZFS as the backend storage and have enabled compression, you need to use an additional flag to du:

du -sh <dir>  Will give you what is stored on the disk
du --apparent-size <dir> will tell you what the actual size of the data is.

I doubt this is what you are running into, but it's useful knowledge to share.

-Marc

----
D. Marc Stearman
Lustre Operations Lead
stearman2 at llnl.gov
925.423.9670




On May 21, 2015, at 9:36 AM, Martin Hecht <hecht at hlrs.de> wrote:

> Hi,
> 
> a few more things which may play a role:
> 
> - as you are suspecting, the difference of used blocks vs. used bytes
> might be the reason, especially if there are many very small files, but
> there are more possible causes:
> 
> - some tools use 2^10 bytes and some others use 1000 bytes as kb which
> might explain small discrpancies. du and ls are examples for different
> output. However, this can not explain the whole difference.
> 
> - your find looks for regular files only (type f), but directories and
> symbolic links consume a few kb as well. If there are many symbolic
> links, this would be the explanation, I think.
> 
> - there are also cases in which quota can get out of sync (I don't
> remember the cause, but I have already seen warnings about this in the
> syslog of lustre servers). e2fsck on the ldiskfs level is supposed to
> fix this issue, but I also had cases in which I had to turn off quota by
> means of tune2fs on the ldisk level and turn it on again in order to
> trigger something like a background quotacheck in lustre 2. In lustre
> 1.8 there used to be a tool "quotacheck"
> 
> - preallocated stripes on many ost's might be an issue as well, although
> I don't see the discrepancy described on our file systems.
> 
> - there might also be orphane objects on the disk, i.e. stripes which
> are not referenced anymore on the lustre level, but which still consume
> disk space (not sure if these may affect quota). An online lfsck is
> supposed to clean them up in lustre 2. In lustre 1.8 one had to run
> several e2fsck on the ldisk level and build databases to run an lfsck,
> but that's not supported anymore in lustre 2.
> 
> best regards,
> Martin
> 
> On 05/20/2015 10:50 AM, Phill Harvey-Smith wrote:
>> Hi all,
>> 
>> One of my users is reporting a massive size difference between the
>> figures reported by du and quota.
>> 
>> doing a du -hs on his directory reports :
>> du -hs .
>> 529G    .
>> 
>> doing a lfs quota -u username /storage reports
>> Filesystem  kbytes   quota   limit   grace   files   quota   limit  
>> grace
>> /storage 621775192  640000000 640010000       -  601284  1000000
>> 1100000       -
>> 
>> Though this user does have a lot of files :
>> 
>> find . -type f | wc -l
>> 581519
>> 
>> So I suspect that it is the typical thing that quota is reporting used
>> blocks whilst du is reporting used bytes, which can of course be
>> wildly different due to filesystem overhead and wasted unused space at
>> the end of files where a block is allocated but only partially used.
>> 
>> Is this likely to be the case ?
>> 
>> I'm also not entirely sure what versions of lustre the client machines
>> and MDS / OSS servers are running, as I didn't initially set the
>> system up.
>> 
>> Cheers.
>> 
>> Phill.
>> 
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 
> 
> 
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org