[lustre-discuss] Size difference between du and quota
Stearman, Marc
stearman2 at llnl.gov
Thu May 21 13:45:13 PDT 2015
And just to add one more item to the list:
If you are running ZFS as the backend storage and have enabled compression, you need to use an additional flag to du:
du -sh <dir> Will give you what is stored on the disk
du --apparent-size <dir> will tell you what the actual size of the data is.
I doubt this is what you are running into, but it's useful knowledge to share.
-Marc
----
D. Marc Stearman
Lustre Operations Lead
stearman2 at llnl.gov
925.423.9670
On May 21, 2015, at 9:36 AM, Martin Hecht <hecht at hlrs.de> wrote:
> Hi,
>
> a few more things which may play a role:
>
> - as you are suspecting, the difference of used blocks vs. used bytes
> might be the reason, especially if there are many very small files, but
> there are more possible causes:
>
> - some tools use 2^10 bytes and some others use 1000 bytes as kb which
> might explain small discrpancies. du and ls are examples for different
> output. However, this can not explain the whole difference.
>
> - your find looks for regular files only (type f), but directories and
> symbolic links consume a few kb as well. If there are many symbolic
> links, this would be the explanation, I think.
>
> - there are also cases in which quota can get out of sync (I don't
> remember the cause, but I have already seen warnings about this in the
> syslog of lustre servers). e2fsck on the ldiskfs level is supposed to
> fix this issue, but I also had cases in which I had to turn off quota by
> means of tune2fs on the ldisk level and turn it on again in order to
> trigger something like a background quotacheck in lustre 2. In lustre
> 1.8 there used to be a tool "quotacheck"
>
> - preallocated stripes on many ost's might be an issue as well, although
> I don't see the discrepancy described on our file systems.
>
> - there might also be orphane objects on the disk, i.e. stripes which
> are not referenced anymore on the lustre level, but which still consume
> disk space (not sure if these may affect quota). An online lfsck is
> supposed to clean them up in lustre 2. In lustre 1.8 one had to run
> several e2fsck on the ldisk level and build databases to run an lfsck,
> but that's not supported anymore in lustre 2.
>
> best regards,
> Martin
>
> On 05/20/2015 10:50 AM, Phill Harvey-Smith wrote:
>> Hi all,
>>
>> One of my users is reporting a massive size difference between the
>> figures reported by du and quota.
>>
>> doing a du -hs on his directory reports :
>> du -hs .
>> 529G .
>>
>> doing a lfs quota -u username /storage reports
>> Filesystem kbytes quota limit grace files quota limit
>> grace
>> /storage 621775192 640000000 640010000 - 601284 1000000
>> 1100000 -
>>
>> Though this user does have a lot of files :
>>
>> find . -type f | wc -l
>> 581519
>>
>> So I suspect that it is the typical thing that quota is reporting used
>> blocks whilst du is reporting used bytes, which can of course be
>> wildly different due to filesystem overhead and wasted unused space at
>> the end of files where a block is allocated but only partially used.
>>
>> Is this likely to be the case ?
>>
>> I'm also not entirely sure what versions of lustre the client machines
>> and MDS / OSS servers are running, as I didn't initially set the
>> system up.
>>
>> Cheers.
>>
>> Phill.
>>
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
More information about the lustre-discuss
mailing list