[lustre-discuss] question regarding du vs df on lustre

[Cameron Harr]

One thing you can look at is running 'zpool iostat 1' (there are many 
options) to monitor that ZFS is still doing I/O during that time gap. 
With NVMe though, as Andreas said, I would expect that time gap to last 
seconds to minutes, not hours.

On 4/19/22 02:16, Einar Næss Jensen wrote:
> Thank you for answering Andreas.
>
> Lustre version is 2.12.8
>
> It is indeed when we delete io500 files when we discovered this, but we also see it when deleting other files, that the "df" lags 1-2 hours behind.
> We see it both on nvme and ssd drives. Haven't checkd hdd drives/osts yet.
>
> This is a new lustre setup, and benchmarks are good (in our opinion). For now it is just this annoyance bugging us.
> We didn't notice on previous lustre setup but will check if we see it there also.
>
>
> Einar
>
>
>
>
> ________________________________________
> From: Andreas Dilger <adilger at whamcloud.com>
> Sent: Monday, April 11, 2022 18:01
> To: Einar Næss Jensen
> Cc: lustre-discuss at lists.lustre.org
> Subject: Re: [lustre-discuss] question regarding du vs df on lustre
>
> Lustre is returning the file unlink from the MDS immediately, but deleting
> the objects from the OSTs asynchronously in the background.
>
> How many files are being deleted in this case?  If you are running tests
> like IO500, where there are many millions of small files plus some huge
> files, then it may be that huge object deletion is behind small objects?
>
> That said, it probably shouldn't take hours to finish if the OST storage is
> NVMe based.
>
> Cheers, Andreas
>
>> On Apr 4, 2022, at 05:05, Einar Næss Jensen <einar.nass.jensen at ntnu.no> wrote:
>>
>> Hello lustre people.
>>
>> We are experimenting with lustre on nvme, and observe the following issue:
>> After running benchmarks and deleting benchmark files, we see that df and du reports different sizes:
>>
>> [root at idun-02-27 ~]#  du -hs /nvme/
>> 38M     /nvme/
>> [root at idun-02-27 ~]# df -h|grep nvme
>> 10.3.1.2 at o2ib:/nvme                   5.5T  3.9T  1.3T  76% /nvme
>>
>>
>> It takes several hours before du and df agrees.
>>
>> What is causing this?
>> How can we get updated records for df immediately when deleting files?
>>
>>
>> Best REegards
>> Einar
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> https://urldefense.us/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!G2kpM7uM-TzIFchu!jfI9cwREBq_m05-u6IlWShROtv7oEhRhs5fbpFCgf_agCpH6myqZ3tudujL7OAU$
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> https://urldefense.us/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!G2kpM7uM-TzIFchu!jfI9cwREBq_m05-u6IlWShROtv7oEhRhs5fbpFCgf_agCpH6myqZ3tudujL7OAU$