<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">If it is taking too long for targets to sync-up you can tune the activity and speed things up by adjusting some osp tunables.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">First, monitor osp sync_in_progress and destroys_in_flight to see if that’s what’s going on. Then you can tune up the MDS’s osp’s max_rpcs_in_progress if necessary.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Cory<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 4/19/22, 7:31 PM, "lustre-discuss" <lustre-discuss-bounces@lists.lustre.org> wrote:</p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">One thing you can look at is running 'zpool iostat 1' (there are many
<br>
options) to monitor that ZFS is still doing I/O during that time gap. <br>
With NVMe though, as Andreas said, I would expect that time gap to last <br>
seconds to minutes, not hours.<br>
<br>
On 4/19/22 02:16, Einar Næss Jensen wrote:<br>
> Thank you for answering Andreas.<br>
><br>
> Lustre version is 2.12.8<br>
><br>
> It is indeed when we delete io500 files when we discovered this, but we also see it when deleting other files, that the "df" lags 1-2 hours behind.<br>
> We see it both on nvme and ssd drives. Haven't checkd hdd drives/osts yet.<br>
><br>
> This is a new lustre setup, and benchmarks are good (in our opinion). For now it is just this annoyance bugging us.<br>
> We didn't notice on previous lustre setup but will check if we see it there also.<br>
><br>
><br>
> Einar<br>
><br>
><br>
><br>
><br>
> ________________________________________<br>
> From: Andreas Dilger <adilger@whamcloud.com><br>
> Sent: Monday, April 11, 2022 18:01<br>
> To: Einar Næss Jensen<br>
> Cc: lustre-discuss@lists.lustre.org<br>
> Subject: Re: [lustre-discuss] question regarding du vs df on lustre<br>
><br>
> Lustre is returning the file unlink from the MDS immediately, but deleting<br>
> the objects from the OSTs asynchronously in the background.<br>
><br>
> How many files are being deleted in this case? If you are running tests<br>
> like IO500, where there are many millions of small files plus some huge<br>
> files, then it may be that huge object deletion is behind small objects?<br>
><br>
> That said, it probably shouldn't take hours to finish if the OST storage is<br>
> NVMe based.<br>
><br>
> Cheers, Andreas<br>
><br>
>> On Apr 4, 2022, at 05:05, Einar Næss Jensen <einar.nass.jensen@ntnu.no> wrote:<br>
>><br>
>> Hello lustre people.<br>
>><br>
>> We are experimenting with lustre on nvme, and observe the following issue:<br>
>> After running benchmarks and deleting benchmark files, we see that df and du reports different sizes:<br>
>><br>
>> [root@idun-02-27 ~]# du -hs /nvme/<br>
>> 38M /nvme/<br>
>> [root@idun-02-27 ~]# df -h|grep nvme<br>
>> 10.3.1.2@o2ib:/nvme 5.5T 3.9T 1.3T 76% /nvme<br>
>><br>
>><br>
>> It takes several hours before du and df agrees.<br>
>><br>
>> What is causing this?<br>
>> How can we get updated records for df immediately when deleting files?<br>
>><br>
>><br>
>> Best REegards<br>
>> Einar<br>
>> _______________________________________________<br>
>> lustre-discuss mailing list<br>
>> lustre-discuss@lists.lustre.org<br>
>> <a href="https://urldefense.us/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">https://urldefense.us/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> lustre-discuss@lists.lustre.org<br>
> <a href="https://urldefense.us/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">https://urldefense.us/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</p>
</div>
</div>
</body>
</html>