<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
The filesystem looks like a test system (only 200GB OSTs, the whole filesystem is smaller than a single HDD), so it doesn't make sense to be testing Lustre 2.9 at this point. At a very minimum it should be 2.12.8, but for a new test filesystem it makes sense
to try out master (2.15.0-RC2) since it will be the new LTS release very soon.
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas.<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 29, 2022, at 11:08, Patrick Farrell via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Hasan,</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
That's quite old - 2.9 was released nearly 6 years ago and wasn't a maintenance release, so had no updates after release. I expect some of those bugs are present there. A newer version should help.</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
-Patrick</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
</div>
<div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
</div>
<hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 1275.953125px;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span>
<div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Hasan Rashid <<a href="mailto:mrashid2@uncc.edu" class="">mrashid2@uncc.edu</a>><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Tuesday, March 29, 2022 11:50 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>Patrick Farrell <<a href="mailto:pfarrell@ddn.com" class="">pfarrell@ddn.com</a>><br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a> <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [EXTERNAL] Re: [lustre-discuss] Write Performance is Abnormal for max_dirty_mb Value of 2047</font>
<div class=""> </div>
</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div dir="ltr" class="">
<div class="x_gmail_default" style="font-family: verdana, sans-serif;">Hi Patrick,</div>
<div class="x_gmail_default" style="font-family: verdana, sans-serif;"><br class="">
</div>
<div class="x_gmail_default" style="font-family: verdana, sans-serif;">Thank you so much for getting back to me.<span class="Apple-converted-space"> </span><b class="">The Lustre installation was 2.9.0.</b><span class="Apple-converted-space"> </span>I am working
to run the workloads on Lustre 2.12 release.<br class="">
<br class="">
I have 1 machine for MGS and MDS and 4 machines for OSS. In each OSS, I have 2 OSTs and each of them is 200 GiB in size. All the OSTs are mounted on SSD. The striping layout is the default one. Since for the workload, we have one large file of 5 GiB in size,
the whole resides in a single OST. We also verified that by looking at the rpc_stats.<br class="">
<br class="">
<span id="cid:ii_l1cdgirg1"><image.png></span><br class="">
<br class="">
Following is the hardware configuration for all the machines.</div>
<div class="x_gmail_default" style="font-family: verdana, sans-serif;"><br class="">
<span id="cid:ii_l1cdd8tm0"><image.png></span><br class="">
</div>
<div class="x_gmail_default" style="font-family: verdana, sans-serif;"><br class="">
</div>
<div class="x_gmail_default" style="font-family: verdana, sans-serif;">Please let me know if you need further information.</div>
<div class="x_gmail_default" style="font-family: verdana, sans-serif;"><br class="">
</div>
<div class="x_gmail_default" style="font-family: verdana, sans-serif;">Thanks,</div>
<div class="x_gmail_default" style="font-family: verdana, sans-serif;">Hasan</div>
</div>
<br class="">
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sun, Mar 27, 2022 at 1:54 PM Patrick Farrell <<a href="mailto:pfarrell@ddn.com" class="">pfarrell@ddn.com</a>> wrote:<br class="">
</div>
<blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div dir="ltr" class="">[<strong class=""><font color="red" class="">Caution</font></strong>: Email from External Sender. Do not click or open links or attachments unless you know this sender.]<br class="">
<span class=""> </span>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Hasan,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Historically, there have been several bugs related to write grant when max_dirty_mb is set to large values (depending on a few other details of system setup).</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Write grant allows the client to write data in to memory and write it out asynchronously. When write grant is not available to the client, the client is forced to do sync writes at small sizes. The result looks exactly like this, write performance drops severely.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Depending on what version you're running, you may not have fixes for these bugs. You could either try a newer Lustre version (you didn't mention what you're running) or just use a smaller value of max_dirty_mb.</div>
<div class=""></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
I am surprised to see you're still seeing a speedup from max_dirty_mb values over 1 GiB in size.<br class="">
<br class="">
Can you describe your system a bit more? How many OSTs do you have and how many stripes are you using? max_dirty_mb is a per OST value on the client, not a global one.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
-Patrick</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
</div>
<div id="x_gmail-m_-2937072809918361995appendonsend" class=""></div>
<hr style="display: inline-block; width: 1263.921875px;" class="">
<div id="x_gmail-m_-2937072809918361995divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank" class="">lustre-discuss-bounces@lists.lustre.org</a>>
on behalf of Hasan Rashid via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Friday, March 25, 2022 11:45 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a><span class="Apple-converted-space"> </span><<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>[lustre-discuss] Write Performance is Abnormal for max_dirty_mb Value of 2047</font>
<div class=""> </div>
</div>
<div class="">
<div dir="ltr" class="">
<div style="font-family: verdana, sans-serif;" class="">Hi Everyone,<br class="">
</div>
<div style="font-family: verdana, sans-serif;" class=""><br class="">
</div>
<div style="font-family: verdana, sans-serif;" class="">As the manual suggests, the valid value range for max_dirty_mb is the values larger than 0 and smaller than the lesser of 2048 MiB or 1/4 of client RAM. In my system, the client's RAM is 196 GiB.<b class=""><span class="Apple-converted-space"> </span>So, the
maximum valid value for max_dirty_mb(mdm) is 2047 MiB.</b></div>
<div style="font-family: verdana, sans-serif;" class=""><br class="">
</div>
<div style="font-family: verdana, sans-serif;" class="">However, when we set the max_dirty_mb value to 2047,<span class="Apple-converted-space"> </span><b class="">we see very low write throughput for multiple Filebench workloads that we have tested so far</b>.
I am providing details for one example of the tested workload below.</div>
<div style="font-family: verdana, sans-serif;" class=""><br class="">
</div>
<div style="font-family: verdana, sans-serif;" class=""><b class=""><u class="">Workload Detail:</u></b><span class="Apple-converted-space"> </span>We are doing only random write operation of 1MiB size from one process and one thread to a single large file
of 5GiB size.</div>
<div style="font-family: verdana, sans-serif;" class=""><br class="">
</div>
<div style="font-family: verdana, sans-serif;" class=""><b class=""><u class="">Observed Result:</u></b><span class="Apple-converted-space"> </span>As you can see from the below diagram, as we increase the mdm value from 768 to 1792 by an amount of 256 in each
step, the write throughput has increased gradually. However, for the mdm value of 2047, the result dropped very significantly. The observation holds true for all the workloads we tested so far.</div>
<div style="font-family: verdana, sans-serif;" class=""><br class="">
</div>
<div style="font-family: verdana, sans-serif;" class=""><br class="">
</div>
<div style="font-family: verdana, sans-serif;" class=""><span id="x_gmail-m_-2937072809918361995x_gmail-docs-internal-guid-18e99bf1-7fff-815f-a4c8-b3622739cea4" class=""><span style="font-size: 20pt; font-family: Arial; background-color: transparent; font-variant-numeric: normal; font-variant-east-asian: normal; vertical-align: baseline; white-space: pre-wrap;" class=""><span style="border: none; display: inline-block; overflow: hidden; width: 280px; height: 277px;" class=""><img width="280" height="277" src="https://lh3.googleusercontent.com/iEqpGNZhI9r9jJCLq0rWPvFADJRXkKKKZnyCV_8m3nhiHggNqWU9d_7WTUU0yeb011nxjULF4_iLkI7TIc0qe5el11PJI3i9Jot9KveXUil98A_UEnBojFqAHfK94ve1foQT39m2" style="margin-left: 0px; margin-top: 0px;" class=""></span></span></span><br class="">
</div>
<div style="font-family: verdana, sans-serif;" class=""><br class="">
</div>
<div style="font-family: verdana, sans-serif;" class="">I am unable to figure out why we would have such low performance at the mdm value of 2047. Please share any insights you have that would be helpful for me to understand the aforementioned scenario.</div>
<div style="font-family: verdana, sans-serif;" class=""><br class="">
</div>
<div style="font-family: verdana, sans-serif;" class="">Best Wishes,</div>
<div style="font-family: verdana, sans-serif;" class="">Md Hasanur Rashid</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">lustre-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span></div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>