[lustre-discuss] [EXTERNAL] Re: Write Performance is Abnormal for max_dirty_mb Value of 2047

[Hasan Rashid]

Hi Patrick,

Thank you so much for getting back to me. *The Lustre installation was
2.9.0.* I am working to run the workloads on Lustre 2.12 release.

I have 1 machine for MGS and MDS and 4 machines for OSS. In each OSS, I
have 2 OSTs and each of them is 200 GiB in size. All the OSTs are mounted
on SSD. The striping layout is the default one. Since for the workload, we
have one large file of 5 GiB in size, the whole resides in a single OST. We
also verified that by looking at the rpc_stats.

[image: image.png]

Following is the hardware configuration for all the machines.

[image: image.png]

Please let me know if you need further information.

Thanks,
Hasan

On Sun, Mar 27, 2022 at 1:54 PM Patrick Farrell <pfarrell at ddn.com> wrote:

> [*Caution*: Email from External Sender. Do not click or open links or
> attachments unless you know this sender.]
>
> Hasan,
>
> Historically, there have been several bugs related to write grant when
> max_dirty_mb is set to large values (depending on a few other details of
> system setup).
>
> Write grant allows the client to write data in to memory and write it out
> asynchronously.  When write grant is not available to the client, the
> client is forced to do sync writes at small sizes.  The result looks
> exactly like this, write performance drops severely.
>
> Depending on what version you're running, you may not have fixes for these
> bugs.  You could either try a newer Lustre version (you didn't mention what
> you're running) or just use a smaller value of max_dirty_mb.
>
> I am surprised to see you're still seeing a speedup from max_dirty_mb
> values over 1 GiB in size.
>
> Can you describe your system a bit more?  How many OSTs do you have and
> how many stripes are you using?  max_dirty_mb is a per OST value on the
> client, not a global one.
>
> -Patrick
> ------------------------------
> *From:* lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on
> behalf of Hasan Rashid via lustre-discuss <lustre-discuss at lists.lustre.org
> >
> *Sent:* Friday, March 25, 2022 11:45 AM
> *To:* lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
> *Subject:* [lustre-discuss] Write Performance is Abnormal for
> max_dirty_mb Value of 2047
>
> Hi Everyone,
>
> As the manual suggests, the valid value range for max_dirty_mb is the
> values larger than 0 and smaller than the lesser of 2048 MiB or 1/4 of
> client RAM. In my system, the client's RAM is 196 GiB.* So, the maximum
> valid value for max_dirty_mb(mdm) is 2047 MiB.*
>
> However, when we set the max_dirty_mb value to 2047, *we see very low
> write throughput for multiple Filebench workloads that we have tested
> so far*. I am providing details for one example of the tested workload
> below.
>
> *Workload Detail:* We are doing only random write operation of 1MiB size
> from one process and one thread to a single large file of 5GiB size.
>
> *Observed Result:* As you can see from the below diagram, as we increase
> the mdm value from 768 to 1792 by an amount of 256 in each step, the write
> throughput has increased gradually. However, for the mdm value of 2047, the
> result dropped very significantly. The observation holds true for all the
> workloads we tested so far.
>
>
>
>
> I am unable to figure out why we would have such low performance at the
> mdm value of 2047. Please share any insights you have that would be helpful
> for me to understand the aforementioned scenario.
>
> Best Wishes,
> Md Hasanur Rashid
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220329/85d3fef2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 193662 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220329/85d3fef2/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 114887 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220329/85d3fef2/attachment-0003.png>