<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Both methods should produce equivalent numbers. On my 2.14.0 system:
<div class=""><br class="">
</div>
<div class=""># ps auxww | grep mdt0<br class="">
root 594183 0.0 0.0 0 0 ? I Sep09 0:41 [mdt00_000]<br class="">
root 594184 0.0 0.0 0 0 ? I Sep09 0:34 [mdt00_001]<br class="">
root 594185 0.0 0.0 0 0 ? I Sep09 0:37 [mdt00_002]<br class="">
root 594288 0.0 0.0 0 0 ? I Sep09 0:36 [mdt00_003]<br class="">
root 594664 0.0 0.0 0 0 ? I Sep09 0:25 [mdt00_004]<br class="">
root 594665 0.0 0.0 0 0 ? I Sep09 0:32 [mdt00_005]<br class="">
root 594667 0.0 0.0 0 0 ? I Sep09 0:41 [mdt00_006]<br class="">
root 594668 0.0 0.0 0 0 ? I Sep09 0:30 [mdt00_007]<br class="">
root 594670 0.0 0.0 0 0 ? I Sep09 0:30 [mdt00_008]<br class="">
root 594673 0.0 0.0 0 0 ? I Sep09 0:37 [mdt00_009]<br class="">
root 594680 0.0 0.0 0 0 ? I Sep09 0:40 [mdt00_010]</div>
<div class=""># lctl get_param mds.MDS.mdt.threads_started<br class="">
mds.MDS.mdt.threads_started=11</div>
<div class=""><br class="">
</div>
<div class="">This is the service thread for most MDT RPC requests. Definitely increasing the number of "mdt" threads will improve metadata performance, as long as the underlying storage has IOPS for it. Having too many threads would use more memory, and
in the rare case of an HDD-based MDT this might cause excessive seeking (that is obviously not a problem for SSD/NVMe MDTs).</div>
<div class=""><br class="">
<div># ps auxww | grep mdt_rdpg<br class="">
root 594186 0.0 0.0 0 0 ? I Sep09 0:02 [mdt_rdpg00_000]<br class="">
root 594187 0.0 0.0 0 0 ? I Sep09 0:02 [mdt_rdpg00_001]<br class="">
root 594663 0.0 0.0 0 0 ? I Sep09 0:03 [mdt_rdpg00_002]<br class="">
# lctl get_param mds.MDS.mdt_readpage.threads_started<br class="">
mds.MDS.mdt_readpage.threads_started=3</div>
<div><br class="">
</div>
<div>This service is for bulk readdir RPCs.</div>
<div><br class="">
</div>
<div>Cheers, Andreas</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Sep 22, 2021, at 15:16, Houkun Zhu <<a href="mailto:diskun.zhu@gmail.com" class="">diskun.zhu@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">I’m running lustre 2.12.7. The workload I was running was generated by fio, i.e., 6 process send I/O requests to the server. As I could see the non-trivial performance difference when I increase the parameter mds.MDS.mdt.threads_max, I assume
it played a role in the performance. <br class="">
<br class="">
Thanks to the tip from Patrick, I just executed command ps axu|grep mdt and got the following result,<br class="">
<br class="">
root 17654 0.0 0.0 0 0 ? S Aug07 0:13 [mdt00_006]<br class="">
root 18672 0.0 0.0 0 0 ? S Aug07 0:21 [mdt00_007]<br class="">
root 18902 0.0 0.0 0 0 ? S Aug07 0:11 [mdt00_008]<br class="">
root 23778 0.0 0.0 0 0 ? S Sep21 0:46 [mdt01_003]<br class="">
root 24032 0.0 0.0 0 0 ? S Sep21 0:14 [mdt_rdpg01_002]<br class="">
root 25292 0.0 0.0 0 0 ? S Sep21 0:34 [mdt01_004]<br class="">
root 25293 0.0 0.0 0 0 ? S Sep21 0:36 [mdt01_005]<br class="">
root 25294 0.0 0.0 0 0 ? S Sep21 0:35 [mdt01_006]<br class="">
root 25295 0.0 0.0 0 0 ? S Sep21 0:37 [mdt01_007]<br class="">
root 25296 0.0 0.0 0 0 ? S Sep21 0:36 [mdt01_008]<br class="">
root 25297 0.0 0.0 0 0 ? S Sep21 0:11 [mdt_rdpg01_003]<br class="">
root 25298 0.0 0.0 0 0 ? S Sep21 0:38 [mdt01_009]<br class="">
root 25299 0.0 0.0 0 0 ? S Sep21 0:38 [mdt01_010]<br class="">
root 25301 0.0 0.0 0 0 ? S Sep21 0:11 [mdt_rdpg01_004]<br class="">
root 25302 0.0 0.0 0 0 ? S Sep21 0:37 [mdt01_011]<br class="">
root 25303 0.0 0.0 0 0 ? S Sep21 0:35 [mdt01_012]<br class="">
root 25304 0.0 0.0 0 0 ? S Sep21 0:11 [mdt_rdpg01_005]<br class="">
root 25370 0.0 0.0 0 0 ? S Sep21 0:11 [mdt_rdpg01_006]<br class="">
root 25375 0.0 0.0 0 0 ? S Sep21 0:09 [mdt_rdpg01_007]<br class="">
root 29073 0.0 0.0 0 0 ? S Aug26 0:00 [mdt_rdpg00_003]<br class="">
<br class="">
Could I verify my assumption by counting the number of process mdt\d\d_\d*?<br class="">
<br class="">
Best regards,<br class="">
Houkun<br class="">
<br class="">
<blockquote type="cite" class="">On 22. Sep 2021, at 21:21, Andreas Dilger <<a href="mailto:adilger@whamcloud.com" class="">adilger@whamcloud.com</a>> wrote:<br class="">
<br class="">
What version of Lustre are you running? I tested with 2.14.0 and observed that *.*.threads_started increased and (eventually) decreased as the service threads were being used. Note that the "*.*.threads_max" parameter is the *maximum* number of threads for
a particular service (e.g. ost.OSS.ost_io.* is for bulk read/write IO operations, while ost.OSS.ost.* is for most other OST operations). New threads are only started if the number of incoming requests in the queue exceeds the number of currently running threads,
so if the requests are processed quickly and/or there are not enough clients generating RPCs, then new threads will not be started beyond the number to manage the current workload.<br class="">
<br class="">
For example, I had reduced ost_io.threads_max=16 on my home filesystemyesterday to verify that the threads eventually stopped (that needed some ongoing IO workload until the higher-numbered threads processed a request and were the last thread running (see comment
at ptlrpc_thread_should_stop() for details):<br class="">
<br class="">
# lctl get_param ost.OSS.ost_io.threads*<br class="">
ost.OSS.ost_io.threads_max=16<br class="">
ost.OSS.ost_io.threads_min=3<br class="">
ost.OSS.ost_io.threads_started=16<br class="">
<br class="">
When I increased threads_max=32 and ran a parallel IO workload on a client it increased the threads_started, but wasn't able to generate enough RPCs in flight to hit the maximum number of threads:<br class="">
<br class="">
# lctl get_param ost.OSS.ost_io.threads*<br class="">
ost.OSS.ost_io.threads_max=32<br class="">
ost.OSS.ost_io.threads_min=3<br class="">
ost.OSS.ost_io.threads_started=26<br class="">
<br class="">
On Sep 22, 2021, at 11:37, Houkun Zhu <<a href="mailto:diskun.zhu@gmail.com" class="">diskun.zhu@gmail.com</a>> wrote:<br class="">
<blockquote type="cite" class=""><br class="">
Hi Andres,<br class="">
<br class="">
Thanks a lot for your help. I actually record the parameter mds.MDS.mdt.threads_started. But its value never changes. However, I can observe the performance difference (i.e., throughput is increased tremendously) when I set a higher value of threads_max for
mds. <br class="">
<br class="">
<br class="">
Best regards,<br class="">
Houkun<br class="">
<br class="">
<blockquote type="cite" class="">On 22. Sep 2021, at 07:21, Andreas Dilger <<a href="mailto:adilger@whamcloud.com" class="">adilger@whamcloud.com</a>> wrote:<br class="">
<br class="">
There is actually a parameter for this:<br class="">
<br class="">
$ lctl get_param ost.OSS.*.thread*<br class="">
ost.OSS.ost.threads_max=16<br class="">
ost.OSS.ost.threads_min=3<br class="">
ost.OSS.ost.threads_started=16<br class="">
ost.OSS.ost_create.threads_max=10<br class="">
ost.OSS.ost_create.threads_min=2<br class="">
ost.OSS.ost_create.threads_started=3<br class="">
ost.OSS.ost_io.threads_max=16<br class="">
ost.OSS.ost_io.threads_min=3<br class="">
ost.OSS.ost_io.threads_started=16<br class="">
ost.OSS.ost_out.threads_max=10<br class="">
ost.OSS.ost_out.threads_min=2<br class="">
ost.OSS.ost_out.threads_started=2<br class="">
ost.OSS.ost_seq.threads_max=10<br class="">
ost.OSS.ost_seq.threads_min=2<br class="">
ost.OSS.ost_seq.threads_started=2<br class="">
<br class="">
$ lctl get_param mds.MDS.*.thread*<br class="">
mds.MDS.mdt.threads_max=80<br class="">
mds.MDS.mdt.threads_min=3<br class="">
mds.MDS.mdt.threads_started=11<br class="">
mds.MDS.mdt_fld.threads_max=256<br class="">
mds.MDS.mdt_fld.threads_min=2<br class="">
mds.MDS.mdt_fld.threads_started=3<br class="">
mds.MDS.mdt_io.threads_max=80<br class="">
mds.MDS.mdt_io.threads_min=3<br class="">
mds.MDS.mdt_io.threads_started=4<br class="">
mds.MDS.mdt_out.threads_max=80<br class="">
mds.MDS.mdt_out.threads_min=2<br class="">
mds.MDS.mdt_out.threads_started=2<br class="">
mds.MDS.mdt_readpage.threads_max=56<br class="">
mds.MDS.mdt_readpage.threads_min=2<br class="">
mds.MDS.mdt_readpage.threads_started=3<br class="">
mds.MDS.mdt_seqm.threads_max=256<br class="">
mds.MDS.mdt_seqm.threads_min=2<br class="">
mds.MDS.mdt_seqm.threads_started=2<br class="">
mds.MDS.mdt_seqs.threads_max=256<br class="">
mds.MDS.mdt_seqs.threads_min=2<br class="">
mds.MDS.mdt_seqs.threads_started=2<br class="">
mds.MDS.mdt_setattr.threads_max=56<br class="">
mds.MDS.mdt_setattr.threads_min=2<br class="">
mds.MDS.mdt_setattr.threads_started=2<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">On Sep 21, 2021, at 19:21, Patrick Farrell <<a href="mailto:pfarrell@ddn.com" class="">pfarrell@ddn.com</a>> wrote:<br class="">
<br class="">
“<br class="">
<blockquote type="cite" class="">Though I can wait for the number threads to automatically decrease, I didn’t find ways which can really indicate the current running threads. I’ve tried thread_started (e.g., lctl get_param mds.MDS.mdt.threads_,started). But
this param doesn’t change. ”<br class="">
<br class="">
I don’t think Lustre exposes a stat which gives *current* count of worker threads. I’ve always used ps, grep, and wc -l to answer that question :)<br class="">
</blockquote>
From: lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" class="">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Andreas Dilger via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
Sent: Tuesday, September 21, 2021 8:03 PM<br class="">
To: Houkun Zhu <<a href="mailto:diskun.zhu@gmail.com" class="">diskun.zhu@gmail.com</a>><br class="">
Cc: <a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a> <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
Subject: Re: [lustre-discuss] Question about max service threads<br class="">
<br class="">
Hello Houkun,<br class="">
There was patch <a href="https://review.whamcloud.com/34400" class="">https://review.whamcloud.com/34400</a> "LU-947 ptlrpc: allow stopping threads above threads_max" landed for the 2.13 release. You could apply this patch to your 2.12 release, or test with
2.14.0. Note that this patch only lazily stops threads as they become idle, so there is no guarantee that they will all stop immediately when the parameter is changed. It may be some time and processed RPCs before the higher-numbered threads exit.
<br class="">
<br class="">
It might be possible to wake up all of the threads when the threads_max parameter is reduced, to have them check for this condition and exit. However, this is a very unlikely condition under normal usage.
<br class="">
<br class="">
I would recommend to test with increasing the thread count, rather than decreasing it...<br class="">
<br class="">
Cheers, Andreas<br class="">
<br class="">
<blockquote type="cite" class="">On Sep 20, 2021, at 02:29, Houkun Zhu via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:<br class="">
<br class="">
<br class="">
Hi guys,<br class="">
<br class="">
I’m creating an automatic lustre performance tuning system. But I find it’s hard to tune parameter regarding max service threads because it seems there is only guarantee of max threads when we increase the parameter. I’ve found a similar discussion from 2011,
is there any updates?<br class="">
<br class="">
Though I can wait for the number threads to automatically decrease, I didn’t find ways which can really indicate the current running threads. I’ve tried thread_started (e.g., lctl get_param mds.MDS.mdt.threads_,started). But this param doesn’t change.
<br class="">
<br class="">
Looking forward to your help! Thank you in advance!<br class="">
<br class="">
Best regards,<br class="">
Houkun<br class="">
<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</blockquote>
</blockquote>
<br class="">
Cheers, Andreas<br class="">
--<br class="">
Andreas Dilger<br class="">
Lustre Principal Architect<br class="">
Whamcloud<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</blockquote>
<br class="">
</blockquote>
<br class="">
Cheers, Andreas<br class="">
--<br class="">
Andreas Dilger<br class="">
Lustre Principal Architect<br class="">
Whamcloud<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>