<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
There is actually a parameter for this:
<div class=""><br class="">
</div>
<div class="">$ lctl get_param ost.OSS.*.thread*<br class="">
ost.OSS.ost.threads_max=16<br class="">
ost.OSS.ost.threads_min=3<br class="">
ost.OSS.ost.threads_started=16<br class="">
ost.OSS.ost_create.threads_max=10<br class="">
ost.OSS.ost_create.threads_min=2<br class="">
ost.OSS.ost_create.threads_started=3<br class="">
ost.OSS.ost_io.threads_max=16<br class="">
ost.OSS.ost_io.threads_min=3<br class="">
ost.OSS.ost_io.threads_started=16<br class="">
ost.OSS.ost_out.threads_max=10<br class="">
ost.OSS.ost_out.threads_min=2<br class="">
ost.OSS.ost_out.threads_started=2<br class="">
ost.OSS.ost_seq.threads_max=10<br class="">
ost.OSS.ost_seq.threads_min=2<br class="">
ost.OSS.ost_seq.threads_started=2</div>
<div class=""><br class="">
$ lctl get_param mds.MDS.*.thread*<br class="">
mds.MDS.mdt.threads_max=80<br class="">
mds.MDS.mdt.threads_min=3<br class="">
mds.MDS.mdt.threads_started=11<br class="">
mds.MDS.mdt_fld.threads_max=256<br class="">
mds.MDS.mdt_fld.threads_min=2<br class="">
mds.MDS.mdt_fld.threads_started=3<br class="">
mds.MDS.mdt_io.threads_max=80<br class="">
mds.MDS.mdt_io.threads_min=3<br class="">
mds.MDS.mdt_io.threads_started=4<br class="">
mds.MDS.mdt_out.threads_max=80<br class="">
mds.MDS.mdt_out.threads_min=2<br class="">
mds.MDS.mdt_out.threads_started=2<br class="">
mds.MDS.mdt_readpage.threads_max=56<br class="">
mds.MDS.mdt_readpage.threads_min=2<br class="">
mds.MDS.mdt_readpage.threads_started=3<br class="">
mds.MDS.mdt_seqm.threads_max=256<br class="">
mds.MDS.mdt_seqm.threads_min=2<br class="">
mds.MDS.mdt_seqm.threads_started=2<br class="">
mds.MDS.mdt_seqs.threads_max=256<br class="">
mds.MDS.mdt_seqs.threads_min=2<br class="">
mds.MDS.mdt_seqs.threads_started=2<br class="">
mds.MDS.mdt_setattr.threads_max=56<br class="">
mds.MDS.mdt_setattr.threads_min=2<br class="">
mds.MDS.mdt_setattr.threads_started=2<br class="">
<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Sep 21, 2021, at 19:21, Patrick Farrell <<a href="mailto:pfarrell@ddn.com" class="">pfarrell@ddn.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<div class="">“
<blockquote type="cite" style="box-sizing: border-box; caret-color: rgb(33, 33, 33); color: rgb(33, 33, 33); font-family: "Segoe UI WestEuropean", "Segoe UI", Helvetica, Arial, sans-serif; font-size: 15px" class="">
<div dir="ltr" style="box-sizing: border-box; margin: 0px" class="">
<div style="box-sizing: border-box; margin: 0px; line-break: after-white-space" class="">
<div style="box-sizing: border-box; margin: 0px" class="">Though I can wait for the number threads to automatically decrease, I didn’t find ways which can really indicate the current running threads. I’ve tried thread_started (e.g., lctl get_param mds.MDS.mdt.threads_,started).
But this param doesn’t change. ”</div>
<div style="box-sizing: border-box; margin: 0px" class=""><br class="">
</div>
<div style="box-sizing: border-box; margin: 0px" class="">I don’t think Lustre exposes a stat which gives *current* count of worker threads. I’ve always used ps, grep, and wc -l to answer that question :)</div>
</div>
</div>
</blockquote>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1" class="">
<div id="divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size:11pt" class=""><b class="">From:</b> lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" class="">lustre-discuss-bounces@lists.lustre.org</a>>
on behalf of Andreas Dilger via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Sent:</b> Tuesday, September 21, 2021 8:03 PM<br class="">
<b class="">To:</b> Houkun Zhu <<a href="mailto:diskun.zhu@gmail.com" class="">diskun.zhu@gmail.com</a>><br class="">
<b class="">Cc:</b> <a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a> <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:</b> Re: [lustre-discuss] Question about max service threads</font>
<div class=""> </div>
</div>
<div dir="auto" class=""><span style="background-color:rgb(255,255,255)" class="">Hello Houkun,</span>
<div style="" class="">There was patch <a href="https://review.whamcloud.com/34400" class="">https://review.whamcloud.com/34400</a> "<a href="https://jira.whamcloud.com/browse/LU-947" style="color:rgb(6,84,172); text-decoration:none; font-family:monospace; white-space:pre" class="">LU-947</a> ptlrpc:
allow stopping threads above threads_max" landed for the 2.13 release. You could apply this patch to your 2.12 release, or test with 2.14.0. Note that this patch only lazily stops threads as they become idle, so there is no guarantee that they will all stop
immediately when the parameter is changed. It may be some time and processed RPCs before the higher-numbered threads exit. </div>
<div style="" class=""><br class="">
<div dir="ltr" class="">It might be possible to wake up all of the threads when the threads_max parameter is reduced, to have them check for this condition and exit. However, this is a very unlikely condition under normal usage. </div>
<div dir="ltr" class=""><br class="">
</div>
<div dir="ltr" class="">I would recommend to test with increasing the thread count, rather than decreasing it...</div>
</div>
<br class="">
<div dir="ltr" class="">Cheers, Andreas</div>
<div dir="ltr" class=""><br class="">
<blockquote type="cite" class="">On Sep 20, 2021, at 02:29, Houkun Zhu via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:<br class="">
<br class="">
</blockquote>
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="" style="word-wrap:break-word; line-break:after-white-space">Hi guys,
<div class=""><br class="">
</div>
<div class="">I’m creating an automatic lustre performance tuning system. But I find it’s hard to tune parameter regarding max service threads because it seems there is only guarantee of max threads when we increase the parameter. I’ve found
<a href="http://lists.lustre.org/htdig.cgi/lustre-discuss-lustre.org/2011-August/010358.html" class="">
a similar discussion</a> from 2011, is there any updates?</div>
<div class=""><br class="">
</div>
<div class="">Though I can wait for the number threads to automatically decrease, I didn’t find ways which can really indicate the current running threads. I’ve tried thread_started (e.g., lctl get_param mds.MDS.mdt.threads_,started). But this param doesn’t
change. </div>
<div class=""><br class="">
</div>
<div class="">Looking forward to your help! Thank you in advance!</div>
<div class=""><br class="">
<div class="">
<div class="">Best regards,</div>
<div class="">Houkun</div>
</div>
<br class="">
</div>
</div>
<span class="">_______________________________________________</span><br class="">
<span class="">lustre-discuss mailing list</span><br class="">
<span class=""><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a></span><br class="">
<span class=""><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span><br class="">
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>