<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> This is a module parameter, since it cannot be changed at runtime. This is visible at /sys/module/libcfs/parameters/cpu_npartitions and the default value depends on the number of CPU cores and NUMA configuration. It can be specified with "options libcfs
cpu_npartitions=" in /etc/modprobe.d/lustre.conf.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> The "cpu_npartitions" module parameter controls how many groups the cores are split into. The "cpu_pattern" parameter can control the specific cores in each of the CPTs, which would affect the default per-CPT ptlrpcd threads location. It is possible to further
use the "ptlrpcd_cpts" and "ptlrpcd_per_cpt_max" parameters to control specifically which cores are used for the threads.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Just a comment on the tuning parameters could be tricky.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
"cpu_npartitions" is ignored at the profit of "cpu_pattern", except if cpu_pattern is the empty string. cpu_pattern can achieve the same results as cpu_npartitions, but at the cost of a more complex declaration. If you just want to split your cores into multiple
subgroups, you can use cpu_npartition.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
options libcfs cpu_pattern="" cpu_npartition=8 # You need to set the empty string for cpu_pattern to avoid cpu_npartition to be ignored</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
or</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
options libcfs cpu_pattern="N" # the default, split on the NUMA groups</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
or</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
options libcfs cpu_pattern="0[0-3] 1[4-7] 2[8-11] 3[12-15] 4[16-19] 5[21-23] 6[24-27] 7[28-31]" # same as cpu_npartition=8</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
or even more complex distribution, see Lustre Manual for details.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Also check "lctl get_param cpu_partition_table" to see your current partition table.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Aurélien<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>De :</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> de la part de Andreas Dilger via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Envoyé :</b> vendredi 3 mai 2024 07:25<br>
<b>À :</b> Anna Fuchs <anna.fuchs@uni-hamburg.de><br>
<b>Cc :</b> lustre <lustre-discuss@lists.lustre.org><br>
<b>Objet :</b> Re: [lustre-discuss] kernel threads for rpcs in flight</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<table bgcolor="#FFEB9C" border="1">
<tbody>
<tr>
<td><font face="verdana" color="black" size="1"><b>External email: Use caution opening links or attachments</b>
</font></td>
</tr>
</tbody>
</table>
<br>
<div>On May 2, 2024, at 18:10, Anna Fuchs <<a href="mailto:anna.fuchs@uni-hamburg.de" class="">anna.fuchs@uni-hamburg.de</a>> wrote:
<div>
<blockquote type="cite" class="">
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">The number of ptlrpc threads per CPT is set by the "ptlrpcd_partner_group_size" module parameter, and defaults to 2 threads per CPT, IIRC. I don't think that clients dynamically start/stop ptlrpcd threads at runtime.</div>
</blockquote>
<blockquote type="cite" class="">
<div class="">When there are RPCs in the queue for any ptlrpcd it will be woken up and scheduled by the kernel, so it will compete with the application threads. IIRC, if a ptlrpcd thread is woken up and there are no RPCs in the local CPT queue it will try
to steal RPCs from another CPT on the assumption that the local CPU is not generating any RPCs so it would be beneficial to offload threads on another CPU that *is* generating RPCs. If the application thread is extremely CPU hungry, then the kernel will not
schedule the ptlrpcd threads on those codes very often, and the "idle" core ptlrpcd threads will be be able to run more frequently.</div>
</blockquote>
<br class="">
Sorry, maybe I am confusing things. I am still not sure how many threads I get.<br class="">
For example I have a 32 cores AMD Epyc machine as a client and I am running a serial stream io application with a single stripesize, 1 OST.
<br class="">
I am struggeling to find out how many CPU partitions I have - is it something on the hardware side or something configurable?<br class="">
There is no file /proc/sys/lnet/cpu_partitions on my client. <br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
This is a module parameter, since it cannot be changed at runtime. This is visible at /sys/module/libcfs/parameters/cpu_npartitions and the default value depends on the number of CPU cores and NUMA configuration. It can be specified with "options libcfs cpu_npartitions="
in /etc/modprobe.d/lustre.conf.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">Assuming I had 2 CPU partitions, that would result in 4 ptlrpc threads at system start, right?</div>
</div>
</blockquote>
<div><br class="">
</div>
Correct.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">Now I set rpcs_in_flight to 1 or to 8, what effect does that have on the number and the activity of the threads?<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
Setting rpcs_in_flight has no effect on the number of ptlrpcd threads. The ptlrpcd threads process RPCs asynchronously (unlike server threads) so they can keep many RPCs in progress. </div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">Serial stream, 1 rpcs_in_flight is waking up only one ptlrpc thread, 3 remain inactive/sleep/do nothing?<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
This depends. There are two ptlrpcd threads for the CPT that can process the RPCs from the one user thread. If they can send the RPCs quickly enough then the other ptlrpcd threads may not steal the RPCs from that CPT.</div>
<div><br class="">
</div>
<div>That said, even a single threaded userspace writer may have up to 8 RPCs in flight *per OST* (depending on the file striping and if IO submission allows it - buffered or AIO+DIO) so if there are a lot of outstanding RPCs and RPC generation takes a long
time (e.g. compression) then it may be that all ptlrpcd threads will be busy.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">Does not seem to be the case, as I've applied the rpctracing (thanks a lot for the hint!!), and rpcs_in_flight being 1 still show at least 3 different threads from at least 2 different partitions for writing a 1MB file with ten blocks.
<br class="">
I don't get the relationship between these values.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
What are the opcodes from the different RPCs? The ptlrpcd threads are only handling asynchronous RPCs like buffered writes, statfs, and a few others. Many RPCs are processed in the context of the application thread, not by ptlrpcd.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">And, if I had compression or any other heavy load, which settings could clearly control how many resources I want to give Lustre for this load? I can see a clear scaling with higher rpcs in flight, but I am struggeling to understand the numbers
and attribute them to a specific settings. Uncompressed case already benefits a bit by higher RPCs number due to multiple "substreaming", but there must be much more happening in parallel behind the scenes for compressed case even with rpcs_in_flight=1.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
The "cpu_npartitions" module parameter controls how many groups the cores are split into. The "cpu_pattern" parameter can control the specific cores in each of the CPTs, which would affect the default per-CPT ptlrpcd threads location. It is possible to further
use the "ptlrpcd_cpts" and "ptlrpcd_per_cpt_max" parameters to control specifically which cores are used for the threads.</div>
<div><br class="">
</div>
<div>It is entirely possible that the number of ptlrpcd threads and CPT configuration is becoming sub-optimal as the number of multi-chip package CPUs with many cores grows dramatically. It is a balance between having enough threads to maximize performance
without having so many that it goes down hill again. Ideally this should all happen without the need to hand-tune the CPT and thread count for every CPU on the market.</div>
<div><br class="">
</div>
<div>Cheers, Andreas</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">Thank you!<br class="">
<br class="">
Anna<br class="">
<br class="">
<blockquote type="cite" class="">
<div class=""><br class="">
</div>
<div class="">Whether this behavior is optimal or not is subject to debate, and investigation/improvements are of course welcome. Definitely, data checksums have some overhead (a few percent), and client-side data compression (which is done by ptlrpcd threads)
would have a significant usage of CPU cycles, but given the large number of CPU cores on client nodes these days this may still provide a net performance benefit if the IO bottleneck is on the server.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><font class="" face="monospace">With <code class="">max_</code><code class="">rpcs_in_flight = 1</code>, multiple cores are loaded, presumably alternately, but the statistics are too inaccurate to capture this. The distribution of threads to
cores is regulated by the Linux kernel, right? Does anyone have experience with what happens when all CPUs are under full load with the application or something else?
<br class="">
</font></div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Note that {osc,mdc}.*.max_rpcs_in_flight is a *per target* parameter, so a single client can still have tens or hundreds of RPCs in flight to different servers. The client will send many RPC types directly from the process context, since they are waiting on
the result anyway. For asynchronous bulk RPCs, the ptlrpcd thread will try to process the bulk IO on the same CPT (= Lustre CPU Partition Table, roughly aligned to NUMA nodes) as the userspace application was running when the request was created. This minimizes
the cross-NUMA traffic when accessing pages for bulk RPCs, so long as those cores are not busy with userspace tasks. Otherwise, the ptlrpcd thread on another CPT will steal RPCs from the queues. </div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><font class="" face="monospace">Do the Lustre threads suffer? Is there a prioritization of the Lustre threads over other tasks?<br class="">
</font></div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Are you asking about the client or the server? Many of the client RPCs are generated by the client threads, but for the running ptlrpcd threads do not have a higher priority than client application threads. If the application threads are running on some cores,
but other cores are idle, then the ptlrpcd threads on other cores will try to process the RPCs to allow the application threads to continue running there. Otherwise, if all cores are busy (as is typical for HPC applications) then they will be scheduled by
the kernel as needed.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><font class="" face="monospace">Are there readily available statistics or tools for this scenario?<br class="">
</font></div>
</div>
</blockquote>
<br class="">
</div>
<div class="">What statistics are you looking for? There are "{osc,mdc}.*.stats" and "{osc,mdc}.*rpc_stats" that have aggregate information about RPC counts and latency.</div>
</blockquote>
</div>
</div>
</blockquote>
<blockquote type="cite" class=""><br class="">
</blockquote>
<blockquote type="cite" class="">
<div class="">
<div class="">
<blockquote type="cite" class=""></blockquote>
Oh, right, these tell a lot. Isn't there also something to log the utilization and location of these threads? Otherwise, I'll continue trying with perf, which seems to be more complex with kernel threads.<br class="">
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
There are kernel debug logs available when "lctl set_param debug=+rpctrace" is enabled, that will show which ptlrpcd or application thread is handling each RPC, and on which core it was run on. These can be found on the client by searching for "Sending RPC|Completed
RPC" in the debug logs, for example:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"># lctl set_param debug=+rpctrace</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"># lctl set_param jobid_var=procname_uid</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"># cp -a /etc /mnt/testfs</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"># lctl dk /tmp/debug</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"># grep -E "Sending RPC|Completed RPC" /tmp/debug</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> :</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> :</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures">00000100:00100000:2.0:1714502851.435000:0:23892:0:(client.c:1758:ptlrpc_send_new_req())</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> Sending RPC req@ffff90c9b2948640 pname:cluuid:pid:xid:nid:opc:job</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> <a class="x_moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634353438336:0@lo:2:cp.0">ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634353438336:0@lo:2:cp.0</a></span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures">00000100:00100000:2.0:1714502851.436117:0:23892:0:(client.c:2239:ptlrpc_check_set())</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> Completed RPC req@ffff90c9b2948640 pname:cluuid:pid:xid:nid:opc:job</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> <a class="x_moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634353438336:0@lo:2:cp.0">ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634353438336:0@lo:2:cp.0</a></span></div>
</div>
<div class=""><br class="">
</div>
Shows that thread "ptlrpcd_01_00" (CPT 01, thread 00, pid 23892) was sent on core 2.0 (no hyperthread) and sent an OST_SETATTR (opc = 2) RPC on behalf of "cp" for root (uid=0), and competed in 1117msec.
<div class=""><br class="">
</div>
<div class="">Similarly, with a "dd" sync write workload it shows write RPCs by the ptlrpcd threads, and sync RPCs in the "dd" process context:</div>
<div class="">
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"># dd if=/dev/zero of=/mnt/testfs/file bs=4k count=10000 oflag=dsync</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures">#</span> lctl dk /tmp/debug</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"># grep -E "Sending RPC|Completed RPC" /tmp/debug</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> :</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> :</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures">00000100:00100000:2.0:1714503761.136971:0:23892:0:(client.c:1758:ptlrpc_send_new_req())</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> Sending RPC req@ffff90c9a6ad6640 pname:cluuid:pid:xid:nid:opc:job</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> <a class="x_moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634358961024:0@lo:4:dd.0">ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634358961024:0@lo:4:dd.0</a></span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures">00000100:00100000:2.0:1714503761.140288:0:23892:0:(client.c:2239:ptlrpc_check_set())</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> Completed RPC req@ffff90c9a6ad6640 pname:cluuid:pid:xid:nid:opc:job</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> <a class="x_moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634358961024:0@lo:4:dd.0">ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634358961024:0@lo:4:dd.0</a></span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures">00000100:00100000:2.0:1714503761.140518:0:17993:0:(client.c:1758:ptlrpc_send_new_req())</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> Sending RPC req@ffff90c9a6ad3040 pname:cluuid:pid:xid:nid:opc:job</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> <a class="x_moz-txt-link-abbreviated" href="mailto:dd:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:17993:1797634358961088:0@lo:44:dd.0">dd:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:17993:1797634358961088:0@lo:44:dd.0</a></span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures">00000100:00100000:2.0:1714503761.141556:0:17993:0:(client.c:2239:ptlrpc_check_set())</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> Completed RPC req@ffff90c9a6ad3040 pname:cluuid:pid:xid:nid:opc:job</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> <a class="x_moz-txt-link-abbreviated" href="mailto:dd:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:17993:1797634358961088:0@lo:44:dd.0">dd:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:17993:1797634358961088:0@lo:44:dd.0</a></span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures">00000100:00100000:2.0:1714503761.141885:0:23893:0:(client.c:1758:ptlrpc_send_new_req())</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> Sending RPC req@ffff90c9a6ad3040 pname:cluuid:pid:xid:nid:opc:job</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> <a class="x_moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_01:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23893:1797634358961152:0@lo:16:dd.0">ptlrpcd_01_01:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23893:1797634358961152:0@lo:16:dd.0</a></span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures">00000100:00100000:2.0:1714503761.144172:0:23893:0:(client.c:2239:ptlrpc_check_set())</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> Completed RPC req@ffff90c9a6ad3040 pname:cluuid:pid:xid:nid:opc:job</span></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal; font-family:Monaco; color:rgba(242,242,242,0.901961); background-color:rgba(0,0,0,0.862745)">
<span class="" style="font-variant-ligatures:no-common-ligatures"> <a class="x_moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_01:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23893:1797634358961152:0@lo:16:dd.0">ptlrpcd_01_01:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23893:1797634358961152:0@lo:16:dd.0</a></span></div>
<div class=""><br class="">
</div>
<div class="">There are no stats files that aggregate information about ptlrpcd thread utilization. <br class="">
<div class=""><br class="">
<div class="">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div class="">Cheers, Andreas</div>
<div class="">--</div>
<div class="">Andreas Dilger</div>
<div class="">Lustre Principal Architect</div>
<div class="">Whamcloud</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
<br class="x_Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
<br class="x_Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
</body>
</html>