<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<br>
<blockquote type="cite"
cite="mid:FC50B8EA-DDBC-4C93-B843-E048177C5176@whamcloud.com">
<div>
The number of ptlrpc threads per CPT is set by the
"ptlrpcd_partner_group_size" module parameter, and defaults to 2
threads per CPT, IIRC. I don't think that clients dynamically
start/stop ptlrpcd threads at runtime.</div>
</blockquote>
<blockquote type="cite"
cite="mid:FC50B8EA-DDBC-4C93-B843-E048177C5176@whamcloud.com">
<div>
When there are RPCs in the queue for any ptlrpcd it will be
woken up and scheduled by the kernel, so it will compete with
the application threads. IIRC, if a ptlrpcd thread is woken up
and there are no RPCs in the local CPT queue it will try to
steal RPCs from another CPT on the assumption that the local CPU
is not generating any RPCs so it would be beneficial to offload
threads on another CPU that *is* generating RPCs. If the
application thread is extremely CPU hungry, then the kernel will
not schedule the ptlrpcd threads on those codes very often, and
the "idle" core ptlrpcd threads will be be able to run more
frequently.</div>
</blockquote>
<br>
Sorry, maybe I am confusing things. I am still not sure how many
threads I get.<br>
For example I have a 32 cores AMD Epyc machine as a client and I am
running a serial stream io application with a single stripesize, 1
OST. <br>
I am struggeling to find out how many CPU partitions I have - is it
something on the hardware side or something configurable?<br>
There is no file /proc/sys/lnet/cpu_partitions on my client. <br>
<br>
Assuming I had 2 CPU partitions, that would result in 4 ptlrpc
threads at system start, right? Now I set rpcs_in_flight to 1 or to
8, what effect does that have on the number and the activity of the
threads?<br>
Serial stream, 1 rpcs_in_flight is waking up only one ptlrpc thread,
3 remain inactive/sleep/do nothing?<br>
<br>
Does not seem to be the case, as I've applied the rpctracing (thanks
a lot for the hint!!), and rpcs_in_flight being 1 still show at
least 3 different threads from at least 2 different partitions for
writing a 1MB file with ten blocks. <br>
I don't get the relationship between these values.<br>
<br>
And, if I had compression or any other heavy load, which settings
could clearly control how many resources I want to give Lustre for
this load? I can see a clear scaling with higher rpcs in flight, but
I am struggeling to understand the numbers and attribute them to a
specific settings. Uncompressed case already benefits a bit by
higher RPCs number due to multiple "substreaming", but there must be
much more happening in parallel behind the scenes for compressed
case even with rpcs_in_flight=1.<br>
<br>
Thank you!<br>
<br>
Anna<br>
<br>
<blockquote type="cite"
cite="mid:FC50B8EA-DDBC-4C93-B843-E048177C5176@whamcloud.com">
<div><br class="">
</div>
<div>Whether this behavior is optimal or not is subject to debate,
and investigation/improvements are of course welcome.
Definitely, data checksums have some overhead (a few percent),
and client-side data compression (which is done by ptlrpcd
threads) would have a significant usage of CPU cycles, but given
the large number of CPU cores on client nodes these days this
may still provide a net performance benefit if the IO bottleneck
is on the server.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">
<blockquote type="cite"
cite="mid:ABC8B857-9356-4F6F-B9F2-3E8AC2A21EF5@whamcloud.com" class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><font class="" face="monospace">With
<code class="">max_</code><code class="">rpcs_in_flight
= 1</code>, multiple cores are loaded,
presumably alternately, but the statistics are
too inaccurate to capture this. The
distribution of threads to cores is regulated
by the Linux kernel, right? Does anyone have
experience with what happens when all CPUs are
under full load with the application or
something else?
<br class="">
</font></div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Note that {osc,mdc}.*.max_rpcs_in_flight is a *per
target* parameter, so a single client can still have
tens or hundreds of RPCs in flight to different
servers. The client will send many RPC types directly
from the process context, since they are waiting on
the result anyway. For asynchronous bulk RPCs, the
ptlrpcd thread will try to process the bulk IO on the
same CPT (= Lustre CPU Partition Table, roughly
aligned to NUMA nodes) as the userspace application
was running when the request was created. This
minimizes the cross-NUMA traffic when accessing pages
for bulk RPCs, so long as those cores are not busy
with userspace tasks. Otherwise, the ptlrpcd thread
on another CPT will steal RPCs from the queues. </div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><font class="" face="monospace">Do
the Lustre threads suffer? Is there a
prioritization of the Lustre threads over
other tasks?<br class="">
</font></div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Are you asking about the client or the server? Many
of the client RPCs are generated by the client
threads, but for the running ptlrpcd threads do not
have a higher priority than client application
threads. If the application threads are running on
some cores, but other cores are idle, then the ptlrpcd
threads on other cores will try to process the RPCs to
allow the application threads to continue running
there. Otherwise, if all cores are busy (as is
typical for HPC applications) then they will be
scheduled by the kernel as needed.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><font class="" face="monospace">Are
there readily available statistics or tools
for this scenario?<br class="">
</font></div>
</div>
</blockquote>
<br class="">
</div>
<div class="">What statistics are you looking for?
There are "{osc,mdc}.*.stats" and
"{osc,mdc}.*rpc_stats" that have aggregate information
about RPC counts and latency.</div>
</blockquote>
</div>
</div>
</blockquote>
<blockquote type="cite" class=""><br class="">
</blockquote>
<blockquote type="cite" class="">
<div class="">
<div class="">
<blockquote type="cite"
cite="mid:ABC8B857-9356-4F6F-B9F2-3E8AC2A21EF5@whamcloud.com" class="">
</blockquote>
Oh, right, these tell a lot. Isn't there also something to
log the utilization and location of these threads?
Otherwise, I'll continue trying with perf, which seems to
be more complex with kernel threads.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
There are kernel debug logs available when "lctl set_param
debug=+rpctrace" is enabled, that will show which ptlrpcd or
application thread is handling each RPC, and on which core it
was run on. These can be found on the client by searching for
"Sending RPC|Completed RPC" in the debug logs, for example:</div>
<div><br class="">
</div>
<div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""># lctl set_param debug=+rpctrace</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""># lctl set_param jobid_var=procname_uid</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""># cp -a /etc /mnt/testfs</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""># lctl dk /tmp/debug</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""># grep -E "Sending RPC|Completed RPC" /tmp/debug</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""> :</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""> :</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">00000100:00100000:2.0:1714502851.435000:0:23892:0:(client.c:1758:ptlrpc_send_new_req())</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""> Sending RPC req@ffff90c9b2948640
pname:cluuid:pid:xid:nid:opc:job</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">
<a class="moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634353438336:0@lo:2:cp.0">ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634353438336:0@lo:2:cp.0</a></span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">00000100:00100000:2.0:1714502851.436117:0:23892:0:(client.c:2239:ptlrpc_check_set())</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""> Completed RPC req@ffff90c9b2948640
pname:cluuid:pid:xid:nid:opc:job</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">
<a class="moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634353438336:0@lo:2:cp.0">ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634353438336:0@lo:2:cp.0</a></span></div>
</div>
<div><br class="">
</div>
Shows that thread "ptlrpcd_01_00" (CPT 01, thread 00, pid 23892)
was sent on core 2.0 (no hyperthread) and sent an OST_SETATTR (opc
= 2) RPC on behalf of "cp" for root (uid=0), and competed in
1117msec.
<div class=""><br class="">
</div>
<div class="">Similarly, with a "dd" sync write workload it shows
write RPCs by the ptlrpcd threads, and sync RPCs in the "dd"
process context:</div>
<div class="">
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""># dd if=/dev/zero of=/mnt/testfs/file bs=4k
count=10000 oflag=dsync</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">#</span> lctl dk /tmp/debug</div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures;"
class=""># grep -E "Sending RPC|Completed RPC" /tmp/debug</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures;"
class=""> :</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures;"
class=""> :</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">00000100:00100000:2.0:1714503761.136971:0:23892:0:(client.c:1758:ptlrpc_send_new_req())</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""> Sending RPC req@ffff90c9a6ad6640
pname:cluuid:pid:xid:nid:opc:job</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">
<a class="moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634358961024:0@lo:4:dd.0">ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634358961024:0@lo:4:dd.0</a></span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">00000100:00100000:2.0:1714503761.140288:0:23892:0:(client.c:2239:ptlrpc_check_set())</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""> Completed RPC req@ffff90c9a6ad6640
pname:cluuid:pid:xid:nid:opc:job</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">
<a class="moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634358961024:0@lo:4:dd.0">ptlrpcd_01_00:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23892:1797634358961024:0@lo:4:dd.0</a></span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">00000100:00100000:2.0:1714503761.140518:0:17993:0:(client.c:1758:ptlrpc_send_new_req())</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""> Sending RPC req@ffff90c9a6ad3040
pname:cluuid:pid:xid:nid:opc:job</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">
<a class="moz-txt-link-abbreviated" href="mailto:dd:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:17993:1797634358961088:0@lo:44:dd.0">dd:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:17993:1797634358961088:0@lo:44:dd.0</a></span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">00000100:00100000:2.0:1714503761.141556:0:17993:0:(client.c:2239:ptlrpc_check_set())</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""> Completed RPC req@ffff90c9a6ad3040
pname:cluuid:pid:xid:nid:opc:job</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">
<a class="moz-txt-link-abbreviated" href="mailto:dd:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:17993:1797634358961088:0@lo:44:dd.0">dd:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:17993:1797634358961088:0@lo:44:dd.0</a></span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">00000100:00100000:2.0:1714503761.141885:0:23893:0:(client.c:1758:ptlrpc_send_new_req())</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""> Sending RPC req@ffff90c9a6ad3040
pname:cluuid:pid:xid:nid:opc:job</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">
<a class="moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_01:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23893:1797634358961152:0@lo:16:dd.0">ptlrpcd_01_01:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23893:1797634358961152:0@lo:16:dd.0</a></span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">00000100:00100000:2.0:1714503761.144172:0:23893:0:(client.c:2239:ptlrpc_check_set())</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class=""> Completed RPC req@ffff90c9a6ad3040
pname:cluuid:pid:xid:nid:opc:job</span></div>
<div
style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgba(242, 242, 242, 0.901961); background-color: rgba(0, 0, 0, 0.862745);"
class="">
<span style="font-variant-ligatures: no-common-ligatures"
class="">
<a class="moz-txt-link-abbreviated" href="mailto:ptlrpcd_01_01:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23893:1797634358961152:0@lo:16:dd.0">ptlrpcd_01_01:e81f3122-b1bc-4ac4-afcb-f6629a81e5bd:23893:1797634358961152:0@lo:16:dd.0</a></span></div>
<div class=""><br class="">
</div>
<div class="">There are no stats files that aggregate
information about ptlrpcd thread utilization. <br class="">
<div class=""><br class="">
<div class="">
<div dir="auto"
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"
class="">
<div dir="auto"
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"
class="">
<div dir="auto"
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"
class="">
<div dir="auto"
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"
class="">
<div dir="auto"
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"
class="">
<div dir="auto"
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"
class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>