<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
When the CPT code was added to LNet back in 2012, it was to address one primary case: a need for finer grained locking on metadata servers. LNet used to have global locks and metadata servers, which do many small messages (high IOPS), much time in the worker
threads was spent in spinlocks. So, CPT configuration was added so locks/resources could be allocated per CPT. This way, users have control over how they want CPTs to be configured and how they want resources/locks to be divided. For example, users may
want finer grained locking on the metadata servers but not on clients. Leaving this to be automatically configured by Linux API calls would take this flexibility away from the users who, for HPC, are very knowledgable about what they want (i.e. we do not
want to protect them from themselves).
<div class=""><br class="">
</div>
<div class="">The CPT support in LNet and LNDs has morphed to encompass more traditional NUMA and core affinity performance improvements. For example, you can restrict a network interface to a socket (NUMA node) which has better affinity to the PCIe lanes
that interface is connected to. Rather than try to do this sort of thing automatically, we have left it to the user to know what they are doing and configure the CPTs accordingly. </div>
<div class=""><br class="">
</div>
<div class="">I think the many changes to the CPT code has realty clouded its purpose. In summary, the original purpose was finer grained locking and that needs to be maintained as the IOPS requirements of metadata servers is paramount.</div>
<div class=""><br class="">
</div>
<div class="">James: The Verbs RDMA interface has very poor support for NUMA/core affinity. I was going to try to devise some patches to address that but have been too busy on other things. Perhaps the RDMA maintainer could consider updating it?</div>
<div class=""><br class="">
</div>
<div class="">Doug</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jul 5, 2018, at 8:11 PM, NeilBrown <<a href="mailto:neilb@suse.com" class="">neilb@suse.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On
Fri, Jul 06 2018, James Simmons wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" class="">NeilBrown [<a href="mailto:neilb@suse.com" class="">mailto:neilb@suse.com</a>] wrote:<br class="">
<br class="">
To help contextualize things: the Lustre code can be decomposed into three parts:<br class="">
<br class="">
1) The filesystem proper: Lustre.<br class="">
2) The communication protocol it uses: LNet.<br class="">
3) Supporting code used by Lustre and LNet: CFS.<br class="">
<br class="">
Part of the supporting code is the CPT mechanism, which provides a way to<br class="">
partition the CPUs of a system. These partitions are used to distribute queues,<br class="">
locks, and threads across the system. It was originally introduced years ago, as<br class="">
far as I can tell mainly to deal with certain hot locks: these were converted into<br class="">
read/write locks with one spinlock per CPT.<br class="">
<br class="">
As a general rule, CPT boundaries should respect node and socket boundaries,<br class="">
but at the higher end, where CPUs have 20+ cores, it may make sense to split<br class="">
a CPUs cores across several CPTs.<br class="">
<br class="">
<blockquote type="cite" class="">Thanks everyone for your patience in explaining things to me.<br class="">
I'm beginning to understand what to look for and where to find it.<br class="">
<br class="">
So the answers to Greg's questions:<br class="">
<br class="">
Where are you reading the host memory NUMA information from?<br class="">
<br class="">
And why would a filesystem care about this type of thing? Are you<br class="">
going to now mirror what the scheduler does with regards to NUMA<br class="">
topology issues? How are you going to handle things when the topology<br class="">
changes? What systems did you test this on? What performance<br class="">
improvements were seen? What downsides are there with all of this?<br class="">
<br class="">
<br class="">
Are:<br class="">
</blockquote>
<br class="">
<blockquote type="cite" class=""> - NUMA info comes from ACPI or device-tree just like for every one<br class="">
else. Lustre just uses node_distance().<br class="">
</blockquote>
<br class="">
Correct, the standard kernel interfaces for this information are used to<br class="">
obtain it, so ultimately Lustre/LNet uses the same source of truth as<br class="">
everyone else.<br class="">
<br class="">
<blockquote type="cite" class=""> - The filesystem cares about this because... It has service<br class="">
thread that does part of the work of some filesystem operations<br class="">
(handling replies for example) and these are best handled "near"<br class="">
the CPU the initiated the request. Lustre partitions<br class="">
all CPUs into "partitions" (cpt) each with a few cores.<br class="">
If the request thread and the reply thread are on different<br class="">
CPUs but in the same partition, then we get best throughput<br class="">
(is that close?)<br class="">
</blockquote>
<br class="">
At the filesystem level, it does indeed seem to help to have the service<br class="">
threads that do work for requests run on a different core that is close to<br class="">
the core that originated the request. So preferably on the same CPU, and<br class="">
on certain multi-core CPUs there are also distance effects between cores.<br class="">
That too is one of the things the CPT mechanism handles.<br class="">
</blockquote>
<br class="">
Their is another very important aspect to why Lustre has a CPU partition<span class="Apple-converted-space"> </span><br class="">
layer. At least at the place I work at. While the Linux kernel manages all<br class="">
the NUMA nodes and CPU cores Lustre adds the ability for us to specify a<span class="Apple-converted-space"> </span><br class="">
subset of everything on the system. The reason is to limit the impact of<br class="">
noise on the compute nodes. Noise has a heavy impact on large scale HP<br class="">
work loads that can run days or even weeks at a time. Lets take an<span class="Apple-converted-space"> </span><br class="">
example system:<br class="">
<br class="">
|-------------| |-------------|<br class="">
|-------| | NUMA 0 | | NUMA 1 | |-------|<br class="">
| eth0 | - | | --- | | - | eth1 | <br class="">
|_______| | CPU0 CPU1 | | CPU2 CPU3 | |_______|<br class="">
|_____________| |_____________|<br class="">
<br class="">
In such a system it is possible with the right job schedular to start a<span class="Apple-converted-space"> </span><br class="">
large parallel application on NUMA 0/ (CPU0 and CPU1). Normally such<br class="">
large parallel applications will communicate between nodes using MPI,<br class="">
such as openmpi, which can be configured to use eth0 only. Using the<br class="">
CPT layer in lustre we can isolate lustre to NUMA 1 and use only eth1.<br class="">
This greatly reducess the noise impact on the application running.<br class="">
<br class="">
BTW this is one of the reasons ko2iblnd for lustre doesn't use the<br class="">
generic RDMA api. The core IB layer doesn't support such isolation.<br class="">
At least to my knowledge.<br class="">
</blockquote>
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Thanks
for that background (and for the separate explanation of how</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">jitter
multiplies when jobs needs to synchronize periodically).</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I
can see that setting CPU affinity for lustre/lnet worker threads could</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">be
important, and that it can be valuable to tie services to a</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">particular
interface. I cannot yet see why we need partitions for this,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">rather
that doing it at the CPU (or NODE) level.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Thanks,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">NeilBrown</span></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>