<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<br>
A tiny bit more about noise for Neil, since its a bit subtle and I had never heard of it before working in HPC. Sorry if this is old news.<br>
<br>
Noise here means differences in execution time. A typical HPC Job consists of thousands of processes running across a large system. The basic model is they all run a compute step, then they all communicate part of their results (generally to some neighboring
subset of processes, not all-to-all). The results which are communicated are then used as part of the input to the next compute step. As you can see, effectively, everyone must finish each step before anyone can continue (or at least, continue very far).<br>
<br>
So if everyone finishes every step in the same amount of time, great. But if theres jitter in the completion time for a step for a particular process - as can be introduced by a scheduler with ideas that dont quite line up with your job priorities - it delays
the completion of the step overall. This is compounded at each step of the job and so can be quite serious. (Job steps can be quite short - double digit microseconds is not unusual - so relatively small jitter can really add up.)<br>
<br>
So HPC users are really fussy about affinity and placement control. Which isnt to say Lustre gets it all right, but its why we care so much.<br>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf of James Simmons <jsimmons@infradead.org><br>
<b>Sent:</b> Thursday, July 5, 2018 7:20:37 PM<br>
<b>To:</b> Weber, Olaf (HPC Data Management & Storage)<br>
<b>Cc:</b> Lustre Development List<br>
<b>Subject:</b> Re: [lustre-devel] [PATCH v3 07/26] staging: lustre: libcfs: NUMA support</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText"><br>
> NeilBrown [<a href="mailto:neilb@suse.com">mailto:neilb@suse.com</a>] wrote:<br>
> <br>
> To help contextualize things: the Lustre code can be decomposed into three parts:<br>
> <br>
> 1) The filesystem proper: Lustre.<br>
> 2) The communication protocol it uses: LNet.<br>
> 3) Supporting code used by Lustre and LNet: CFS.<br>
> <br>
> Part of the supporting code is the CPT mechanism, which provides a way to<br>
> partition the CPUs of a system. These partitions are used to distribute queues,<br>
> locks, and threads across the system. It was originally introduced years ago, as<br>
> far as I can tell mainly to deal with certain hot locks: these were converted into<br>
> read/write locks with one spinlock per CPT.<br>
> <br>
> As a general rule, CPT boundaries should respect node and socket boundaries,<br>
> but at the higher end, where CPUs have 20+ cores, it may make sense to split<br>
> a CPUs cores across several CPTs.<br>
> <br>
> > Thanks everyone for your patience in explaining things to me.<br>
> > I'm beginning to understand what to look for and where to find it.<br>
> > <br>
> > So the answers to Greg's questions:<br>
> > <br>
> > Where are you reading the host memory NUMA information from?<br>
> > <br>
> > And why would a filesystem care about this type of thing? Are you<br>
> > going to now mirror what the scheduler does with regards to NUMA<br>
> > topology issues? How are you going to handle things when the topology<br>
> > changes? What systems did you test this on? What performance<br>
> > improvements were seen? What downsides are there with all of this?<br>
> > <br>
> > <br>
> > Are:<br>
> <br>
> > - NUMA info comes from ACPI or device-tree just like for every one<br>
> > else. Lustre just uses node_distance().<br>
> <br>
> Correct, the standard kernel interfaces for this information are used to<br>
> obtain it, so ultimately Lustre/LNet uses the same source of truth as<br>
> everyone else.<br>
> <br>
> > - The filesystem cares about this because... It has service<br>
> > thread that does part of the work of some filesystem operations<br>
> > (handling replies for example) and these are best handled "near"<br>
> > the CPU the initiated the request. Lustre partitions<br>
> > all CPUs into "partitions" (cpt) each with a few cores.<br>
> > If the request thread and the reply thread are on different<br>
> > CPUs but in the same partition, then we get best throughput<br>
> > (is that close?)<br>
> <br>
> At the filesystem level, it does indeed seem to help to have the service<br>
> threads that do work for requests run on a different core that is close to<br>
> the core that originated the request. So preferably on the same CPU, and<br>
> on certain multi-core CPUs there are also distance effects between cores.<br>
> That too is one of the things the CPT mechanism handles.<br>
<br>
Their is another very important aspect to why Lustre has a CPU partition <br>
layer. At least at the place I work at. While the Linux kernel manages all<br>
the NUMA nodes and CPU cores Lustre adds the ability for us to specify a <br>
subset of everything on the system. The reason is to limit the impact of<br>
noise on the compute nodes. Noise has a heavy impact on large scale HP<br>
work loads that can run days or even weeks at a time. Lets take an <br>
example system:<br>
<br>
|-------------| |-------------|<br>
|-------| | NUMA 0 | | NUMA 1 | |-------|<br>
| eth0 | - | | --- | | - | eth1 | <br>
|_______| | CPU0 CPU1 | | CPU2 CPU3 | |_______|<br>
|_____________| |_____________|<br>
<br>
In such a system it is possible with the right job schedular to start a <br>
large parallel application on NUMA 0/ (CPU0 and CPU1). Normally such<br>
large parallel applications will communicate between nodes using MPI,<br>
such as openmpi, which can be configured to use eth0 only. Using the<br>
CPT layer in lustre we can isolate lustre to NUMA 1 and use only eth1.<br>
This greatly reducess the noise impact on the application running.<br>
<br>
BTW this is one of the reasons ko2iblnd for lustre doesn't use the<br>
generic RDMA api. The core IB layer doesn't support such isolation.<br>
At least to my knowledge.<br>
<br>
> > - Not really mirroring the scheduler, maybe mirroring parts of the<br>
> > network layer(?)<br>
> <br>
> The LNet code, which is derived from Portals 3.x, is mostly an easier-to-use<br>
> abstraction of RDMA interfaces provided by Infiniband and other similar<br>
> hardware. It can also use TCP/IP, but that's not the primary use case.<br>
> <br>
> As a communication layer that builds on top of RDMA-capable hardware,<br>
> LNet cares about such things as whether the CPU driving communication<br>
> is close to the memory used, and also whether it is close to the interface<br>
> used. Even in a 2-socket machine, there are measurable performance<br>
> differences depending on whether the memory an interface connect<br>
> to the same socket or to different sockets. On bigger hardware, like a<br>
> 32-socket machine, the penalties are much more pronounced. At the<br>
> time we found that the QPI links between sockets were a bottleneck<br>
> and that performance cratered if they had to handle too much traffic.<br>
> <br>
> UPI, the successor to QPI is better -- has more bandwidth -- but with<br>
> the CPUs having more and more cores I expect the scaling issues to<br>
> remain similar.<br>
> <br>
> > - We don't handle topology changes yet except in very minimal ways<br>
> > (cpts *can* become empty, and that can cause problems).<br>
> <br>
> Yes, this is a known deficiency.<br>
> <br>
> > - This has been tested on .... great big things.<br>
> <br>
> The basic CPT mechanism predates my involvement with Lustre. I did<br>
> work on making it more NUMA-aware. A 32-socket system was one of<br>
> the primary test beds.<br>
> <br>
> > - When multi-rails configurations are used (like ethernet-bonding,<br>
> > but for RDMA), we get ??? closer to theoretical bandwidth.<br>
> > Without these changes it scales poorly (??)<br>
> <br>
> The basic idea behind muti-rail configurations is that we use several<br>
> Infiniband interfaces and LNet presents them as a single logical interface<br>
> to Lustre. For each message, LNet picks the IB interface it should go across<br>
> using several criteria, including NUMA distance of the interface and how<br>
> busy it is.<br>
> <br>
> With these changes we could get pretty much linear scaling of LNet<br>
> throughput by adding more interfaces.<br>
> <br>
> > - The down-sides primarily are that we don't auto-configure<br>
> > perfectly. This particularly affects hot-plug, but without<br>
> > hotplug the grouping of cpus and interfaces are focussed<br>
> > on .... avoiding worst case rather than achieving best case.<br>
> <br>
> Without hotplug the CPT grouping should be tuned to achieve a best<br>
> case in a static configuration.<br>
> <br>
> Adding simple-minded hotplug tolerance (let's not call it support) would<br>
> focus on avoiding truly pathological behaviour.<br>
> <br>
> > I've made up a lot of stuff there. I'm happy not to pursue this further at the<br>
> > moment, but if anyone would like to enhance my understanding by<br>
> > correcting the worst errors in the above, I wouldn't object :-)<br>
> > <br>
> > Thanks,<br>
> > NeilBrown<br>
> <br>
> PS: the NUMA effects I've mentioned above have been making the news<br>
> lately under other names: they are part of the side channels used in various<br>
> timing based attacks.<br>
> <br>
> Olaf<br>
> <br>
> _______________________________________________<br>
> lustre-devel mailing list<br>
> lustre-devel@lists.lustre.org<br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><br>
> <br>
_______________________________________________<br>
lustre-devel mailing list<br>
lustre-devel@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><br>
</div>
</span></font></div>
</body>
</html>