[lustre-discuss] Binding OST to CPU NUMA

[Jeffrey Wong]

Hello,

I am relatively new to Lustre. I have a running Lustre setup and  would like to ensure that there is no cross-numa traffic going on in the server side.

Each Lustre OSS has the following:

  *   Dual socket AMD CPUs
  *   4 NUMA regions (2 NUMA going to each socket)
  *   2x Mellanox NIC (one for each CPU socket).
  *   16x NVMe drives (8 for each CPU socket).

The way I currently have it set up:

  *   Each group of 8 NVMe on the same socket, I created a RAID0 group. Each RAID0 group is an OST.
  *   In the LNET configuration, each NIC is assigned to the corresponding CPT. We are using RDMA over Ethernet.
     *   o2ib0(ens3f0np0)[0,1], o2ib1(ens7f0np0)[2,3]


According to the documentation, messages coming to the OSS will have the corresponding CPT/NUMA cores that match the NIC handle the incoming traffic. However when accessing the OST themselves I did not find any tuning to bind a specific OST to CPU cores.

I did see an option called “options ksocklnd enable_irq_affinity=0” but it appears it is only for the ethernet driver and not RDMA.

Is there  a way to limit each OST’s read/write threads to a specific set of CPU cores?


Thank you,
Jeff


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20230201/68161947/attachment.htm>