[lustre-discuss] Issue with High-Load Write Operations in Lustre Cluster

[zufei chen]

Dear Lustre Community,

I am encountering an issue with the Lustre high-availability component,
*Corosync*, which experiences packet loss under high load, triggering
fencing and powering down Lustre nodes. I am seeking advice on how to
resolve this issue. Below are the details of our environment and the
problem:
Environment:

   1. *Lustre version:* 2.15.5
   2. *Physical machines:* 11 machines, each with 128 CPU cores and 376GB
   of memory.
   3. *Virtualization:* Each physical machine runs a KVM virtual machine
   with 20 cores and 128GB of memory, using Rocky Linux 8.10.
   4. *Lustre setup:* Each VM has 2 MDTs (512GB each) and 16 OSTs (670GB
   each).
   5. *Configuration* (/etc/modprobe.d/lustre.conf):

   options lnet networks="o2ib(enp0s5f0np0)"
   options libcfs cpu_npartitions=2
   options ost oss_num_threads=512
   options mds mds_num_threads=512
   options ofd adjust_blocks_percent=11

   6. *Network:* 100GB RDMA network.
   7. *Clients:* 11 clients using *vdbench* to perform large file writes
   (total write bandwidth approximately 50GB).

Issue:

Under high load write operations, the *Corosync* component experiences
packet loss. There is a probability that heartbeat loss triggers *Pacemaker's
fencing mechanism*, which powers down the Lustre nodes.
Analysis Conducted:

   1. *CPU usage:* The CPU utilization is not very high, but the cpu load
   is very high (reaching around 400).
   2. *Packet loss:* There is packet loss observed when pinging between
   Lustre nodes.
   3. *Tuning ost_num_threads and mds_num_threads:* Reducing these values
   reduced the system load and improved packet loss significantly, but it also
   led to a decrease in the Vdbench write bandwidth.
   4. *Network tuning:* After adjusting net.ipv4.udp_mem (three times
   larger than default), packet loss improved, but it still persists.

   sysctl -w net.ipv4.udp_mem="9217055 12289407 18434106"


Assistance Requested:

I would appreciate any suggestions from the community on how to resolve
this issue effectively. If anyone has faced similar challenges, your
insights would be especially valuable.

Thank you for your time and assistance. I look forward to your responses.

Best regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20241125/0414e020/attachment.htm>