[lustre-discuss] Lustre NVMe Flash Pool – lfs mirror Resync Performance and Best Practices

[Ben Halfon]

Hi all,
I’m writing to ask the Lustre community for advice on the best way to utilize NVMe OSTs as a flash pool in a ZFS-based Lustre environment.
>From our initial research, it seems that one approach for implementing a flash-pool-like setup is to use the Robinhood v4 policy engine to manage files and periodically perform operations such as:

lfs mirror resync
lfs mirror split --mirror-id <nvme_stripe_id> -d

Our main issue is the performance of lfs mirror resync.
We observed that lfs mirror resync uses Direct I/O, and in our environment, we are currently seeing very low throughput, around 8–20 MB/s. Increasing the maximum number of RPCs in flight improved the performance somewhat, but we still cannot get close to the 250–300 MB/s write performance that we can achieve with a single process using dd directly to the NVMe OST.
We also noticed that Lustre doesn't have Direct I/O support in the ZFS layer, so we initially suspected this could be contributing to the problem. We tested the Direct I/O ZFS patch:
https://review.whamcloud.com/c/fs/lustre-release/+/41689/
However, applying the patch did not improve the lfs mirror resync performance in our case.
We would therefore appreciate any feedback from the community regarding:

  *   What kind of performance should we expect from lfs mirror resync?
  *   Are there any Lustre, ZFS, or RPC-related tunables that are particularly relevant to improving mirror resync performance?
  *   Are there known limitations or performance considerations when using lfs mirror resync with ZFS-based OSTs?
  *   Has anyone successfully used FLR/mirrors as the basis for implementing an NVMe flash pool?
  *   Is there a better or more recommended way to implement a hot/cold flash-pool architecture, where files are initially written to NVMe and later migrated to HDD when they become cold or when the NVMe tier reaches a certain utilization level?

Test Environment
Lustre servers

  *   Lustre 2.17
  *   Additional patch: https://review.whamcloud.com/c/fs/lustre-release/+/61103/ (fixes an issue with small 4K random blocks)
  *   2 × MDS, 1 MDT each
  *   2 × NVMe OSS, 1 OST each
  *   2 × HDD OSS, 1 OST each
  *   AlmaLinux 9.7
  *   Kernel: 5.14.0-611.5.1.el9_7.x86_64
  *   ZFS: 2.3.8
  *   Network: RoCEv2 and TCP on the Lustre server side

Clients

  *   AlmaLinux 9.7 — 2 × 100 Gb/s RoCEv2
  *   AlmaLinux 9.6 — 2 × 40 Gb/s TCP

This is currently a test environment, so we are open to different approaches and would especially appreciate recommendations based on real-world experience.
Thanks in advance for any advice, tuning recommendations, or alternative approaches you can share.
Best regards,
Ben

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260820/d074f5b8/attachment.html>