[lustre-discuss] [EXTERNAL] Lustre NVMe Flash Pool – lfs mirror Resync Performance and Best Practices

[Mohr, Rick]

Ben,

Have you tried using "lfs migrate" to see if it performs better?  The migrate command has an option to avoid using direct IO.

On Orion at ORNL, our default layout uses PFL but no mirroring.  (Users can create their own mirrors if they want, but that's not the default.)  We have a 90-day purge policy to remove old files in an attempt to keep lustre utilization under control, but it's not uncommon for the performance tier (the ost pool for nvme osts) to fill up.  When necessary, we identity files on the performance tier and use "lfs migrate" to move them to the capacity tier.

-Rick


On 8/20/26, 1:41 AM, "lustre-discuss on behalf of Ben Halfon via lustre-discuss"  wrote:

Hi all,
I’m writing to ask the Lustre community for advice on the best way to utilize NVMe OSTs as a flash pool in a ZFS-based Lustre environment.
From our initial research, it seems that one approach for implementing a flash-pool-like setup is to use the Robinhood v4 policy engine to manage files and periodically perform operations such as:

lfs mirror resync
lfs mirror split --mirror-id <nvme_stripe_id> -d

Our main issue is the performance of lfs mirror resync.
We observed that lfs mirror resync uses Direct I/O, and in our environment, we are currently seeing very low throughput, around 8–20 MB/s. Increasing the maximum number of RPCs in flight improved the performance somewhat, but we still cannot get close to the 250–300 MB/s write performance that we can achieve with a single process using dd directly to the NVMe OST.
We also noticed that Lustre doesn't have Direct I/O support in the ZFS layer, so we initially suspected this could be contributing to the problem. We tested the Direct I/O ZFS patch:
https://review.whamcloud.com/c/fs/lustre-release/+/41689/ <https://review.whamcloud.com/c/fs/lustre-release/+/41689/> <OWAAutoLink>
However, applying the patch did not improve the lfs mirror resync performance in our case.
We would therefore appreciate any feedback from the community regarding:


* What kind of performance should we expect from lfs mirror resync?
* Are there any Lustre, ZFS, or RPC-related tunables that are particularly relevant to improving mirror resync performance?
* Are there known limitations or performance considerations when using lfs mirror resync with ZFS-based OSTs?
* Has anyone successfully used FLR/mirrors as the basis for implementing an NVMe flash pool?
* Is there a better or more recommended way to implement a hot/cold flash-pool architecture, where files are initially written to NVMe and later migrated to HDD when they become cold or when the NVMe tier reaches a certain utilization level?
Test Environment
Lustre servers


* Lustre 2.17
* Additional patch: https://review.whamcloud.com/c/fs/lustre-release/+/61103/ <https://review.whamcloud.com/c/fs/lustre-release/+/61103/> <OWAAutoLink> (fixes an issue with small 4K random blocks)
* 2 × MDS, 1 MDT each
* 2 × NVMe OSS, 1 OST each
* 2 × HDD OSS, 1 OST each
* AlmaLinux 9.7
* Kernel: 5.14.0-611.5.1.el9_7.x86_64
* ZFS: 2.3.8
* Network: RoCEv2 and TCP on the Lustre server side
Clients


* AlmaLinux 9.7 — 2 × 100 Gb/s RoCEv2
* AlmaLinux 9.6 — 2 × 40 Gb/s TCP
This is currently a test environment, so we are open to different approaches and would especially appreciate recommendations based on real-world experience.
Thanks in advance for any advice, tuning recommendations, or alternative approaches you can share.
Best regards,
Ben