<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hi Rick,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Thank you for your response.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
We have tested the LFS migrate approach, and we can confirm that using the -D / --non-direct flag is the configuration that produces the expected bandwidth results.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
We will proceed with this method to perform the migration from the Flash Pool to the HDD Pool.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Best regards,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Ben</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg">
<div style="direction: ltr; font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<b>From:</b> Mohr, Rick <mohrrf@ornl.gov><br>
<b>Sent:</b> Thursday, August 20, 2026 6:04 PM<br>
<b>To:</b> Ben Halfon <ben.Halfon@mobileye.com>; lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> Re: [EXTERNAL] [lustre-discuss] Lustre NVMe Flash Pool \u2013 lfs mirror Resync Performance and Best Practices</div>
<div style="direction: ltr;"> </div>
</div>
<div style="font-size: 11pt;">EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe.<br>
<br>
Ben,<br>
<br>
Have you tried using "lfs migrate" to see if it performs better? The migrate command has an option to avoid using direct IO.<br>
<br>
On Orion at ORNL, our default layout uses PFL but no mirroring. (Users can create their own mirrors if they want, but that's not the default.) We have a 90-day purge policy to remove old files in an attempt to keep lustre utilization under control, but it's
not uncommon for the performance tier (the ost pool for nvme osts) to fill up. When necessary, we identity files on the performance tier and use "lfs migrate" to move them to the capacity tier.<br>
<br>
-Rick<br>
<br>
<br>
\ufeffOn 8/20/26, 1:41 AM, "lustre-discuss on behalf of Ben Halfon via lustre-discuss" wrote:<br>
<br>
Hi all,<br>
I\u2019m writing to ask the Lustre community for advice on the best way to utilize NVMe OSTs as a flash pool in a ZFS-based Lustre environment.<br>
From our initial research, it seems that one approach for implementing a flash-pool-like setup is to use the Robinhood v4 policy engine to manage files and periodically perform operations such as:<br>
<br>
lfs mirror resync<br>
lfs mirror split --mirror-id <nvme_stripe_id> -d<br>
<br>
Our main issue is the performance of lfs mirror resync.<br>
We observed that lfs mirror resync uses Direct I/O, and in our environment, we are currently seeing very low throughput, around 8\u201320 MB/s. Increasing the maximum number of RPCs in flight improved the performance somewhat, but we still cannot get close to the
250\u2013300 MB/s write performance that we can achieve with a single process using dd directly to the NVMe OST.<br>
We also noticed that Lustre doesn't have Direct I/O support in the ZFS layer, so we initially suspected this could be contributing to the problem. We tested the Direct I/O ZFS patch:<br>
<a href="https://review.whamcloud.com/c/fs/lustre-release/+/41689/" id="OWAe241263d-1ed3-966e-7416-ff823c9edbab" class="OWAAutoLink" data-auth="NotApplicable">https://review.whamcloud.com/c/fs/lustre-release/+/41689/</a> <<a href="https://review.whamcloud.com/c/fs/lustre-release/+/41689/" id="OWAe856c7e4-7f52-4086-a316-5fc03fbeacae" class="OWAAutoLink" data-auth="NotApplicable">https://review.whamcloud.com/c/fs/lustre-release/+/41689/</a>>
<OWAAutoLink><br>
However, applying the patch did not improve the lfs mirror resync performance in our case.<br>
We would therefore appreciate any feedback from the community regarding:<br>
<br>
<br>
* What kind of performance should we expect from lfs mirror resync?<br>
* Are there any Lustre, ZFS, or RPC-related tunables that are particularly relevant to improving mirror resync performance?<br>
* Are there known limitations or performance considerations when using lfs mirror resync with ZFS-based OSTs?<br>
* Has anyone successfully used FLR/mirrors as the basis for implementing an NVMe flash pool?<br>
* Is there a better or more recommended way to implement a hot/cold flash-pool architecture, where files are initially written to NVMe and later migrated to HDD when they become cold or when the NVMe tier reaches a certain utilization level?<br>
Test Environment<br>
Lustre servers<br>
<br>
<br>
* Lustre 2.17<br>
* Additional patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/61103/" id="OWAe75474fb-edde-114a-0177-2807444abb73" class="OWAAutoLink" data-auth="NotApplicable">
https://review.whamcloud.com/c/fs/lustre-release/+/61103/</a> <<a href="https://review.whamcloud.com/c/fs/lustre-release/+/61103/" id="OWA20f0a0fb-6c37-403a-46e3-fa753591b3e0" class="OWAAutoLink" data-auth="NotApplicable">https://review.whamcloud.com/c/fs/lustre-release/+/61103/</a>>
<OWAAutoLink> (fixes an issue with small 4K random blocks)<br>
* 2 × MDS, 1 MDT each<br>
* 2 × NVMe OSS, 1 OST each<br>
* 2 × HDD OSS, 1 OST each<br>
* AlmaLinux 9.7<br>
* Kernel: 5.14.0-611.5.1.el9_7.x86_64<br>
* ZFS: 2.3.8<br>
* Network: RoCEv2 and TCP on the Lustre server side<br>
Clients<br>
<br>
<br>
* AlmaLinux 9.7 \u2014 2 × 100 Gb/s RoCEv2<br>
* AlmaLinux 9.6 \u2014 2 × 40 Gb/s TCP<br>
This is currently a test environment, so we are open to different approaches and would especially appreciate recommendations based on real-world experience.<br>
Thanks in advance for any advice, tuning recommendations, or alternative approaches you can share.<br>
Best regards,<br>
Ben<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</div>
</body>
</html>