<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hi all,</div>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Im writing to ask the Lustre community for advice on the best way to utilize NVMe OSTs as a flash pool in a ZFS-based Lustre environment.</div>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
>From our initial research, it seems that one approach for implementing a flash-pool-like setup is to use the Robinhood v4 policy engine to manage files and periodically perform operations such as:</div>
<pre class="elementToProof"><div class="elementToProof" style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><code>lfs mirror resync
lfs mirror split --mirror-id <nvme_stripe_id> -d</code></div></pre>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Our main issue is the performance of <code>lfs mirror resync</code>.</div>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
We observed that <code>lfs mirror resync</code> uses Direct I/O, and in our environment, we are currently seeing very low throughput, around
<b>820 MB/s</b>. Increasing the maximum number of RPCs in flight improved the performance somewhat, but we still cannot get close to the
<b>250300 MB/s</b> write performance that we can achieve with a single process using
<code>dd</code> directly to the NVMe OST.</div>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
We also noticed that Lustre doesn't have Direct I/O support in the ZFS layer, so we initially suspected this could be contributing to the problem. We tested the Direct I/O ZFS patch:</div>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<a href="https://review.whamcloud.com/c/fs/lustre-release/+/41689/" id="OWAebbbed38-1ef4-46c7-d3de-cd2db9b52325" class="OWAAutoLink" data-rich-text-autolink="">https://review.whamcloud.com/c/fs/lustre-release/+/41689/</a></div>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
However, applying the patch did not improve the <code>lfs mirror resync</code> performance in our case.</div>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
We would therefore appreciate any feedback from the community regarding:</div>
<ul data-spread="false">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
What kind of performance should we expect from <code>lfs mirror resync</code>?</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Are there any Lustre, ZFS, or RPC-related tunables that are particularly relevant to improving mirror resync performance?</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Are there known limitations or performance considerations when using <code>lfs mirror resync</code> with ZFS-based OSTs?</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Has anyone successfully used FLR/mirrors as the basis for implementing an NVMe flash pool?</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Is there a better or more recommended way to implement a <b>hot/cold flash-pool architecture</b>, where files are initially written to NVMe and later migrated to HDD when they become cold or when the NVMe tier reaches a certain utilization level?</li></ul>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<b>Test Environment</b></div>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<b>Lustre servers</b></div>
<ul data-spread="false">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Lustre 2.17</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Additional patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/61103/" id="OWAd362a340-f5c9-7d53-58fc-97ea6eaf0c03" class="OWAAutoLink" data-rich-text-autolink="">
https://review.whamcloud.com/c/fs/lustre-release/+/61103/</a> (fixes an issue with small 4K random blocks)</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
2 MDS, 1 MDT each</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
2 NVMe OSS, 1 OST each</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
2 HDD OSS, 1 OST each</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
AlmaLinux 9.7</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Kernel: <code>5.14.0-611.5.1.el9_7.x86_64</code></li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
ZFS: <code>2.3.8</code></li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Network: RoCEv2 and TCP on the Lustre server side</li></ul>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<b>Clients</b></div>
<ul data-spread="false">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
AlmaLinux 9.7 2 100 Gb/s RoCEv2</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
AlmaLinux 9.6 2 40 Gb/s TCP</li></ul>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
This is currently a test environment, so we are open to different approaches and would especially appreciate recommendations based on real-world experience.</div>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Thanks in advance for any advice, tuning recommendations, or alternative approaches you can share.</div>
<div class="elementToProof" style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Best regards,<br>
Ben</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
</body>
</html>