<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Jan 22, 2021, at 02:01, Tung-Han Hsieh <<a href="mailto:thhsieh@twcp1.phys.ntu.edu.tw" class="">thhsieh@twcp1.phys.ntu.edu.tw</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">Recently we read from this mailing list and learned that currently<br class="">
Lustre file system MDT with ZFS backend has poor performance. Too<br class="">
bad that we did not notice this before deploying this configuration<br class="">
to many of our cluster systems. Now that there are already too much<br class="">
amount of data to reconfigure them back to MDT with ldiskfs backend.<br class="">
<br class="">
So now the only way for us is looking forward and try to do our best.<br class="">
I read this article about the improvement of MDT with ZFS:<br class="">
<br class="">
<a href="https://www.nextplatform.com/2017/01/11/bolstering-lustre-zfs-highlights-continuing-work/" class="">https://www.nextplatform.com/2017/01/11/bolstering-lustre-zfs-highlights-continuing-work/</a><br class="">
<br class="">
It seems that MDT with ZFS already had large improvement over time,<br class="">
but in our applications it is still not enough. So I am asking that<br class="">
whether there is a roadmap to improve this part ?<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
There are a number of different organizations that are using Lustre and</div>
<div>ZFS, and of course many developers working to improve ZFS itself.</div>
<div><br class="">
</div>
<div>Definitely you would want to use all-flash storage for the MDT, if you</div>
<div>are not already.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">Furthermore, I saw that ZFS already has version 2.0.1. Is there any<br class="">
plan of Lustre part to integrate and take advantages of the new ZFS<br class="">
software ?<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
The soon-to-be-released Lustre 2.14 is known to work with ZFS 2.0, so</div>
<div>you are welcome to test that out and report if it improves performance.</div>
<div><br class="">
</div>
<div>Cheers, Andreas</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">In the meanwhile, currently we have the following configurations and<br class="">
tunings in our Lustre system to try to overcome various performance<br class="">
bottlenecks (both MDT and OST are ZFS backend):<br class="">
<br class="">
- Linux kernel 4.19.126 + MLNX_OFED-4.6 + Lustre-2.12.6 + ZFS-0.7.3<br class="">
(ps. We build all the above software by ourselves in Debian-9.12)<br class="">
<br class="">
- Loading zfs module with the following options<br class="">
(Many thanks to the suggestions by Riccardo Veraldi):<br class="">
options zfs zfs_prefetch_disable=1<br class="">
options zfs zfs_txg_history=120<br class="">
options zfs metaslab_debug_unload=1<br class="">
options zfs zfs_vdev_async_write_active_min_dirty_percent=20<br class="">
options zfs zfs_vdev_scrub_min_active=48<br class="">
options zfs zfs_vdev_scrub_max_active=128<br class="">
options zfs zfs_vdev_sync_write_min_active=8<br class="">
options zfs zfs_vdev_sync_write_max_active=32<br class="">
options zfs zfs_vdev_sync_read_min_active=8<br class="">
options zfs zfs_vdev_sync_read_max_active=32<br class="">
options zfs zfs_vdev_async_read_min_active=8<br class="">
options zfs zfs_vdev_async_read_max_active=32<br class="">
options zfs zfs_top_maxinflight=320<br class="">
options zfs zfs_txg_timeout=30<br class="">
options zfs zfs_dirty_data_max_percent=40<br class="">
options zfs zfs_vdev_async_write_min_active=8<br class="">
options zfs zfs_vdev_async_write_max_active=32<br class="">
<br class="">
- ZFS pool is configured with the following options:<br class="">
zfs set atime=off <pool><br class="">
zfs set redundant_metadata=most <pool><br class="">
zfs set xattr=sa <pool><br class="">
zfs set recordsize=1M <pool><br class="">
<br class="">
- Set the grant_shrink option to 0 for all clients of Lustre:<br class="">
lctl set_param osc.*.grant_shrink=0<br class="">
<br class="">
These are all we have learned so far. We are wondering whether<br class="">
there are still something we have overlooked (e.g., are iommu<br class="">
and intel_iommu settings of kernel parameters help)? We will be<br class="">
very appreciated if anyone could give us further suggestions.<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>