<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Yes, we've done that already. Sorry, I should have posted all our module parameters.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">[root@hpfs-fsl-mds0 lustre]# cat /etc/modprobe.d/lustre.conf<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"># Lustre modprobe configuration<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">options lnet networks=tcp0(enp4s0),o2ib0(ib1),o2ib1(ib0)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">options ko2iblnd map_on_demand=32<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">options osd_zfs osd_txg_sync_delay_us=0 osd_object_sync_delay_us=0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">options zfs zfs_txg_history=120 zfs_txg_timeout=15 zfs_prefetch_disable=1 zfs_txg_timeout=30 zfs_vdev_scheduler=deadline<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">[root@hpfs-fsl-mds0 lustre]#<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Would you mind elaborating on why you think ZFS was a bad choice? In general, we love the features it brings.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Carlson, Timothy S" <Timothy.Carlson@pnnl.gov><br>
<b>Date: </b>Tuesday, January 5, 2021 at 9:21 AM<br>
<b>To: </b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <darby.vicker-1@nasa.gov>, "lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[EXTERNAL] Re: [lustre-discuss] Tuning for metadata performance<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">On the MDS, make sure you have turned off pre-fetch</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">echo 1 > /sys/module/zfs/parameters/zfs_prefetch_disable</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">On our ancient (similar vintage) file system this greatly reduced the load on the MDS so it didn\u2019t come to a complete standstill when under pressure. ZFS for the MDS was a horrible choice (I made the same
mistake).</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Tim</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of "Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <darby.vicker-1@nasa.gov><br>
<b>Date: </b>Tuesday, January 5, 2021 at 7:48 AM<br>
<b>To: </b>"Lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[lustre-discuss] Tuning for metadata performance</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I'm looking for some advice on tuning our existing lustre file system to achieve better metadata performance. This file system is getting fairly old \u2013 its been in production for almost 4 years now. The hardware
and our existing tuning efforts can be found here. </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fpipermail%2Flustre-discuss-lustre.org%2F2017-April%2F014390.html&data=04%7C01%7Cdarby.vicker-1%40nasa.gov%7Cc535e40f55cf4bb5ac9208d8b195efa9%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637454604822649714%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=v7itMDRs9jWCw43ScQqmejiSHeDDbiogEOaT4ThOkdM%3D&reserved=0">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2017-April/014390.html</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The hardware is the same but we have upgraded the software stack a few times \u2013 now on CentOS 7.6, ZFS 0.7.9 and lustre 2.10.8. We do plan to upgrade to the latest CentOS 7.x and either lustre 2.12 or 2.13
soon. The MDS hardware isn't well-described in that thread so here are more details:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Chassis: Supermicro 2U Twin Server</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Processor: 4 x QuadCore Xeon Processor E52637 v2 3.50GHz (2 sockets/8 cores per node)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Memory: 16 x 16GB PC314900 1866MHz DDR3 ECC Registered DIMM (128GB per node)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">External JBOD:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Chassis: 24x HotSwap 2.5" SAS 12Gb/s SAS Dual Expander</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Drives: 12 x 600GB SAS 3.0 12.0Gb/s 15000RPM 2.5" Seagate Enterprise Performance 15K HDD (512n)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Controller Card: LSI SAS 9300-8e SAS 12Gb/s PCIe 3.0 8-Port Host Bus Adapter</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The above hardware and tuning served us well for a long time but the lab has grown, both in number of lustre clients (now up to ~200 ethernet clients and ~500 IB clients) and the number of users in the lab.
With the extra users have come different types of workloads. Peviously, the file system was most used for workloads with a fairly small number of large files. We now see workloads that include 100's of concurrent processes all doing mixed small and large
file IO on a lot of files (e.g. each process clones a repo, compiles a code and runs a serial sim that writes a lot of data).
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I recently ran the io500 tests and our LFS stats for MDEasy and MDHard are pretty bad, even when compared to the lowest MD stats on the current io500 list. Our standard NFS server handily beats our LFS wrt
MD performance. So I'm hopeful that we can squeeze more MD performance out of our LFS. Obviously, software tuning on the existing hardware would be preferred but we are open to hardware additions/upgrades if that would help (e.g. adding more MDS's). There
are a lot of tuning options in both ZFS and lustre so I'm hoping someone can point me in the right direction. Are DNE and/or DoM expected to help? I attended the SC20 Lustre BoF and it sounds like 2.13 has some metadata performance improvements, so just
an upgrade might help. We have dual MDS's now but for HA, not performance. I'd hate to lose the HA aspect as we utilize it for failover quite a bit (maintenance, etc.) but it would probably be worth it if MD performance was significantly improved. If I
understand correctly, there is some overhead with DNE and performance suffers with just two MDS's with a benefit with 4 or more MDS's, correct? So that wouldn't be a good option for us unless we add MDS's? Would an upgrade to SSD or NVMe in our MDTs help?
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I would greatly appreciate thoughts on the best path forward for making improvements.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Darby </span><o:p></o:p></p>
</div>
</body>
</html>