<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:Courier;}
span.EmailStyle17
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Courier;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText">Hello, <o:p></o:p></p>
<p class="MsoPlainText"> <o:p></o:p></p>
<p class="MsoPlainText">We are having a few performance issues with our newest lustre file system. Here is the overview of our setup:<o:p></o:p></p>
<p class="MsoPlainText"> <o:p></o:p></p>
<p class="MsoPlainText">-) Supermicro servers connected to external 12Gb/s SAS JBODs for MDT/OSS storage</p>
<p class="MsoPlainText">-) CentOS version = 7.3.1611 (kernel 3.10.0-514.2.2.el7.x86_64) on the servers and clients<o:p></o:p></p>
<p class="MsoPlainText">-) stock kernels on servers (i.e. no lustre patches to the kernel)<o:p></o:p></p>
<p class="MsoPlainText">-) ZFS backend </p>
<p class="MsoPlainText"> o version = zfs-0.6.5.8-1.el7_3.centos.x86_64<o:p></o:p></p>
<p class="MsoPlainText"> o ZFS metadata pool: RAID10, 12x 15,000 rpm Seagate Cheeta 12 Gb/s SAS</p>
<p class="MsoPlainText"> o ZFS ost pools (12x): RAIDZ3 15x 7,200 rpm HGST 8TB 12 Gb/s SAS </p>
<p class="MsoPlainText"> o ZFS compression on OST's (not MDT)</p>
<p class="MsoPlainText">-) lustre version 2.9.0 (plus a patch to get dual homed servers with failover working properly)<o:p></o:p></p>
<p class="MsoPlainText">-) lnet dual home config \u2013 ethernet and infiniband<o:p></o:p></p>
<p class="MsoPlainText">-) ~400 IB clients, ~60 ethernet clients<o:p></o:p></p>
<p class="MsoPlainText"> <o:p></o:p></p>
<p class="MsoPlainText">Our large file read/write performance is excellent. However, "everyday" operations (creating, moving, removing, and opening files) are noticeably slower as compared to our older LFS (version 2.4.3). We are surprised at this since both
the server hardware and software are newer and more capable. One concrete thing we have discovered is that any operation that forces a flush to the disk is more than 10x slower on our new setup. We discovered this when pasting a large amount of text into
vim \u2013 this took 30+ seconds to return on a file that lives on our new LFS but almost no lag for a file that is on our old LFS, an NFS mount or a local disk. It turns out vim forces a sync the swap file to disk every 200 characters by default. We replicated
this with a simple dd:<o:p></o:p></p>
<p class="MsoPlainText"> <o:p></o:p></p>
<p class="MsoPlainText">for i in $(seq 0 99) ; do<o:p></o:p></p>
<p class="MsoPlainText"> dd if=/dev/zero of=dd.dat.$i bs=1k count=1 conv=fsync > /dev/null 2>&1<o:p></o:p></p>
<p class="MsoPlainText">done<o:p></o:p></p>
<p class="MsoPlainText"> <o:p></o:p></p>
<p class="MsoPlainText">The timing of this ranges from 0.1 to 1 sec on our old LFS but ranges from 20 to 60 sec on our newer 2.9 LFS. We\u2019ve tried a number of ZFS tuning options (zfs_txg_timeout, zfs_vdev_scheduler, zfs_prefetch_disable,\u2026) with little/no impact.</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Additionally, we temporarily toggled sync\u2019ing on the ZFS filesystems underlying our LFS:</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"># zfs set sync=disabled metadata/meta-fsl</p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier"># zfs set sync=disabled oss00-0/ost-fsl<o:p></o:p></span></p>
<p class="MsoPlainText"># (repeat 11x for other oss/ost's)</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">later restoring via</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"># zfs set sync=standard metadata/meta-fsl</p>
<p class="MsoPlainText"># zfs set sync=standard oss00-0/ost-fsl</p>
<p class="MsoPlainText"># (repeat 11x for other oss/ost's)</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">We tried the same test on a raw zfs FS (oss00-0/testfs) in the same pool. This option significantly speeds up fsync() calls to an individual ZFS filesystem (i.e. oss00-0/testfs shows ~10x speedup), but only marginally benefited the
LFS fsync() behavior (~10%).</p>
<p class="MsoPlainText"> <o:p></o:p></p>
<p class="MsoPlainText">Any ideas on what might be going on here? Any other ZFS or lustre tuning you would try?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Thanks, <o:p></o:p></p>
<p class="MsoPlainText">Darby<o:p></o:p></p>
</div>
</body>
</html>