<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="en-RU" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Dear all,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I had a chance to try the Lustre DIO/AIO improvements appeared at 2.15 version (described here
<a href="https://jira.whamcloud.com/browse/LU-13798">https://jira.whamcloud.com/browse/LU-13798</a> and here
<a href="https://jira.whamcloud.com/browse/LU-13799">https://jira.whamcloud.com/browse/LU-13799</a> ). But It seems to me, that it does not fully work or I dont understand how it should work.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Ive two servers as OSSs with one RAID-on-NVMe-based OST on each and a client. MDS and MGT work at another server. All servers and client are interconnected with Infiniband HDR. All the settings during the filesystem
creation and mount were default.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The test command is:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ior -w -r -t 256M -b 64G -e -o /<dirpath>/iorfile --posix.odirect<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I also applied the following tunning at the client:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">lctl set_param osc.*.max_pages_per_rpc=4M osc.*.checksums=0 osc.*.max_rpcs_in_flight=16<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">So, when I run a test without any striping settings (and the test file appears only on one OST) the results are about:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Write: 4008.78 MiB/sec (4203.51 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Read: 7061.42 MiB/sec (7404.44 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">When I set striping to -S 2M -c 2 the results are about:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Write: 8796.89 MiB/sec (9224.21 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Read: 19043.23 MiB/sec (19968.27 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">But when I disable caching at both OSSs with the following commands<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/usr/sbin/lctl set_param osd-ldiskfs.*.read_cache_enable=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/usr/sbin/lctl set_param osd-ldiskfs.*.writethrough_cache_enable=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">the striping results became about:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Write: 13054.52 MiB/sec (13688.65 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Read: 18323.93 MiB/sec (19214.03 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">It looks like read and write caching on the OSS side works even if the test is launched with --posix.odirect flag, unless caching is manually disabled at OSS side. And cache logic calculations slow down the overall performance
for writing (which is an expected behavior in case of NVMe-based storage). It looks strange for me, since as I got from some docs, when operations come with O_DIRECT flag, it should avoid any caching on both sides on client side and on server side too. So,
from my point of view, DIO just does not work or works incorrectly.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">May I ask dear all enlighten me in this question: maybe my point of view is incorrect, or I missed some settings during the filesystem creation and mounting needed to let DIO work properly.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thank you!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Daniel<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>