<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="en-RU" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Dear all,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I had a chance to try the Lustre DIO/AIO improvements appeared at 2.15 version (described here
<a href="https://jira.whamcloud.com/browse/LU-13798">https://jira.whamcloud.com/browse/LU-13798</a> and here
<a href="https://jira.whamcloud.com/browse/LU-13799">https://jira.whamcloud.com/browse/LU-13799</a> ). But It seems to me, that it does not fully work or I don’t understand how it should work.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I’ve two servers as OSSs with one RAID-on-NVMe-based OST on each and a client. MDS and MGT work at another server. All servers and client are interconnected with Infiniband HDR. All the settings during the filesystem
creation and mount were default.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The test command is:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ior -w -r -t 256M -b 64G -e -o /<dirpath>/iorfile --posix.odirect<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I also applied the following tunning at the client:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">lctl set_param osc.*.max_pages_per_rpc=4M osc.*.checksums=0 osc.*.max_rpcs_in_flight=16<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">So, when I run a test without any striping settings (and the test file appears only on one OST) the results are about:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Write: 4008.78 MiB/sec (4203.51 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Read: 7061.42 MiB/sec (7404.44 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">When I set striping to “-S 2M -c 2” the results are about:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Write: 8796.89 MiB/sec (9224.21 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Read: 19043.23 MiB/sec (19968.27 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">But when I disable caching at both OSSs with the following commands<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/usr/sbin/lctl set_param osd-ldiskfs.*.read_cache_enable=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/usr/sbin/lctl set_param osd-ldiskfs.*.writethrough_cache_enable=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">the striping results became about:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Write: 13054.52 MiB/sec (13688.65 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Max Read: 18323.93 MiB/sec (19214.03 MB/sec)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">It looks like read and write caching on the OSS side works even if the test is launched with --posix.odirect flag, unless caching is manually disabled at OSS side. And cache logic calculations slow down the overall performance
for writing (which is an expected behavior in case of NVMe-based storage). It looks strange for me, since as I got from some docs, when operations come with O_DIRECT flag, it should avoid any caching on both sides – on client side and on server side too. So,
from my point of view, DIO just does not work or works incorrectly.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">May I ask dear all enlighten me in this question: maybe my point of view is incorrect, or I missed some settings during the filesystem creation and mounting needed to let DIO work properly.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thank you!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Daniel<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>