<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Sorry I thought I had told the ZFS and
lustre version but I did not.<br>
Lustre version is 2.8.0 ZFS is 0.6.5.8-1<br>
<br>
I understand that I cannot expect performance from Direct I/O but
on hte previous Lsutre clsuter with ldiskfs<br>
and hardare raid I was having 300MB/sec. Probably the hw raid
caching mechanism was aiding in this.<br>
<br>
thanks<br>
<br>
Riccardo<br>
<br>
<br>
On 14/10/16 13:12, Patrick Farrell wrote:<br>
</div>
<blockquote
cite="mid:CY4PR11MB1751222AD595C8DFFF6A729BCBDF0@CY4PR11MB1751.namprd11.prod.outlook.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" style="font-size:12pt;
color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Riccardo,</p>
<p><br>
</p>
<p>While the difference is extreme, direct I/O write
performance will always be poor. Direct I/O writes cannot
be asynchronous, since they don't use the page cache. This
means Lustre cannot return from one write (and start the
next) until it has finished transferring the data to the
network.</p>
<p><br>
</p>
<p>This means you can only have one I/O in flight at a time.
Good write performance from Lustre (or any network
filesystem) depends on keeping a lot of data in flight at
once.</p>
<p><br>
</p>
<p>What sort of direct write performance were you hoping for?
It will never match that 800 MB/s from one thread you see
with buffered I/O.</p>
<p><br>
</p>
<p>- Patrick<br>
</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
lustre-discuss
<a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss-bounces@lists.lustre.org"><lustre-discuss-bounces@lists.lustre.org></a> on behalf of
Riccardo Veraldi <a class="moz-txt-link-rfc2396E" href="mailto:Riccardo.Veraldi@cnaf.infn.it"><Riccardo.Veraldi@cnaf.infn.it></a><br>
<b>Sent:</b> Friday, October 14, 2016 2:22:32 PM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> [lustre-discuss] Lustre on ZFS pooer direct
I/O performance</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">Hello,<br>
<br>
I would like how may I improve the situation of my lustre
cluster.<br>
<br>
I have 1 MDS and 1 OSS with 20 OST defined.<br>
<br>
Each OST is a 8x Disks RAIDZ2.<br>
<br>
A single process write performance is around 800MB/sec<br>
<br>
anyway if I force direct I/O, for example using oflag=direct
in dd, the <br>
write performance drop as low as 8MB/sec<br>
<br>
with 1MB block size. And each write it's about 120ms
latency.<br>
<br>
I used these ZFS settings<br>
<br>
options zfs zfs_prefetch_disable=1<br>
options zfs zfs_txg_history=120<br>
options zfs metaslab_debug_unload=1<br>
<br>
i am quite worried for the low performance.<br>
<br>
Any hints or suggestions that may help me to improve the
situation ?<br>
<br>
<br>
thank you<br>
<br>
<br>
Rick<br>
<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a moz-do-not-send="true"
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div>
</span></font>
</blockquote>
<p><br>
</p>
</body>
</html>