<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Patrick</div><div id="AppleMailSignature">I thought at one time there was an inode lock held for the duration of the direct I/O read or write. So that even if one had multiple application threads writing direct, only one was "in flight" at a time. Has that changed?</div><div id="AppleMailSignature">John<br><br>Sent from my iPhone</div><div><br>On Oct 14, 2016, at 3:16 PM, Patrick Farrell <<a href="mailto:paf@cray.com">paf@cray.com</a>> wrote:<br><br></div><blockquote type="cite"><div>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Sorry, I phrased one thing wrong:<br>
I said "transferring to the network", but it's actually until it's received confirmation the data has been received successfully, I believe.</p>
<p><br>
</p>
<p>In any case, only one I/O (per thread) can be outstanding at a time with direct I/O.<br>
</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Patrick Farrell <<a href="mailto:paf@cray.com">paf@cray.com</a>><br>
<b>Sent:</b> Friday, October 14, 2016 3:12:22 PM<br>
<b>To:</b> Riccardo Veraldi; <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> Re: [lustre-discuss] Lustre on ZFS pooer direct I/O performance</font>
<div> </div>
</div>
<div>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Riccardo,</p>
<p><br>
</p>
<p>While the difference is extreme, direct I/O write performance will always be poor. Direct I/O writes cannot be asynchronous, since they don't use the page cache. This means Lustre cannot return from one write (and start the next) until it has finished
transferring the data to the network.</p>
<p><br>
</p>
<p>This means you can only have one I/O in flight at a time. Good write performance from Lustre (or any network filesystem) depends on keeping a lot of data in flight at once.</p>
<p><br>
</p>
<p>What sort of direct write performance were you hoping for? It will never match that 800 MB/s from one thread you see with buffered I/O.</p>
<p><br>
</p>
<p>- Patrick<br>
</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Riccardo Veraldi <<a href="mailto:Riccardo.Veraldi@cnaf.infn.it">Riccardo.Veraldi@cnaf.infn.it</a>><br>
<b>Sent:</b> Friday, October 14, 2016 2:22:32 PM<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> [lustre-discuss] Lustre on ZFS pooer direct I/O performance</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">Hello,<br>
<br>
I would like how may I improve the situation of my lustre cluster.<br>
<br>
I have 1 MDS and 1 OSS with 20 OST defined.<br>
<br>
Each OST is a 8x Disks RAIDZ2.<br>
<br>
A single process write performance is around 800MB/sec<br>
<br>
anyway if I force direct I/O, for example using oflag=direct in dd, the <br>
write performance drop as low as 8MB/sec<br>
<br>
with 1MB block size. And each write it's about 120ms latency.<br>
<br>
I used these ZFS settings<br>
<br>
options zfs zfs_prefetch_disable=1<br>
options zfs zfs_txg_history=120<br>
options zfs metaslab_debug_unload=1<br>
<br>
i am quite worried for the low performance.<br>
<br>
Any hints or suggestions that may help me to improve the situation ?<br>
<br>
<br>
thank you<br>
<br>
<br>
Rick<br>
<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div>
</span></font></div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>lustre-discuss mailing list</span><br><span><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a></span><br><span><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span><br><span></span><br></div></blockquote></body></html>