<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p><br>
</p>
<div>Hi Patrick,</div>
<div><br>
</div>
<div>Excuse me for the huge delay on replying to your e-mail. I just logged in to the web interface of my e-mail account and spotted that most of the e-mails from the Lustre mailing list ended-up on the SPAM folder. So, first, my apologies for this and thank
you very much for your reply.</div>
<div><br>
</div>
<div>I indeed tested back in the day to use MPI I/O for read-only and write-only, and also mmap to evaluate the same tests. I noticed, specially with mmap, that the speeds were asymmetric. "Reads" are clearly using the page cache (or some equivalent buffer
in memory), providing an average of 12000MB/s of bandwidth after the first initialization. "Writes" seem to force network all the time and provide an average of around 1000MB/s, specially for mapped files over 1MB.</div>
<div><br>
</div>
<div>Talking to the people that maintain our cluster, they assumed that the Lustre client is pre-configured to flush data as soon as possible to compensate for bandwidth differences and guarantee data consistency. However, my goal is actually to avoid this
situation and cache as much as possible on the node for data reuse, which gives a performance boost (whether data is consistent with storage or not is another story, but for the use case we are proposing, it's not critical).</div>
<div><br>
</div>
<div>Do you have any additional hints that could help given this information?</div>
<div><br>
</div>
<div>Once again, thank you very much for your reply and excuse me for the delay.</div>
<div><br>
</div>
<div>Kind Regards,</div>
<div>Sergio.</div>
<div><br>
</div>
P.S.: <span style="font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">By the way, </span><span style="font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">I </span><span style="font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">noticed</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;"> that
you work for Cray. We are mostly using our Cray XC40 (<a href="https://www.pdc.kth.se/resources/computers/beskow" class="OWAAutoLink" id="LPlnk188048" previewremoved="true">https://www.pdc.kth.se/resources/computers/beskow</a>) for our research.</span><br>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>De:</b> Patrick Farrell <paf@cray.com><br>
<b>Enviado:</b> domingo, 29 de enero de 2017 22:47<br>
<b>Para:</b> sergio.rivas@um.es; lustre-discuss@lists.lustre.org<br>
<b>Asunto:</b> Re: [lustre-discuss] Write performance low</font>
<div> </div>
</div>
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Sergio,</p>
<p><br>
</p>
<p>In general, Lustre always writes asynchronously, unless explicitly told to do otherwise (direct I/O or O_SYNC). I don't have much experience with memory mapped files - it's possible that they somehow force synchronous behavior... But my point, I suppose,
is that there's probably not a tunable for you. Try - just to see - writes to a normal file (IE, not memory mapped), so you can confirm. Someone on list will probably have a better idea about writes to mmap'ed files and what to expect.</p>
<p><br>
</p>
<p>- Patrick</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of sergio.rivas@um.es <sergio.rivas@um.es><br>
<b>Sent:</b> Sunday, January 29, 2017 8:15:00 AM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> [lustre-discuss] Write performance low</font>
<div> </div>
</div>
<div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
</p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
Good afternoon,</p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
</p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
I’m currently dev<span lang="EN-US">eloping a small library that allows to easily perform an mmap that targets files in storage, and so far the results have been quite positive in local tests. However, trying out the same code on our cluster that uses Lustre
as PFS, I have noticed that the reads are conveniently cached but that the writes seem to be synchronously flushed to storage, hence, decreasing the perceived performance (despite this fact being technically correct from a data-consistency point of view).</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US">My goal is to allow for the page cache of the OS to keep as much as dirty pages as possible with the purpose of aggregating write operations for free, but I’m afraid that I haven’t figured out how to solve this with Lustre after playing around
with some of the settings that I found within the manual.</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US">Could you, please, point out if there is any tunable setting (e.g., under /proc) that allow me to increase the caching or avoid direct flushing?</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US">Thank you very much in advance.</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US">Kind Regards,</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US">Sergio.</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US">P.S.: I think my previous post didn’t go through, so I’m sending it again. Excuse me if it did!</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US"> </span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>