<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">On the second experiment, you’re writing a total of 1000MB and reading 100MB. It could simply be that you’re not putting enough load on the system for long enough to get full performance.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Ben Evans<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Colin Faber via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Reply-To: </b>Colin Faber <cfaber@gmail.com><br>
<b>Date: </b>Monday, September 27, 2021 at 9:50 PM<br>
<b>To: </b>Nagmat Nazarov <nagmat@nevada.unr.edu><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Why reads are slower than writes on lustre file system?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Depending on your IO workloads reads can be slower than writes in cases where the writes may be sequential and optimized for the backing storage hardware, and the reads random, or semi-random. This also can be greatly affected by block
allocator efficiency. Typically on well tuned modern lustre file systems reads and writes are very similar (if not better for reads as Paf mentions).<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Mon, Sep 27, 2021 at 5:30 PM Nagmat Nazarov <<a href="mailto:nagmat@nevada.unr.edu">nagmat@nevada.unr.edu</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">My backing file system is ldiskfs. <o:p></o:p></p>
<div>
<p class="MsoNormal">The storage hardware is HDD I guess(Since I am using emulab cloud storage)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I am doing buffered I/O. <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Mon, Sep 27, 2021 at 4:20 PM Patrick Farrell <<a href="mailto:pfarrell@ddn.com" target="_blank">pfarrell@ddn.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">It does depend on your storage hardware, but modern Lustre software is generally the same or faster for reads.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What is your backing file system - ldiskfs or ZFS? And what is the storage hardware? And are you doing direct or buffered I/O?<o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="gmail-m_8911768358528354338gmail-m_-6075350018947999357divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Nagmat Nazarov <<a href="mailto:nagmat@nevada.unr.edu" target="_blank">nagmat@nevada.unr.edu</a>><br>
<b>Sent:</b> Monday, September 27, 2021 5:57 PM<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a> <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Subject:</b> [lustre-discuss] Why reads are slower than writes on lustre file system?</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Dear Engineers, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I have started working on a lustre file system. I have done couple of experiments so far: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">On the first experiment I am writing 100 files each 10MB and the batch size is 4K. I got 107MB/s bandwidth. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On the second experiment I am (writing 10 files each 10MB and reading 1 10 MB file back) for 10 times. Here also the average bandwidth is 106MB/s while read average bandwidth is 59MB/s which is very weird I guess? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">My question is, generally on ext4 file systems read average bandwidth is faster that write average bandwidth, why is it 2 times slower on lustre file system. My ethernet speed is 1Gb/s. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Kind regards<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Nagmat Nazarov<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal">_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>