<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1431510134;
mso-list-type:hybrid;
mso-list-template-ids:-777325858 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hmm – It’s possible you’ve got an issue, but I think more likely is that your chosen benchmarks aren’t capable of showing the higher speed.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m not really sure about your fio test - writing 4K random blocks will be relatively slow and might not speed up with more disks, but I can’t speak to it in detail for fio. I would try a much larger size and possibly more processes (is
numjobs the number of concurrent processes?)…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But I am sure about your other two:<br>
Both of those tests (dd and cp) are single threaded, and if they’re running to Lustre (rather than to the ZFS volume directly), 1.3 GB/s is around the maximum expected speed. On a recent Xeon, one process can write a maximum of about 1-1.5 GB/s to Lustre,
depending on various details. Improving disk speed won’t affect that limit for one process, it’s a client side thing. Try several processes at once, ideally from multiple clients (and definitely writing to multiple files), if you really want to see your
OST bandwidth limit.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Also, a block size of 10GB is *<b>way</b>* too big for DD and will harm performance. It’s going to cause slowdown vs a smaller block size, like 16M or something.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There’s also limit on how fast /dev/zero can be read, especially with really large block sizes [it cannot provide 10 GiB of zeroes at a time, that’s why you had to add the “fullblock” flag, which is doing multiple reads (and writes)].
Here’s a quick sample on a system here, writing to /dev/null (so there is no real limit on the write bandwidth of the destination):<o:p></o:p></p>
<p class="MsoNormal">dd if=/dev/zero bs=10G of=/dev/null count=1<o:p></o:p></p>
<p class="MsoNormal">0+1 records in<o:p></o:p></p>
<p class="MsoNormal">0+1 records out<o:p></o:p></p>
<p class="MsoNormal">2147479552 bytes (2.1 GB, 2.0 GiB) copied, 1.66292 s, 1.3 GB/s<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Notice that 1.3 GB/s, the same as your result.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Try 16M instead:<o:p></o:p></p>
<p class="MsoNormal">saturn-p2:/lus # dd if=/dev/zero bs=16M of=/dev/null count=1024<o:p></o:p></p>
<p class="MsoNormal">1024+0 records in<o:p></o:p></p>
<p class="MsoNormal">1024+0 records out<o:p></o:p></p>
<p class="MsoNormal">17179869184 bytes (17 GB, 16 GiB) copied, 7.38402 s, 2.3 GB/s<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Also note that multiple dds reading from /dev/zero will run in to issues with the bandwidth of /dev/zero. /dev/zero is different than most people assume – One would think it just magically spews zeroes at any rate needed, but it’s not
really designed to be read at high speed and actually isn’t that fast. If you really want to test high speed storage, you may need a tool that allocates memory and writes that out, not just dd. (ior is one example)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Zeeshan Ali Shah <javaclinic@gmail.com><br>
<b>Date: </b>Tuesday, August 28, 2018 at 9:52 AM<br>
<b>To: </b>Patrick Farrell <paf@cray.com><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] separate SSD only filesystem including HDD<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1) fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=20G --numjobs=4 --runtime=240 --group_reporting
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">2) time cp x x2<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">3) and dd if=/dev/zero of=/ssd/d.data bs=10G count=4 iflag=fullblock<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">any other way to test this plz let me know<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">/Zee<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Aug 28, 2018 at 3:54 PM Patrick Farrell <<a href="mailto:paf@cray.com">paf@cray.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">How are you measuring write speed?<br>
<br>
<o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="m_3679768992855847995divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Zeeshan Ali Shah
<<a href="mailto:javaclinic@gmail.com" target="_blank">javaclinic@gmail.com</a>><br>
<b>Sent:</b> Tuesday, August 28, 2018 1:30:03 AM<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> [lustre-discuss] separate SSD only filesystem including HDD</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Dear All, I recently deployed 10PB+ Lustre solution which is working fine. Recently for genomic pipeline we acquired another racks with dedicated compute nodes with single 24-NVME SSD Servers/Rack . Each SSD server connected to Compute
Node via 100 G Omnipath. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Issue 1: is that when I combined SSDs in stripe mode using zfs we are not linearly scaling in terms of performance . for e..g single SSD write speed is 1.3GB/sec , adding 5 of those in stripe mode should give us atleast 1.3x5 or less
, but we still get 1.3 GB out of those 5 SSD .<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Issue 2: if we resolve issue #1, 2nd challenge is to allow 24 NVMEs to compute nodes distributed and parallel wise , NFS not an option .. tried glusterfs but due to its DHT it is slow.. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I am thinking to add another Filesystem to our existing MDT and install OSTs/OSS over the NVME server.. mounting this specific ssd where needed. so basically we will end up having two filesystem (one with normal 10PB+ and 2nd with SSD).. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
Does this sounds correct ? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">any other advice please ..<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">/Zeeshan<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>