<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body dir="auto">
Unless you have huge directories, you may not see any improvement from DNE, and it may hurt performance because striped directories have more overhead when they are first created.
<div><br>
</div>
<div>DNE is mostly useful when a single MDS is overloaded by many clients, but with the small IO workload here that may not be the case. </div>
<div><br>
</div>
<div>Also, you would likely benefit from IB networking, which is lower latency compared to TCP. <br>
<div><br>
<div id="AppleMailSignature">Cheers, Andreas</div>
<div><br>
On May 29, 2018, at 17:26, meng ding <<a href="mailto:dingmeng@gmail.com">dingmeng@gmail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>We are in the process of helping a client to evaluate share file system solutions for their distributed build use case. Our client is a company with hundreds of developers around the globe doing system software development 24 hours. At anytime, there could
be many builds running, including interactive builds (mostly incremental build), or batch builds (mostly regression tests).</div>
<div><br>
</div>
<div>The size of the build is very big. For example, a single full build may have more than 6000 build tasks (i.e., make rules) that can all be run in parallel. Each build task takes about 6 seconds on average to run. So a sequential build using 1 CPU (e.g.,
make -j 1) will take 10 hours to complete. </div>
<div><br>
</div>
<div>Our client is using a distributed build software to run the build across a cluster of build hosts. Think of (make -j N), only the build tasks are running simultaneously on multiple hosts, instead of one. Obviously for this to work, they need to have a
shared file system with good performance. Our client is currently using NFS on NetApp, which most of the time provides good performance, but at a very high cost. With this combination, our client is able to complete the above mentioned build in less than 5
minutes in a build cluster with about 30 build hosts (25 cores per host). Another advantage of using a cluster of build hosts is to accommodate many builds from multiple developers at the same time, with each developer dynamically assigned a fair share of
the total cores in the build cluster at any given time based on the resource requirement of each build.</div>
<div><br>
</div>
<div>The distributed build use case has the following characteristics:</div>
<div>
<ul>
<li>Mostly very small source files (tens of thousands of them) less than 16K to start with.</li><li>Source files are read-only. <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">
All reads are sequential.</span> </li><li>Source files are read repetitively (e.g., the header files). So it can benefit hugely from client-side caching.</li><li>Intermediate object files, libraries, or binary files are small to medium in size, the biggest binary generated is about several hundred megabytes. </li><li>Binary/object files are generated by <b>small random writes</b>.</li><li>There is NO concurrent/shared access to the same file. Each build task generates its own output file.</li></ul>
</div>
<div>With this use case in mind, we are trying to explore alternative solutions to NFS on NetApp with the goal to achieve comparable performance with reduced cost. So far, we have done some benchmark with Lustre on distributed build of GCC 8.1 on AWS, but the
performance is lagging quite a bit behind even kernel NFS:<br>
</div>
<div><br>
</div>
<div>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<b>Lustre Setup</b></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span style="color:rgb(34,34,34);font-family:Calibri,sans-serif;font-size:11pt;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><b><br>
</b></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span style="color:rgb(34,34,34);font-family:Calibri,sans-serif;font-size:11pt;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><u>Lustre
Server</u></span><br>
</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
</p>
<ul>
<li>2 MDS each has m5.2xlarge instance (8 vCPUS, 32GiB Mem, up to 10Gb network), backed by 80 GiB SSD formated with LDISKFS.</li><li>DNE phase II (striped directory) is enabled.<br>
</li><li>No data striping is enabled because most files are small.<br>
</li><li>4 OSS each has m5.xlarge instance (<span style="font-size:14.6667px">4 vCPUS, 16GiB Mem, up to 10Gb network)</span><span style="font-size:11pt"> , backed by 40 GiB SSD formated with LDISKFS.</span></li></ul>
<p></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span style="font-size:11pt"><u>Build cluster</u></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
30 build hosts m5.xlarge, 120 CPUs in total all mounting the same Lustre volume<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
</p>
<ul style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">
<li>The following is configured on all build hosts:</li></ul>
<p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">
</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(34,34,34);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;font-size:11pt;font-family:Calibri,sans-serif">
<span style="font-size:11pt"></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;color:rgb(34,34,34);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;line-height:15.6933px;font-size:11pt;font-family:Calibri,sans-serif">
<i>mount -t lustre -o localflock …</i><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;color:rgb(34,34,34);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;line-height:15.6933px;font-size:11pt;font-family:Calibri,sans-serif">
<i>lctl set_param osc./*.checksums=0<span></span></i></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;color:rgb(34,34,34);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;line-height:15.6933px;font-size:11pt;font-family:Calibri,sans-serif">
<i>lctl set_param osc./*.max_rpcs_in_flight=32<span></span></i></p>
<i style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:11pt;line-height:15.6933px;font-family:Calibri,sans-serif">lctl
set_param osc./*.max_dirty_mb=128</span></i><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span> </span></span>
<br>
<p></p>
<div><br>
</div>
<div><u>Test and results</u></div>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
Running distributed build of GCC 8.1 in the Lustre mount across the build cluster:</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<br>
</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
Launching 1 build only:<span></span></p>
<ul style="margin-top:0in;margin-bottom:0in" type="disc">
<li class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
Takes on average <b>17 minutes 45</b> seconds to finish.<br>
</li></ul>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
Launching 20 builds at the same time all sharing the same build cluster:<span></span></p>
<ul style="margin-top:0in;margin-bottom:0in" type="disc">
<li class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
Takes on average <b>46 minutes</b> to finish for each build.<br>
</li></ul>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span>By the way, we have tried the Data-on-MDT feature since we are using Lustre 2.11, but we did not observe performance improvement.</span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<b>Kernel NFS Setup</b><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<u><br>
</u></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<u>NFS Server</u></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
1 NFS server m5.2xlarge <span style="color:rgb(34,34,34);font-family:Calibri,sans-serif;font-size:14.6667px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">
(8 vCPUS, 32GiB Mem, up to 10Gb network), backed by 300 GiB SSD formatted with XFS</span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(34,34,34);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;font-size:11pt;font-family:Calibri,sans-serif">
<u>Build cluster</u><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(34,34,34);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;font-size:11pt;font-family:Calibri,sans-serif">
30 build hosts m5.xlarge, 120 CPUs in total all mounting the same NFS volume using NFS v3 protocol.</p>
<div><br>
</div>
<u>Test and results</u></div>
<div><span style="color:rgb(34,34,34);font-family:Calibri,sans-serif;font-size:14.6667px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline">Running
distributed build of GCC 8.1 in the NFS mount across the build cluster:</span> <br>
<p></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(34,34,34);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;font-size:11pt;font-family:Calibri,sans-serif">
Launching 1 build only:<span></span></p>
<ul type="disc" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;margin-top:0in;margin-bottom:0in">
<li class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
Takes on average <b>16 minutes 36 seconds</b> to finish. About 1 minute faster than Lustre.<br>
</li></ul>
<br class="gmail-Apple-interchange-newline">
<span style="color:rgb(34,34,34);font-family:Calibri,sans-serif;font-size:14.6667px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Launching
20 builds at the same time all sharing the same build cluster:</span> <br>
<p></p>
<ul style="margin-top:0in;margin-bottom:0in" type="disc">
<li class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
Takes on average <b>38 minutes</b> to finish for each build. About 8 minutes faster than Lustre.</li></ul>
<br>
</div>
<div><br>
</div>
<div>So our question to the Lustre experts, given the distributed build use, case do you suggest anything else that we can try to potentially improve the performance further?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>ading</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>lustre-discuss mailing list</span><br>
<span><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a></span><br>
<span><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span><br>
</div>
</blockquote>
</div>
</div>
</body>
</html>