<div dir="ltr">Hi Brett,<div><br></div><div style>Sorry, I think my choice in wording is not correct.</div><div style><br></div><div style>One VM is holding the metadata and the objects. I guess that would mean ti is the OSS and MDS?</div>
<div style>Another VM is the client.It has mounted the lusture filesystem and also has some local disks. The test is to just to use cp to read data to local disk.</div><div style><br></div><div style>Thanks,</div><div style>
Andy</div><div style><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jun 21, 2013 at 3:22 PM, Lee, Brett <span dir="ltr"><<a href="mailto:brett.lee@intel.com" target="_blank">brett.lee@intel.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">“test between 2 virtual machines from MDT/OST server to client's local disk.”<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Andrew,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I’m confused by the description of your test. Can you clarify?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#1f497d">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#1f497d">Brett Lee<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#1f497d">Sr. Systems Engineer<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#1f497d">Intel High Performance Data Division<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a> [mailto:<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>]
<b>On Behalf Of </b>Andrew Mast<br>
<b>Sent:</b> Friday, June 21, 2013 3:42 PM<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> [Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?<u></u><u></u></span></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hello, I am new to Lustre and wanted to run a small simple small copy test between 2 virtual machines from MDT/OST server to client's local disk.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I realize small file performance is never fast, but this seems particularly slow considering the data is all buffered in memory with little to no disk activity.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Setup Info<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Version is 2.4.50<u></u><u></u></p>
<div>
<p class="MsoNormal">Average file size is small. < 10KB<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The amount of data being copied is about 250MB.<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">The VMs are on separate hosts.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Performance<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">7 minutes over a gigabit network. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">NFS takes only 3 minutes.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Observations<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">iostat on the OST/MDT is usually 0% during the copy. Assuming all buffered.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Additional network traffic is minimal. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">CPU load on the VMs is 15-20% during copy.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">RPC stats on the client shows only 1 RPC in flight at a time. max inflight is set to 64. Is that expected behavior for a copy?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Here is a snapshot of rpc_stats early during the copy:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> read write<u></u><u></u></p>
<p class="MsoNormal">pages per rpc rpcs % cum % | rpcs % cum %<u></u><u></u></p>
<p class="MsoNormal">1: 1653 90 90 | 0 0 0<u></u><u></u></p>
<p class="MsoNormal">2: 164 8 98 | 0 0 0<u></u><u></u></p>
<p class="MsoNormal">4: 7 0 99 | 0 0 0<u></u><u></u></p>
<p class="MsoNormal">8: 3 0 99 | 0 0 0<u></u><u></u></p>
<p class="MsoNormal">16: 3 0 99 | 0 0 0<u></u><u></u></p>
<p class="MsoNormal">32: 5 0 99 | 0 0 0<u></u><u></u></p>
<p class="MsoNormal">64: 0 0 99 | 0 0 0<u></u><u></u></p>
<p class="MsoNormal">128: 1 0 100 | 0 0 0<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> read write<u></u><u></u></p>
<p class="MsoNormal">rpcs in flight rpcs % cum % | rpcs % cum %<u></u><u></u></p>
<p class="MsoNormal">0: 0 0 0 | 0 0 0<u></u><u></u></p>
<p class="MsoNormal">1: 1836 100 100 | 0 0 0<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> read write<u></u><u></u></p>
<p class="MsoNormal">offset rpcs % cum % | rpcs % cum %<u></u><u></u></p>
<p class="MsoNormal">0: 1836 100 100 | 0 0 0<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">As I am new, any suggestions for what to look for or improve would be greatly appreciated.<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></blockquote></div><br></div>