Apparently I didn't read your email very carefully since you were actually only wondering about LNET performance and not client performance. You can ignore the max_rpcs_in_flight for lnet. There used to be a srpc_peer_credits setting in lnet_selftest, not sure if it exists in your version. Should be able to find out with "modinfo lnet_selftest"<br>
<br><div class="gmail_quote">On Mon, Feb 20, 2012 at 7:43 PM, Jeremy Filizetti <span dir="ltr"><<a href="mailto:jeremy.filizetti@gmail.com">jeremy.filizetti@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Am I reading your earlier post correctly that you have a single server acting as the MDS and OSS? Have you changed your peer_credits and credits for ko2iblnd kernel module on the server and client? You also mentioned changing osc.*.max_dirty_mb, you probably need to adjust osc.*.max_rpcs_in_flight as well. Can you post your rpc stats "lctl get_param osc.*.rpc_stats"? I would guess they are bunching up around 7-8 if your running with the default max_rpcs_in_flight=8.<span class="HOEnZb"><font color="#888888"><br>
<br>Jeremy</font></span><div class="HOEnZb"><div class="h5"><br><br><br><div class="gmail_quote">On Mon, Feb 20, 2012 at 4:59 PM, Barberi, Carl E <span dir="ltr"><<a href="mailto:carl.e.barberi@lmco.com" target="_blank">carl.e.barberi@lmco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thank you. This did help. With the concurrency set to 16, I was able to get a max write speed of 1138 MB/s. Any ideas on how we can make that faster, though?
Ideally, we’d like to get to 1.5 GB/s.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Carl<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Liang Zhen [mailto:<a href="mailto:liang@whamcloud.com" target="_blank">liang@whamcloud.com</a>]
<br>
<b>Sent:</b> Thursday, February 16, 2012 1:45 AM<br>
<b>To:</b> Barberi, Carl E<br>
<b>Cc:</b> '<a href="mailto:lustre-discuss@lists.Lustre.org" target="_blank">lustre-discuss@lists.Lustre.org</a>'<br>
<b>Subject:</b> EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi, I assume you are using "size=1M" for brw test right? performance could increase if you set "concurrency" while adding brw test, i.e: --concurrency=16<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Liang<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><br>
<br>
<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">We are having issues with LNET performance over Infiniband. We have a configuration with a single MDT and six (6) OSTs. The Lustre client I am using to test is configured
to use 6 stripes (lfs setstripe -c 6 /mnt/lustre). When I perform a test using the following command:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s. When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when I run a series
of lnet-selftests, the read and write rates range from 850MB/s – 875MB/s max. I have performed the following optimizations to increase the data rate:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">On the Client:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">lctl set_param osc.*.checksums=0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">lctl set_param osc.*.max_dirty_mb=256<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">On the OSTs<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">lctl set_param obdfilter.*.writethrough_cache_enable=0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">lctl set_param obdfilter.*.read_cache_enable=0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">echo 4096 > /sys/block/<devices>/queue/nr_requests<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">I have also loaded the ib_sdp module, which also brought an increase in speed. However, we need to be able to record at no less than 1GB/s, which we cannot achieve right
now. Any thoughts on how I can optimize LNET, which clearly seems to be the bottleneck?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Thank you for any help you can provide,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Carl Barberi<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica","sans-serif"">_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<br>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></blockquote></div><br>
</div></div></blockquote></div><br>