<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
The first thing to note is that lst reports results in binary units
<div>(MiB/s) while iperf reports results in decimal units (Gbps). If you do the</div>
<div>conversion you get 2055.31 MiB/s = 2155 MB/s.</div>
<div><br>
</div>
<div>The other thing to check is the CPU usage. For TCP the CPU usage can</div>
<div>be high. You should try RoCE+o2iblnd instead. <br>
<div><br>
<div id="AppleMailSignature" dir="ltr">Cheers, Andreas</div>
<div dir="ltr"><br>
On Nov 26, 2019, at 21:26, Pinkesh Valdria <<a href="mailto:pinkesh.valdria@oracle.com">pinkesh.valdria@oracle.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;
color:#272AD8;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Menlo;
color:#65798C;}
p.p3, li.p3, div.p3
{mso-style-name:p3;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Helvetica;}
p.p4, li.p4, div.p4
{mso-style-name:p4;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Menlo;
color:#D12F1B;}
span.s2
{mso-style-name:s2;
color:#272AD8;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello All, <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I created a new Lustre cluster on CentOS7.6 and I am running lnet_selftest_wrapper.sh to measure throughput on the network. The nodes are connected to each other using 25Gbps ethernet, so theoretical max
is 25 Gbps * 125 = 3125 MB/s. Using iperf3, I get 22Gbps (2750 MB/s) between the nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@lustre-client-2 ~]# for c in 1 2 4 8 12 16 20 24 ; do echo $c ; ST=lst-output-$(date +%Y-%m-%d-%H:%M:%S) CN=$c SZ=1M TM=30 BRW=write CKSUM=simple LFROM="10.0.3.7@tcp1" LTO="10.0.3.6@tcp1" /root/lnet_selftest_wrapper.sh;
done ;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">When I run lnet_selftest_wrapper.sh (from
<a href="http://wiki.lustre.org/LNET_Selftest">Lustre wiki</a>) between 2 nodes, I get a max of 2055.31 MiB/s, Is that expected at the Lnet level? Or can I further tune the network and OS kernel (tuning I applied are below) to get better throughput?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><u><span style="font-size:11.0pt">Result Snippet from lnet_selftest_wrapper.sh<o:p></o:p></span></u></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[LNet Rates of lfrom]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[R] Avg: 4112 RPC/s Min: 4112 RPC/s Max: 4112 RPC/s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[W] Avg: 4112 RPC/s Min: 4112 RPC/s Max: 4112 RPC/s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[LNet Bandwidth of lfrom]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[R] Avg: 0.31 MiB/s Min: 0.31 MiB/s Max: 0.31 MiB/s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;background:yellow;mso-highlight:yellow">[W] Avg: 2055.30 MiB/s Min: 2055.30 MiB/s Max: 2055.30 MiB/s</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[LNet Rates of lto]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[R] Avg: 4136 RPC/s Min: 4136 RPC/s Max: 4136 RPC/s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[W] Avg: 4136 RPC/s Min: 4136 RPC/s Max: 4136 RPC/s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[LNet Bandwidth of lto]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;background:yellow;mso-highlight:yellow">[R] Avg: 2055.31 MiB/s Min: 2055.31 MiB/s Max: 2055.31 MiB/s</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[W] Avg: 0.32 MiB/s Min: 0.32 MiB/s Max: 0.32 MiB/s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><u><span style="font-size:11.0pt">Tuning applied: <o:p></o:p></span></u></b></p>
<p class="MsoNormal"><u><span style="font-size:11.0pt">Ethernet NICs: <o:p></o:p></span></u></p>
<p class="p1">ip link set dev ens3 mtu <span class="s1">9000</span> <o:p></o:p></p>
<p class="p1">ethtool -G ens3 rx <span class="s1">2047</span> tx <span class="s1">
2047</span> rx-jumbo <span class="s1">8191</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><u><span style="font-size:11.0pt">less /etc/sysctl.conf<o:p></o:p></span></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.core.wmem_max=16777216<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.core.rmem_max=16777216<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.core.wmem_default=16777216<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.core.rmem_default=16777216<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.core.optmem_max=16777216<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.core.netdev_max_backlog=27000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">kernel.sysrq=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">kernel.shmmax=18446744073692774399<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.core.somaxconn=8192<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_adv_win_scale=2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_low_latency=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_rmem = 212992 87380 16777216<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_sack = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_timestamps = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_window_scaling = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_wmem = 212992 65536 16777216<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">vm.min_free_kbytes = 65536<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_congestion_control = cubic<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_timestamps = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_congestion_control = htcp<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net.ipv4.tcp_no_metrics_save = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">echo
</span><span style="font-size:9.0pt;font-family:Menlo;color:#D12F1B">"#</span><span style="font-size:9.0pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><i><span style="font-size:9.0pt;font-family:Menlo;color:#65798C"># tuned configuration</span></i><span style="font-size:9.0pt;font-family:Menlo;color:#65798C"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><i><span style="font-size:9.0pt;font-family:Menlo;color:#65798C">#</span></i><span style="font-size:9.0pt;font-family:Menlo;color:#65798C"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">[main]<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">summary=Broadly applicable tuning that provides excellent performance across a variety of common server workloads<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">[disk]<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">devices=!dm-*, !sda1, !sda2, !sda3<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">readahead=></span><span style="font-size:9.0pt;font-family:Menlo;color:#272AD8">4096</span><span style="font-size:9.0pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">[cpu]<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">force_latency=</span><span style="font-size:9.0pt;font-family:Menlo;color:#272AD8">1</span><span style="font-size:9.0pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">governor=performance<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">energy_perf_bias=performance<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">min_perf_pct=</span><span style="font-size:9.0pt;font-family:Menlo;color:#272AD8">100</span><span style="font-size:9.0pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">[vm]<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">transparent_huge_pages=never<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">[sysctl]<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">kernel.sched_min_granularity_ns =
</span><span style="font-size:9.0pt;font-family:Menlo;color:#272AD8">10000000</span><span style="font-size:9.0pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">kernel.sched_wakeup_granularity_ns =
</span><span style="font-size:9.0pt;font-family:Menlo;color:#272AD8">15000000</span><span style="font-size:9.0pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">vm.dirty_ratio =
</span><span style="font-size:9.0pt;font-family:Menlo;color:#272AD8">30</span><span style="font-size:9.0pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">vm.dirty_background_ratio =
</span><span style="font-size:9.0pt;font-family:Menlo;color:#272AD8">10</span><span style="font-size:9.0pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">vm.swappiness=</span><span style="font-size:9.0pt;font-family:Menlo;color:#272AD8">30</span><span style="font-size:9.0pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:#D12F1B">" > lustre-performance/tuned.conf<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:Menlo;color:black">tuned-adm profile lustre-performance<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Pinkesh Valdria<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr"><span>_______________________________________________</span><br>
<span>lustre-discuss mailing list</span><br>
<span><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a></span><br>
<span><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span><br>
</div>
</blockquote>
</div>
</div>
</body>
</html>