<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Menlo;
color:black;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Menlo;
color:#65798C;}
p.p3, li.p3, div.p3
{mso-style-name:p3;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Helvetica;}
p.p4, li.p4, div.p4
{mso-style-name:p4;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Menlo;
color:#D12F1B;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.s1
{mso-style-name:s1;
color:#272AD8;}
span.s2
{mso-style-name:s2;
color:#272AD8;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle26
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt'>Thanks Andreas for your response. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>I ran anotherLnet Self test with 48 concurrent processes, since the nodes have 52 physical cores and I was able to achieve same throughput (</span><span style='font-size:9.0pt;font-family:Menlo;color:black'>2052.71 MiB/s = 2152 MB/s</span><span style='font-size:11.0pt'>).</span><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Is it expected to lose almost 600 MB/s (2750-2150= ) due to overheads on ethernet with Lnet?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Pinkesh Valdria<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Oracle Cloud Infrastructure <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='color:black'>From: </span></b><span style='color:black'>Andreas Dilger <adilger@whamcloud.com><br><b>Date: </b>Wednesday, November 27, 2019 at 1:25 AM<br><b>To: </b>Pinkesh Valdria <pinkesh.valdria@oracle.com><br><b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br><b>Subject: </b>Re: [lustre-discuss] Lnet Self Test<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><p class=MsoNormal>The first thing to note is that lst reports results in binary units <o:p></o:p></p><div><p class=MsoNormal>(MiB/s) while iperf reports results in decimal units (Gbps). If you do the<o:p></o:p></p></div><div><p class=MsoNormal>conversion you get 2055.31 MiB/s = 2155 MB/s.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>The other thing to check is the CPU usage. For TCP the CPU usage can<o:p></o:p></p></div><div><p class=MsoNormal>be high. You should try RoCE+o2iblnd instead. <o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p><div id=AppleMailSignature><p class=MsoNormal>Cheers, Andreas<o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><br>On Nov 26, 2019, at 21:26, Pinkesh Valdria <<a href="mailto:pinkesh.valdria@oracle.com">pinkesh.valdria@oracle.com</a>> wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal><span style='font-size:11.0pt'>Hello All, </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>I created a new Lustre cluster on CentOS7.6 and I am running lnet_selftest_wrapper.sh to measure throughput on the network. The nodes are connected to each other using 25Gbps ethernet, so theoretical max is 25 Gbps * 125 = 3125 MB/s. Using iperf3, I get 22Gbps (2750 MB/s) between the nodes.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[root@lustre-client-2 ~]# for c in 1 2 4 8 12 16 20 24 ; do echo $c ; ST=lst-output-$(date +%Y-%m-%d-%H:%M:%S) CN=$c SZ=1M TM=30 BRW=write CKSUM=simple LFROM="10.0.3.7@tcp1" LTO="10.0.3.6@tcp1" /root/lnet_selftest_wrapper.sh; done ;</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>When I run lnet_selftest_wrapper.sh (from <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.lustre.org_LNET-5FSelftest&d=DwMGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=HpfvG0tozSl7HgJJuyxxo2149EjwqpQDE7ytv-4sZuI&m=dEosA07cQm7WPohubrpzab8agc4uFDGesC-4tI4ylm0&s=-ne2Yke64JRw4BQu9pa0DXwf3tHkDqaUbp7S6Eq_C_Q&e=">Lustre wiki</a>) between 2 nodes, I get a max of 2055.31 MiB/s, Is that expected at the Lnet level? Or can I further tune the network and OS kernel (tuning I applied are below) to get better throughput?</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><b><u><span style='font-size:11.0pt'>Result Snippet from lnet_selftest_wrapper.sh</span></u></b><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[LNet Rates of lfrom]</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[R] Avg: 4112 RPC/s Min: 4112 RPC/s Max: 4112 RPC/s</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[W] Avg: 4112 RPC/s Min: 4112 RPC/s Max: 4112 RPC/s</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[LNet Bandwidth of lfrom]</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[R] Avg: 0.31 MiB/s Min: 0.31 MiB/s Max: 0.31 MiB/s</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;background:yellow;mso-highlight:yellow'>[W] Avg: 2055.30 MiB/s Min: 2055.30 MiB/s Max: 2055.30 MiB/s</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[LNet Rates of lto]</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[R] Avg: 4136 RPC/s Min: 4136 RPC/s Max: 4136 RPC/s</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[W] Avg: 4136 RPC/s Min: 4136 RPC/s Max: 4136 RPC/s</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[LNet Bandwidth of lto]</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;background:yellow;mso-highlight:yellow'>[R] Avg: 2055.31 MiB/s Min: 2055.31 MiB/s Max: 2055.31 MiB/s</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>[W] Avg: 0.32 MiB/s Min: 0.32 MiB/s Max: 0.32 MiB/s</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><b><u><span style='font-size:11.0pt'>Tuning applied: </span></u></b><o:p></o:p></p><p class=MsoNormal><u><span style='font-size:11.0pt'>Ethernet NICs: </span></u><o:p></o:p></p><p class=p1>ip link set dev ens3 mtu <span class=s1>9000</span> <o:p></o:p></p><p class=p1>ethtool -G ens3 rx <span class=s1>2047</span> tx <span class=s1>2047</span> rx-jumbo <span class=s1>8191</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><u><span style='font-size:11.0pt'>less /etc/sysctl.conf</span></u><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.wmem_max=16777216</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.rmem_max=16777216</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.wmem_default=16777216</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.rmem_default=16777216</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.optmem_max=16777216</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.netdev_max_backlog=27000</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>kernel.sysrq=1</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>kernel.shmmax=18446744073692774399</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.somaxconn=8192</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_adv_win_scale=2</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_low_latency=1</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_rmem = 212992 87380 16777216</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_sack = 1</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_timestamps = 1</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_window_scaling = 1</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_wmem = 212992 65536 16777216</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>vm.min_free_kbytes = 65536</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_congestion_control = cubic</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_timestamps = 0</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_congestion_control = htcp</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_no_metrics_save = 0</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>echo </span><span style='font-size:9.0pt;font-family:Menlo;color:#D12F1B'>"#</span><o:p></o:p></p><p class=MsoNormal style='background:white'><i><span style='font-size:9.0pt;font-family:Menlo;color:#65798C'># tuned configuration</span></i><o:p></o:p></p><p class=MsoNormal style='background:white'><i><span style='font-size:9.0pt;font-family:Menlo;color:#65798C'>#</span></i><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>[main]</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>summary=Broadly applicable tuning that provides excellent performance across a variety of common server workloads</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'> </span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>[disk]</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>devices=!dm-*, !sda1, !sda2, !sda3</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>readahead=></span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>4096</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Helvetica;color:black'> </span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>[cpu]</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>force_latency=</span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>1</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>governor=performance</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>energy_perf_bias=performance</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>min_perf_pct=</span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>100</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>[vm]</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>transparent_huge_pages=never</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>[sysctl]</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>kernel.sched_min_granularity_ns = </span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>10000000</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>kernel.sched_wakeup_granularity_ns = </span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>15000000</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>vm.dirty_ratio = </span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>30</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>vm.dirty_background_ratio = </span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>10</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>vm.swappiness=</span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>30</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:#D12F1B'>" > lustre-performance/tuned.conf</span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Helvetica;color:black'> </span><o:p></o:p></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>tuned-adm profile lustre-performance</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>Thanks,</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>Pinkesh Valdria</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p></div></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal><span style='font-size:11.0pt'>_______________________________________________<br>lustre-discuss mailing list<br><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwMGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=HpfvG0tozSl7HgJJuyxxo2149EjwqpQDE7ytv-4sZuI&m=dEosA07cQm7WPohubrpzab8agc4uFDGesC-4tI4ylm0&s=ejwMDqk5D3TzRE5eTzFdEKo9cQ0I6GVqN04wgaJcn0s&e=">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></span></p></div></blockquote></div></div></div></body></html>