<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;
color:#272AD8;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Menlo;
color:#65798C;}
p.p3, li.p3, div.p3
{mso-style-name:p3;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Helvetica;}
p.p4, li.p4, div.p4
{mso-style-name:p4;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:9.0pt;
font-family:Menlo;
color:#D12F1B;}
span.s2
{mso-style-name:s2;
color:#272AD8;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt'>Hello All, <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>I created a new Lustre cluster on CentOS7.6 and I am running lnet_selftest_wrapper.sh to measure throughput on the network. The nodes are connected to each other using 25Gbps ethernet, so theoretical max is 25 Gbps * 125 = 3125 MB/s. Using iperf3, I get 22Gbps (2750 MB/s) between the nodes.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>[root@lustre-client-2 ~]# for c in 1 2 4 8 12 16 20 24 ; do echo $c ; ST=lst-output-$(date +%Y-%m-%d-%H:%M:%S) CN=$c SZ=1M TM=30 BRW=write CKSUM=simple LFROM="10.0.3.7@tcp1" LTO="10.0.3.6@tcp1" /root/lnet_selftest_wrapper.sh; done ;<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>When I run lnet_selftest_wrapper.sh (from <a href="http://wiki.lustre.org/LNET_Selftest">Lustre wiki</a>) between 2 nodes, I get a max of 2055.31 MiB/s, Is that expected at the Lnet level? Or can I further tune the network and OS kernel (tuning I applied are below) to get better throughput?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><b><u><span style='font-size:11.0pt'>Result Snippet from lnet_selftest_wrapper.sh<o:p></o:p></span></u></b></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'> [LNet Rates of lfrom]<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>[R] Avg: 4112 RPC/s Min: 4112 RPC/s Max: 4112 RPC/s<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>[W] Avg: 4112 RPC/s Min: 4112 RPC/s Max: 4112 RPC/s<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>[LNet Bandwidth of lfrom]<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>[R] Avg: 0.31 MiB/s Min: 0.31 MiB/s Max: 0.31 MiB/s<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;background:yellow;mso-highlight:yellow'>[W] Avg: 2055.30 MiB/s Min: 2055.30 MiB/s Max: 2055.30 MiB/s</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>[LNet Rates of lto]<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>[R] Avg: 4136 RPC/s Min: 4136 RPC/s Max: 4136 RPC/s<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>[W] Avg: 4136 RPC/s Min: 4136 RPC/s Max: 4136 RPC/s<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>[LNet Bandwidth of lto]<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;background:yellow;mso-highlight:yellow'>[R] Avg: 2055.31 MiB/s Min: 2055.31 MiB/s Max: 2055.31 MiB/s</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>[W] Avg: 0.32 MiB/s Min: 0.32 MiB/s Max: 0.32 MiB/s<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><b><u><span style='font-size:11.0pt'>Tuning applied: <o:p></o:p></span></u></b></p><p class=MsoNormal><u><span style='font-size:11.0pt'>Ethernet NICs: <o:p></o:p></span></u></p><p class=p1>ip link set dev ens3 mtu <span class=s1>9000</span> <o:p></o:p></p><p class=p1>ethtool -G ens3 rx <span class=s1>2047</span> tx <span class=s1>2047</span> rx-jumbo <span class=s1>8191</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><u><span style='font-size:11.0pt'>less /etc/sysctl.conf<o:p></o:p></span></u></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.wmem_max=16777216<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.rmem_max=16777216<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.wmem_default=16777216<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.rmem_default=16777216<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.optmem_max=16777216<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.netdev_max_backlog=27000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>kernel.sysrq=1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>kernel.shmmax=18446744073692774399<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.core.somaxconn=8192<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_adv_win_scale=2<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_low_latency=1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_rmem = 212992 87380 16777216<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_sack = 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_timestamps = 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_window_scaling = 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_wmem = 212992 65536 16777216<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>vm.min_free_kbytes = 65536<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_congestion_control = cubic<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_timestamps = 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_congestion_control = htcp<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>net.ipv4.tcp_no_metrics_save = 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>echo </span><span style='font-size:9.0pt;font-family:Menlo;color:#D12F1B'>"#</span><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><i><span style='font-size:9.0pt;font-family:Menlo;color:#65798C'># tuned configuration</span></i><span style='font-size:9.0pt;font-family:Menlo;color:#65798C'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><i><span style='font-size:9.0pt;font-family:Menlo;color:#65798C'>#</span></i><span style='font-size:9.0pt;font-family:Menlo;color:#65798C'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>[main]<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>summary=Broadly applicable tuning that provides excellent performance across a variety of common server workloads<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p> </o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>[disk]<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>devices=!dm-*, !sda1, !sda2, !sda3<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>readahead=></span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>4096</span><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Helvetica'><o:p> </o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>[cpu]<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>force_latency=</span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>1</span><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>governor=performance<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>energy_perf_bias=performance<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>min_perf_pct=</span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>100</span><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>[vm]<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>transparent_huge_pages=never<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>[sysctl]<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>kernel.sched_min_granularity_ns = </span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>10000000</span><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>kernel.sched_wakeup_granularity_ns = </span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>15000000</span><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>vm.dirty_ratio = </span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>30</span><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>vm.dirty_background_ratio = </span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>10</span><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>vm.swappiness=</span><span style='font-size:9.0pt;font-family:Menlo;color:#272AD8'>30</span><span style='font-size:9.0pt;font-family:Menlo;color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:#D12F1B'>" > lustre-performance/tuned.conf<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Helvetica'><o:p> </o:p></span></p><p class=MsoNormal style='background:white'><span style='font-size:9.0pt;font-family:Menlo;color:black'>tuned-adm profile lustre-performance<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Pinkesh Valdria<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div></body></html>