<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<style type="text/css">.style1 {font-family: "Times New Roman";}</style></head><body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:windowtext">Do you have Ethernet flow control configured on all ports (especially the uplink ports)? We’ve found that flow control is critical when there are mismatched uplink/client port speeds.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Shawn<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> lustre-discuss <lustre-discuss-bounces@lists.lustre.org>
<b>On Behalf Of </b>Louis Bailleul<br>
<b>Sent:</b> Monday, August 12, 2019 1:08 PM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> [lustre-discuss] Very bad lnet ethernet read performance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Helvetica",sans-serif">Hi all,<br>
<br>
I am trying to understand what I am doing wrong here.<br>
I have a Lustre 2.12.1 system backed by NVME drives under zfs for which obdfilter-survey gives descent values</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-family:"Helvetica",sans-serif">ost 2 sz 536870912K rsz 1024K obj 2 thr 256 write 15267.49 [6580.36, 8664.20] rewrite 15225.24 [6559.05, 8900.54] read 19739.86 [9062.25, 10429.04]
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-family:"Helvetica",sans-serif">But my actual Lustre performances are pretty poor in comparison (can't top 8GB/s write and 13.5GB/s read)<br>
So I started to question my lnet tuning but playing with peer_credits and max_rpc_per_pages didn't help.<br>
</span><br>
<span style="font-family:"Helvetica",sans-serif">My test setup consist of 133x10G Ethernet clients (uplinks between end devices and OSS are 2x100G for every 20 nodes).<br>
The single OSS is fitted with a bonding of 2x100G Ethernet.<br>
<br>
I have tried to understand the problem using lnet_selftest but I'll need some help/doco as this doesn't make sense to me.<br>
<br>
Testing a single 10G client</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-family:"Helvetica",sans-serif">[LNet Rates of lfrom]</span><br>
<span style="font-family:"Helvetica",sans-serif">[R] Avg: 2231 RPC/s Min: 2231 RPC/s Max: 2231 RPC/s</span><br>
<span style="font-family:"Helvetica",sans-serif">[W] Avg: 1156 RPC/s Min: 1156 RPC/s Max: 1156 RPC/s</span><br>
<span style="font-family:"Helvetica",sans-serif">[LNet Bandwidth of lfrom]</span><br>
<span style="font-family:"Helvetica",sans-serif">[R] Avg: 1075.16 MiB/s Min: 1075.16 MiB/s Max: 1075.16 MiB/s
</span><br>
<span style="font-family:"Helvetica",sans-serif">[W] Avg: 0.18 MiB/s Min: 0.18 MiB/s Max: 0.18 MiB/s
</span><br>
<span style="font-family:"Helvetica",sans-serif">[LNet Rates of lto]</span><br>
<span style="font-family:"Helvetica",sans-serif">[R] Avg: 1179 RPC/s Min: 1179 RPC/s Max: 1179 RPC/s</span><br>
<span style="font-family:"Helvetica",sans-serif">[W] Avg: 2254 RPC/s Min: 2254 RPC/s Max: 2254 RPC/s</span><br>
<span style="font-family:"Helvetica",sans-serif">[LNet Bandwidth of lto]</span><br>
<span style="font-family:"Helvetica",sans-serif">[R] Avg: 0.19 MiB/s Min: 0.19 MiB/s Max: 0.19 MiB/s
</span><br>
<span style="font-family:"Helvetica",sans-serif">[W] Avg: 1075.17 MiB/s Min: 1075.17 MiB/s Max: 1075.17 MiB/s
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-family:"Helvetica",sans-serif">With 10x10G clients :</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-family:"Helvetica",sans-serif">[LNet Rates of lfrom]<br>
[R] Avg: 1416 RPC/s Min: 1102 RPC/s Max: 1642 RPC/s<br>
[W] Avg: 708 RPC/s Min: 551 RPC/s Max: 821 RPC/s<br>
[LNet Bandwidth of lfrom]<br>
[R] Avg: 708.20 MiB/s Min: 550.77 MiB/s Max: 820.96 MiB/s <br>
[W] Avg: 0.11 MiB/s Min: 0.08 MiB/s Max: 0.13 MiB/s <br>
[LNet Rates of lto]<br>
[R] Avg: 7084 RPC/s Min: 7084 RPC/s Max: 7084 RPC/s<br>
[W] Avg: 14165 RPC/s Min: 14165 RPC/s Max: 14165 RPC/s<br>
[LNet Bandwidth of lto]<br>
[R] Avg: 1.08 MiB/s Min: 1.08 MiB/s Max: 1.08 MiB/s <br>
[W] Avg: 7081.86 MiB/s Min: 7081.86 MiB/s Max: 7081.86 MiB/s </span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-family:"Helvetica",sans-serif"><br>
With all 133x10G clients:</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-family:"Helvetica",sans-serif">[LNet Rates of lfrom]<br>
[R] Avg: 510 RPC/s Min: 98 RPC/s Max: 23457 RPC/s<br>
[W] Avg: 510 RPC/s Min: 49 RPC/s Max: 45863 RPC/s<br>
[LNet Bandwidth of lfrom]<br>
[R] Avg: 169.87 MiB/s Min: 48.77 MiB/s Max: 341.26 MiB/s <br>
[W] Avg: 169.86 MiB/s Min: 0.01 MiB/s Max: 22757.92 MiB/s <br>
[LNet Rates of lto]<br>
[R] Avg: 23458 RPC/s Min: 23458 RPC/s Max: 23458 RPC/s<br>
[W] Avg: 45876 RPC/s Min: 45876 RPC/s Max: 45876 RPC/s<br>
[LNet Bandwidth of lto]<br>
[R] Avg: 341.12 MiB/s Min: 341.12 MiB/s Max: 341.12 MiB/s <br>
[W] Avg: 22758.42 MiB/s Min: 22758.42 MiB/s Max: 22758.42 MiB/s </span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<span style="font-family:"Helvetica",sans-serif">So if I add clients the aggregate write bandwidth somewhat stacks, but the read bandwidth decrease ???<br>
When throwing all the nodes at the system, I am pretty happy with the ~22GB/s on write pretty as this is in the 90% of the 2x100G, but the 341MB/s read sounds very weird considering that this is a third of the performance of a single client.<br>
<br>
This are my ksocklnd tuning :</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Helvetica",sans-serif"># for i in /sys/module/ksocklnd/parameters/*; do echo "$i : $(cat $i)"; done<br>
/sys/module/ksocklnd/parameters/credits : 1024<br>
/sys/module/ksocklnd/parameters/eager_ack : 0<br>
/sys/module/ksocklnd/parameters/enable_csum : 0<br>
/sys/module/ksocklnd/parameters/enable_irq_affinity : 0<br>
/sys/module/ksocklnd/parameters/inject_csum_error : 0<br>
/sys/module/ksocklnd/parameters/keepalive : 30<br>
/sys/module/ksocklnd/parameters/keepalive_count : 5<br>
/sys/module/ksocklnd/parameters/keepalive_idle : 30<br>
/sys/module/ksocklnd/parameters/keepalive_intvl : 5<br>
/sys/module/ksocklnd/parameters/max_reconnectms : 60000<br>
/sys/module/ksocklnd/parameters/min_bulk : 1024<br>
/sys/module/ksocklnd/parameters/min_reconnectms : 1000<br>
/sys/module/ksocklnd/parameters/nagle : 0<br>
/sys/module/ksocklnd/parameters/nconnds : 4<br>
/sys/module/ksocklnd/parameters/nconnds_max : 64<br>
/sys/module/ksocklnd/parameters/nonblk_zcack : 1<br>
/sys/module/ksocklnd/parameters/nscheds : 12<br>
/sys/module/ksocklnd/parameters/peer_buffer_credits : 0<br>
/sys/module/ksocklnd/parameters/peer_credits : 128<br>
/sys/module/ksocklnd/parameters/peer_timeout : 180<br>
/sys/module/ksocklnd/parameters/round_robin : 1<br>
/sys/module/ksocklnd/parameters/rx_buffer_size : 0<br>
/sys/module/ksocklnd/parameters/sock_timeout : 50<br>
/sys/module/ksocklnd/parameters/tx_buffer_size : 0<br>
/sys/module/ksocklnd/parameters/typed_conns : 1<br>
/sys/module/ksocklnd/parameters/zc_min_payload : 16384<br>
/sys/module/ksocklnd/parameters/zc_recv : 0<br>
/sys/module/ksocklnd/parameters/zc_recv_min_nfrags : 16</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-family:"Helvetica",sans-serif">Best regards,<br>
Louis</span><o:p></o:p></p>
</div>
<br /><br /><p style="font-family: Verdana; font-size:10pt; color:#666666;"><b>Disclaimer</b></p><p style="font-family: Verdana; font-size:8pt; color:#666666;">Please see our <A href="https://www.nag.co.uk/content/privacy-notice">Privacy Notice</A> for information on how we process personal data.<br /><br />This e-mail has been scanned for all viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business.</p>
</body></html>