<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 8/22/17 9:22 AM, Mannthey, Keith
wrote:<br>
</div>
<blockquote
cite="mid:E8BCA7842FE64F499E796131B007C2A1998EFD78@FMSMSX114.amr.corp.intel.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:2 7 4 9 2 2 5 2 4 4;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Courier;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:217202463;
mso-list-template-ids:-2024763258;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1
{mso-list-id:792989998;
mso-list-template-ids:2116710602;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2
{mso-list-id:2093315743;
mso-list-template-ids:686186146;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3
{mso-list-id:2123835472;
mso-list-template-ids:-2099618820;}
@list l3:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l3:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l3:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l3:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l3:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l3:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l3:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l3:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">You
may want to file a jira ticket if ko2iblnd-opa setting were
being automatically used on your Mellanox setup. That is
not expected.
</span></p>
</div>
</blockquote>
yes they are automatically used on my Mellanox and the script
ko2iblnd-probe seems like not working properly.<br>
<blockquote
cite="mid:E8BCA7842FE64F499E796131B007C2A1998EFD78@FMSMSX114.amr.corp.intel.com"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> <o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">On
another note: As you note you NVMe backend is much faster
than QRD link speed. You may want to look at using the new
Multi-rall lnet feature to boost network bandwidth. You can
add a 2<sup>nd</sup> QRD HCA/Port and get more Lnet bandwith
from your OSS server. It is a new feature that is a bit of
work to use but if you are chasing bandwith it might be
worth the effort.</span></p>
</div>
</blockquote>
I have a dual infiniband card so I was thinking to bond them to have
more bandwidth. Is this that you mean when you are talking about the
Muti-rail feature boost ?<br>
<br>
thanks<br>
<br>
Rick<br>
<br>
<br>
<blockquote
cite="mid:E8BCA7842FE64F499E796131B007C2A1998EFD78@FMSMSX114.amr.corp.intel.com"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Keith
<o:p></o:p></span></p>
<p class="MsoNormal"><a moz-do-not-send="true"
name="_MailEndCompose"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a moz-do-not-send="true"
name="_____replyseparator"></a><b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">
lustre-discuss
[<a class="moz-txt-link-freetext" href="mailto:lustre-discuss-bounces@lists.lustre.org">mailto:lustre-discuss-bounces@lists.lustre.org</a>]
<b>On Behalf Of </b>Chris Horn<br>
<b>Sent:</b> Monday, August 21, 2017 12:40 PM<br>
<b>To:</b> Riccardo Veraldi
<a class="moz-txt-link-rfc2396E" href="mailto:Riccardo.Veraldi@cnaf.infn.it"><Riccardo.Veraldi@cnaf.infn.it></a>; Arman Khalatyan
<a class="moz-txt-link-rfc2396E" href="mailto:arm2arm@gmail.com"><arm2arm@gmail.com></a><br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> Re: [lustre-discuss] Lustre poor
performance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">The
ko2iblnd-opa settings are tuned specifically for Intel
OmniPath. Take a look at the /usr/sbin/ko2iblnd-probe script
to see how OPA hardware is detected and the “ko2iblnd-opa”
settings get used.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">Chris
Horn<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-family:"Calibri",sans-serif;color:black">From:
</span></b><span
style="font-family:"Calibri",sans-serif;color:black">lustre-discuss
<<a moz-do-not-send="true"
href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a>>
on behalf of Riccardo Veraldi <<a
moz-do-not-send="true"
href="mailto:Riccardo.Veraldi@cnaf.infn.it">Riccardo.Veraldi@cnaf.infn.it</a>><br>
<b>Date: </b>Saturday, August 19, 2017 at 5:00 PM<br>
<b>To: </b>Arman Khalatyan <<a moz-do-not-send="true"
href="mailto:arm2arm@gmail.com">arm2arm@gmail.com</a>><br>
<b>Cc: </b>"<a moz-do-not-send="true"
href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>"
<<a moz-do-not-send="true"
href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>><br>
<b>Subject: </b>Re: [lustre-discuss] Lustre poor
performance<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I ran again my Lnet self test and this
time adding --concurrency=16 I can use all of the IB
bandwith (3.5GB/sec).<br>
<br>
the only thing I do not understand is why ko2iblnd.conf is
not loaded properly and I had to remove the alias in the
config file to allow<br>
the proper peer_credit settings to be loaded.<br>
<br>
thanks to everyone for helping<br>
<br>
Riccardo<br>
<br>
On 8/19/17 8:54 AM, Riccardo Veraldi wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><br>
I found out that ko2iblnd is not getting settings from
/etc/modprobe/ko2iblnd.conf<br>
<tt><span style="font-size:10.0pt">alias ko2iblnd-opa
ko2iblnd</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New""><br>
<tt>options ko2iblnd-opa peer_credits=128
peer_credits_hiw=64 credits=1024 concurrent_sends=256
ntx=2048 map_on_demand=32 fmr_pool_size=2048
fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4</tt><br>
<br>
<tt>install ko2iblnd /usr/sbin/ko2iblnd-probe</tt><br>
</span><br>
but if I modify ko2iblnd.conf like this, then settings are
loaded:<br>
<br>
<tt><span style="font-size:10.0pt">options ko2iblnd
peer_credits=128 peer_credits_hiw=64 credits=1024
concurrent_sends=256 ntx=2048 map_on_demand=32
fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1
conns_per_peer=4</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New""><br>
<br>
<tt>install ko2iblnd /usr/sbin/ko2iblnd-probe</tt></span><br>
<br>
Lnet tests show better behaviour but still I Would expect
more than this.<br>
Is it possible to tune parameters in
/etc/modprobe/ko2iblnd.conf so that Mellanox ConnectX-3
will work more efficiently ?<br>
<br>
[LNet Rates of servers]<br>
[R] Avg: 2286 RPC/s Min: 0 RPC/s Max: 4572
RPC/s<br>
[W] Avg: 3322 RPC/s Min: 0 RPC/s Max: 6643
RPC/s<br>
[LNet Bandwidth of servers]<br>
[R] Avg: 625.23 MiB/s Min: 0.00 MiB/s Max: 1250.46
MiB/s <br>
[W] Avg: 1035.85 MiB/s Min: 0.00 MiB/s Max: 2071.69
MiB/s <br>
[LNet Rates of servers]<br>
[R] Avg: 2286 RPC/s Min: 1 RPC/s Max: 4571
RPC/s<br>
[W] Avg: 3321 RPC/s Min: 1 RPC/s Max: 6641
RPC/s<br>
[LNet Bandwidth of servers]<br>
[R] Avg: 625.55 MiB/s Min: 0.00 MiB/s Max: 1251.11
MiB/s <br>
[W] Avg: 1035.05 MiB/s Min: 0.00 MiB/s Max: 2070.11
MiB/s <br>
[LNet Rates of servers]<br>
[R] Avg: 2291 RPC/s Min: 0 RPC/s Max: 4581
RPC/s<br>
[W] Avg: 3329 RPC/s Min: 0 RPC/s Max: 6657
RPC/s<br>
[LNet Bandwidth of servers]<br>
[R] Avg: 626.55 MiB/s Min: 0.00 MiB/s Max: 1253.11
MiB/s <br>
[W] Avg: 1038.05 MiB/s Min: 0.00 MiB/s Max: 2076.11
MiB/s <br>
session is ended<br>
./lnet_test.sh: line 17: 23394 Terminated lst
stat servers<br>
<br>
<br>
<br>
<br>
On 8/19/17 4:20 AM, Arman Khalatyan wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">just minor comment, <o:p></o:p></p>
<div>
<p class="MsoNormal">you should push up performance of
your nodes,they are not running in the max cpu
frequencies.Al tests might be inconsistent. in order
to get most of ib run following:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">tuned-adm profile
latency-performance<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">for more options use:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">tuned-adm list<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">It will be interesting to see the
difference.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Am 19.08.2017 3:57 vorm. schrieb
"Riccardo Veraldi" <<a moz-do-not-send="true"
href="mailto:Riccardo.Veraldi@cnaf.infn.it">Riccardo.Veraldi@cnaf.infn.it</a>>:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello
Keith and Dennis, these are the test I ran.<o:p></o:p></p>
<ul type="disc">
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l3
level1 lfo3">
obdfilter-survey, shows that I Can saturate
disk performance, the NVMe/ZFS backend is
performing very well and it is faster then my
Infiniband network
<o:p></o:p></li>
</ul>
<p><tt><b><span style="font-size:10.0pt">pool
alloc free read write read write</span></b></tt><b><span
style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>------------ ----- ----- -----
----- ----- -----</tt><br>
<tt>drpffb-ost01 3.31T 3.19T 3
35.7K 16.0K 7.03G</tt><br>
<tt> raidz1 3.31T 3.19T 3
35.7K 16.0K 7.03G</tt><br>
<tt> nvme0n1 - - 1
5.95K 7.99K 1.17G</tt><br>
<tt> nvme1n1 - - 0
6.01K 0 1.18G</tt><br>
<tt> nvme2n1 - - 0
5.93K 0 1.17G</tt><br>
<tt> nvme3n1 - - 0
5.88K 0 1.16G</tt><br>
<tt> nvme4n1 - - 1
5.95K 7.99K 1.17G</tt><br>
<tt> nvme5n1 - - 0
5.96K 0 1.17G</tt><br>
<tt>------------ ----- ----- -----
----- ----- -----</tt></span></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">this
are the tests results<br>
<br>
<tt><span style="font-size:10.0pt">Fri Aug 18
16:54:48 PDT 2017 Obdfilter-survey for
case=disk from drp-tst-ffb01</span></tt><span
style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>ost 1 sz 10485760K rsz 1024K obj 1
thr 1 write<b> 7633.08 </b>
SHORT rewrite 7558.78 SHORT read
3205.24 [3213.70, 3226.78]
</tt><br>
<tt>ost 1 sz 10485760K rsz 1024K obj 1
thr 2 write<b> 7996.89 </b>
SHORT rewrite 7903.42 SHORT read
5264.70 SHORT
</tt><br>
<tt>ost 1 sz 10485760K rsz 1024K obj 2
thr 2 write <b>7718.94</b>
SHORT rewrite 7977.84 SHORT read
5802.17 SHORT
</tt></span><o:p></o:p></p>
<ul type="disc">
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1
level1 lfo6">
Lnet self test, and here I see the problems.
For reference 172.21.52.[83,84] are the two
OSSes 172.21.52.86 is the reader/writer. Here
is the script that I ran
<o:p></o:p></li>
</ul>
<p><tt><span style="font-size:10.0pt">#!/bin/bash</span></tt><span
style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>export LST_SESSION=$$</tt><br>
<tt>lst new_session read_write</tt><br>
<tt>lst add_group servers
172.21.52.[83,84]@o2ib5</tt><br>
<tt>lst add_group readers <a
moz-do-not-send="true"
href="mailto:172.21.52.86@o2ib5">172.21.52.86@o2ib5</a></tt><br>
<tt>lst add_group writers <a
moz-do-not-send="true"
href="mailto:172.21.52.86@o2ib5">172.21.52.86@o2ib5</a></tt><br>
<tt>lst add_batch bulk_rw</tt><br>
<tt>lst add_test --batch bulk_rw --from
readers --to servers \</tt><br>
<tt>brw read check=simple size=1M</tt><br>
<tt>lst add_test --batch bulk_rw --from
writers --to servers \</tt><br>
<tt>brw write check=full size=1M</tt><br>
<tt># start running</tt><br>
<tt>lst run bulk_rw</tt><br>
<tt># display server stats for 30 seconds</tt><br>
<tt>lst stat servers & sleep 30; kill $!</tt><br>
<tt># tear down</tt><br>
<tt>lst end_session</tt></span><o:p></o:p></p>
<p><o:p> </o:p></p>
<p>here the results<o:p></o:p></p>
<p style="margin-bottom:12.0pt"><tt><span
style="font-size:10.0pt">SESSION: read_write
FEATURES: 1 TIMEOUT: 300 FORCE: No</span></tt><span
style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>172.21.52.[83,84]@o2ib5 are added to
session</tt><br>
<tt><a moz-do-not-send="true"
href="mailto:172.21.52.86@o2ib5">172.21.52.86@o2ib5</a>
are added to session</tt><br>
<tt><a moz-do-not-send="true"
href="mailto:172.21.52.86@o2ib5">172.21.52.86@o2ib5</a>
are added to session</tt><br>
<tt>Test was added successfully</tt><br>
<tt>Test was added successfully</tt><br>
<tt>bulk_rw is running now</tt><br>
<tt>[LNet Rates of servers]</tt><br>
<tt>[R] Avg: 1751 RPC/s Min: 0
RPC/s Max: 3502 RPC/s</tt><br>
<tt>[W] Avg: 2525 RPC/s Min: 0
RPC/s Max: 5050 RPC/s</tt><br>
<tt>[LNet Bandwidth of servers]</tt><br>
<tt>[R] Avg: 488.79 MiB/s Min: 0.00
MiB/s Max: 977.59 MiB/s </tt><br>
<tt>[W] Avg: 773.99 MiB/s Min: 0.00
MiB/s Max: 1547.99 MiB/s </tt><br>
<tt>[LNet Rates of servers]</tt><br>
<tt>[R] Avg: 1718 RPC/s Min: 0
RPC/s Max: 3435 RPC/s</tt><br>
<tt>[W] Avg: 2479 RPC/s Min: 0
RPC/s Max: 4958 RPC/s</tt><br>
<tt>[LNet Bandwidth of servers]</tt><br>
<tt>[R] Avg: 478.19 MiB/s Min: 0.00
MiB/s Max: 956.39 MiB/s </tt><br>
<tt>[W] Avg: 761.74 MiB/s Min: 0.00
MiB/s Max: 1523.47 MiB/s </tt><br>
<tt>[LNet Rates of servers]</tt><br>
<tt>[R] Avg: 1734 RPC/s Min: 0
RPC/s Max: 3467 RPC/s</tt><br>
<tt>[W] Avg: 2506 RPC/s Min: 0
RPC/s Max: 5012 RPC/s</tt><br>
<tt>[LNet Bandwidth of servers]</tt><br>
<tt>[R] Avg: 480.79 MiB/s Min: 0.00
MiB/s Max: 961.58 MiB/s </tt><br>
<tt>[W] Avg: 772.49 MiB/s Min: 0.00
MiB/s Max: 1544.98 MiB/s </tt><br>
<tt>[LNet Rates of servers]</tt><br>
<tt>[R] Avg: 1722 RPC/s Min: 0
RPC/s Max: 3444 RPC/s</tt><br>
<tt>[W] Avg: 2486 RPC/s Min: 0
RPC/s Max: 4972 RPC/s</tt><br>
<tt>[LNet Bandwidth of servers]</tt><br>
<tt>[R] Avg: 479.09 MiB/s Min: 0.00
MiB/s Max: 958.18 MiB/s </tt><br>
<tt>[W] Avg: 764.19 MiB/s Min: 0.00
MiB/s Max: 1528.38 MiB/s </tt><br>
<tt>[LNet Rates of servers]</tt><br>
<tt>[R] Avg: 1741 RPC/s Min: 0
RPC/s Max: 3482 RPC/s</tt><br>
<tt>[W] Avg: 2513 RPC/s Min: 0
RPC/s Max: 5025 RPC/s</tt><br>
<tt>[LNet Bandwidth of servers]</tt><br>
<tt>[R] Avg: 484.59 MiB/s Min: 0.00
MiB/s Max: 969.19 MiB/s </tt><br>
<tt>[W] Avg: 771.94 MiB/s Min: 0.00
MiB/s Max: 1543.87 MiB/s </tt><br>
<tt>session is ended</tt><br>
<tt>./lnet_test.sh: line 17: 4940
Terminated lst stat servers</tt></span><o:p></o:p></p>
<p class="MsoNormal">so looks like Lnet is really
under performing going at least half and less
than InfiniBand capabilities.<br>
How can I find out what is causing this ? <o:p></o:p></p>
<p>running perf tools tests with infiniband tools
I have good results:<o:p></o:p></p>
<p><o:p> </o:p></p>
<p style="margin-bottom:12.0pt"><tt><span
style="font-size:10.0pt">************************************</span></tt><span
style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>* Waiting for client to connect... *</tt><br>
<tt>************************************</tt><br>
<br>
<tt>---------------------------------------------------------------------------------------</tt><br>
<tt> Send BW Test</tt><br>
<tt> Dual-port : OFF
Device : mlx4_0</tt><br>
<tt> Number of qps : 1 Transport type
: IB</tt><br>
<tt> Connection type : RC Using
SRQ : OFF</tt><br>
<tt> RX depth : 512</tt><br>
<tt> CQ Moderation : 100</tt><br>
<tt> Mtu : 2048[B]</tt><br>
<tt> Link type : IB</tt><br>
<tt> Max inline data : 0[B]</tt><br>
<tt> rdma_cm QPs : OFF</tt><br>
<tt> Data ex. method : Ethernet</tt><br>
<tt>---------------------------------------------------------------------------------------</tt><br>
<tt> local address: LID 0x07 QPN 0x020f PSN
0xacc37a</tt><br>
<tt> remote address: LID 0x0a QPN 0x020f PSN
0x91a069</tt><br>
<tt>---------------------------------------------------------------------------------------</tt><br>
<tt> #bytes #iterations BW
peak[MB/sec] BW average[MB/sec]
MsgRate[Mpps]</tt><br>
<tt>Conflicting CPU frequency values detected:
1249.234000 != 1326.000000. CPU Frequency is
not max.</tt><br>
<tt> 2 1000
0.00 11.99
6.285330</tt><br>
<tt>Conflicting CPU frequency values detected:
1314.910000 != 1395.460000. CPU Frequency is
not max.</tt><br>
<tt> 4 1000
0.00 28.26
7.409324</tt><br>
<tt>Conflicting CPU frequency values detected:
1314.910000 != 1460.207000. CPU Frequency is
not max.</tt><br>
<tt> 8 1000
0.00 54.47
7.139164</tt><br>
<tt>Conflicting CPU frequency values detected:
1314.910000 != 1244.320000. CPU Frequency is
not max.</tt><br>
<tt> 16 1000
0.00 113.13
7.413889</tt><br>
<tt>Conflicting CPU frequency values detected:
1314.910000 != 1460.207000. CPU Frequency is
not max.</tt><br>
<tt> 32 1000
0.00 226.07
7.407811</tt><br>
<tt>Conflicting CPU frequency values detected:
1469.703000 != 1301.031000. CPU Frequency is
not max.</tt><br>
<tt> 64 1000
0.00 452.12
7.407465</tt><br>
<tt>Conflicting CPU frequency values detected:
1469.703000 != 1301.031000. CPU Frequency is
not max.</tt><br>
<tt> 128 1000
0.00 845.45
6.925918</tt><br>
<tt>Conflicting CPU frequency values detected:
1469.703000 != 1362.257000. CPU Frequency is
not max.</tt><br>
<tt> 256 1000
0.00 1746.93
7.155406</tt><br>
<tt>Conflicting CPU frequency values detected:
1469.703000 != 1362.257000. CPU Frequency is
not max.</tt><br>
<tt> 512 1000
0.00 2766.93
5.666682</tt><br>
<tt>Conflicting CPU frequency values detected:
1296.714000 != 1204.675000. CPU Frequency is
not max.</tt><br>
<tt> 1024 1000
0.00 3516.26
3.600646</tt><br>
<tt>Conflicting CPU frequency values detected:
1296.714000 != 1325.535000. CPU Frequency is
not max.</tt><br>
<tt> 2048 1000
0.00 3630.93
1.859035</tt><br>
<tt>Conflicting CPU frequency values detected:
1296.714000 != 1331.312000. CPU Frequency is
not max.</tt><br>
<tt> 4096 1000
0.00 3702.39
0.947813</tt><br>
<tt>Conflicting CPU frequency values detected:
1296.714000 != 1200.027000. CPU Frequency is
not max.</tt><br>
<tt> 8192 1000
0.00 3724.82
0.476777</tt><br>
<tt>Conflicting CPU frequency values detected:
1384.902000 != 1314.113000. CPU Frequency is
not max.</tt><br>
<tt> 16384 1000
0.00 3731.21
0.238798</tt><br>
<tt>Conflicting CPU frequency values detected:
1578.078000 != 1200.027000. CPU Frequency is
not max.</tt><br>
<tt> 32768 1000
0.00 3735.32
0.119530</tt><br>
<tt>Conflicting CPU frequency values detected:
1578.078000 != 1200.027000. CPU Frequency is
not max.</tt><br>
<tt> 65536 1000
0.00 3736.98
0.059792</tt><br>
<tt>Conflicting CPU frequency values detected:
1578.078000 != 1200.027000. CPU Frequency is
not max.</tt><br>
<tt> 131072 1000
0.00 3737.80
0.029902</tt><br>
<tt>Conflicting CPU frequency values detected:
1578.078000 != 1200.027000. CPU Frequency is
not max.</tt><br>
<tt> 262144 1000
0.00 3738.43
0.014954</tt><br>
<tt>Conflicting CPU frequency values detected:
1570.507000 != 1200.027000. CPU Frequency is
not max.</tt><br>
<tt> 524288 1000
0.00 3738.50
0.007477</tt><br>
<tt>Conflicting CPU frequency values detected:
1457.019000 != 1236.152000. CPU Frequency is
not max.</tt><br>
<tt> 1048576 1000
0.00 3738.65
0.003739</tt><br>
<tt>Conflicting CPU frequency values detected:
1411.597000 != 1234.957000. CPU Frequency is
not max.</tt><br>
<tt> 2097152 1000
0.00 3738.65
0.001869</tt><br>
<tt>Conflicting CPU frequency values detected:
1369.828000 != 1516.851000. CPU Frequency is
not max.</tt><br>
<tt> 4194304 1000
0.00 3738.80
0.000935</tt><br>
<tt>Conflicting CPU frequency values detected:
1564.664000 != 1247.574000. CPU Frequency is
not max.</tt><br>
<tt> 8388608 1000
0.00 3738.76
0.000467</tt><br>
<tt>---------------------------------------------------------------------------------------</tt></span><o:p></o:p></p>
<p><tt><span style="font-size:10.0pt">RDMA modules
are loaded</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New""><br>
<br>
<tt>rpcrdma 90366 0 </tt><br>
<tt>rdma_ucm 26837 0 </tt><br>
<tt>ib_uverbs 51854 2
ib_ucm,rdma_ucm</tt><br>
<tt>rdma_cm 53755 5
rpcrdma,ko2iblnd,ib_iser,rdma_ucm,ib_isert</tt><br>
<tt>ib_cm 47149 5
rdma_cm,ib_srp,ib_ucm,ib_srpt,ib_ipoib</tt><br>
<tt>iw_cm 46022 1 rdma_cm</tt><br>
<tt>ib_core 210381 15
rdma_cm,ib_cm,iw_cm,rpcrdma,ko2iblnd,mlx4_ib,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert</tt><br>
<tt>sunrpc 334343 17
nfs,nfsd,rpcsec_gss_krb5,auth_rpcgss,lockd,nfsv4,rpcrdma,nfs_acl</tt></span><o:p></o:p></p>
<p>I do not know where to look to have Lnet
performing faster. I am running my ib0 interface
in connected mode with 65520 MTU size.<o:p></o:p></p>
<p>Any hint will be much appreciated<o:p></o:p></p>
<p>thank you<o:p></o:p></p>
<p>Rick<o:p></o:p></p>
<div>
<p><o:p> </o:p></p>
<p><o:p> </o:p></p>
<p><o:p> </o:p></p>
<p class="MsoNormal">On 8/18/17 9:05 AM,
Mannthey, Keith wrote:<o:p></o:p></p>
</div>
</div>
<div>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>I would suggest you a few other tests to help isolate where the issue might be. <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>1. What is the single thread "DD" write speed?<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>2. Lnet_selfttest: Please see " Chapter 28. Testing Lustre Network Performance (LNet Self-Test)" in the Lustre manual if this is a new test for you. <o:p></o:p></pre>
<pre>This will help show how much Lnet bandwith you have from your single client. There are tunable in the lnet later that can affect things. Which QRD HCA are you using?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>3. OBDFilter_survey : Please see " 29.3. Testing OST Performance (obdfilter-survey)" in the Lustre manual. This test will help demonstrate what the backed NVMe/ZFS setup can do at the OBD layer in Lustre. <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks,<o:p></o:p></pre>
<pre> Keith <o:p></o:p></pre>
<pre>-----Original Message-----<o:p></o:p></pre>
<pre>From: lustre-discuss [<a moz-do-not-send="true" href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">mailto:lustre-discuss-bounces@lists.lustre.org</a>] On Behalf Of Riccardo Veraldi<o:p></o:p></pre>
<pre>Sent: Thursday, August 17, 2017 10:48 PM<o:p></o:p></pre>
<pre>To: Dennis Nelson <a moz-do-not-send="true" href="mailto:dnelson@ddn.com" target="_blank"><dnelson@ddn.com></a>; <a moz-do-not-send="true" href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><o:p></o:p></pre>
<pre>Subject: Re: [lustre-discuss] Lustre poor performance<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>this is my lustre.conf<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>[drp-tst-ffb01:~]$ cat /etc/modprobe.d/lustre.conf options lnet networks=o2ib5(ib0),tcp5(enp1s0f0)<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>data transfer is over infiniband<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 65520<o:p></o:p></pre>
<pre> inet 172.21.52.83 netmask 255.255.252.0 broadcast 172.21.55.255<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>On 8/17/17 10:45 PM, Riccardo Veraldi wrote:<o:p></o:p></pre>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>On 8/17/17 9:22 PM, Dennis Nelson wrote:<o:p></o:p></pre>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>It appears that you are running iozone on a single client? What kind of network is tcp5? Have you looked at the network to make sure it is not the bottleneck?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
<pre>yes the data transfer is on ib0 interface and I did a memory to memory <o:p></o:p></pre>
<pre>test through InfiniBand QDR resulting in 3.7GB/sec.<o:p></o:p></pre>
<pre>tcp is used to connect to the MDS. It is tcp5 to differentiate it from <o:p></o:p></pre>
<pre>my other many Lustre clusters. I could have called it tcp but it does <o:p></o:p></pre>
<pre>not make any difference performance wise.<o:p></o:p></pre>
<pre>I ran the test from one single node yes, I ran the same test also <o:p></o:p></pre>
<pre>locally on a zpool identical to the one on the Lustre OSS.<o:p></o:p></pre>
<pre> Ihave 4 identical servers each of them with the aame nvme disks:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>server1: OSS - OST1 Lustre/ZFS raidz1<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>server2: OSS - OST2 Lustre/ZFS raidz1<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>server3: local ZFS raidz1<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>server4: Lustre client<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>lustre-discuss mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></pre>
</blockquote>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>lustre-discuss mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
</div>
</div>
<p class="MsoNormal">_______________________________________________
lustre-discuss mailing list
<a moz-do-not-send="true"
href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a moz-do-not-send="true"
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
target="_blank">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a> <o:p></o:p></p>
</blockquote>
</div>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>