<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">No, it is not necessary to tune map_on_demand with modern NICs/MOFED drivers. Latest Lustre can only accept values of ‘0’ or ‘1’. This forces (‘0’) the use of global memory regions (when available), but global MR API was removed (or deprecated?)
by Mellanox. A recent change was made to default map_on_demand to 1 (LU-15186) so FMR/FastReg is used by default even if global MR is available. We expect global MR to be removed completely at some point.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chris Horn<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Thomas Roth <t.roth@gsi.de><br>
<b>Date: </b>Monday, January 31, 2022 at 5:31 AM<br>
<b>To: </b>Horn, Chris <chris.horn@hpe.com>, lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] 'queue depth too large', but connection works<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Very good!<br>
<br>
Digging a bit more into the ko2iblnd parameters, it seems the default for 'map_on_demand' comes out as '1' - both on mlx4 and mlx5 boxes.<br>
I was reading about earlier issues with in rdma, which supposedly pushed the default to 256 - but that was perhaps to long ago.<br>
Is it necessary to tune this parameter nowadays?<br>
<br>
Regards<br>
Thomas<br>
<br>
<br>
On 1/30/22 20:41, Horn, Chris wrote:<br>
> Yes, this means the server has peer_credits=8, so can only accept that value. It informs the client of this so subsequent client connection attempt uses the lower value.<br>
> ________________________________<br>
> From: lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Thomas Roth <t.roth@gsi.de><br>
> Sent: Saturday, January 29, 2022 11:46 AM<br>
> To: lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
> Subject: [lustre-discuss] 'queue depth too large', but connection works<br>
> <br>
> Dear all,<br>
> <br>
> test system: servers 2.12.7, and a client 2.12.6., all mlx4.<br>
> <br>
> The client has some non-default ko2iblnd parameters, including "peer_credits=16".<br>
> <br>
> I mounted my test system there and happily copied around some directories. Only afterwards I found<br>
> > LNetError: 5278:0:(o2iblnd_cb.c:2551:kiblnd_passive_connect()) Can't accept conn from 10.20.3.64@o2ib6, queue depth too large: 16 (<=8 wanted)<br>
> in the MDS log.<br>
> <br>
> I did read LU-3322, but obviously did not the point. "Can't accept conn" used to deny client access, but the MDS that didn't like my client just<br>
> created some ~25k new objects on behalf of that client.<br>
> <br>
> Does this mean client and server negotiate a suitable value, but behind the scenes?<br>
> <br>
> Regards,<br>
> Thomas<br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> lustre-discuss@lists.lustre.org<br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<br>
> <o:p></o:p></p>
</div>
</div>
</body>
</html>