<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
The ko2iblnd-opa settings are only used if you have Intel OPA instead of Mellanox cards (depends on the ko2iblnd-probe script). You should still have ko2iblnd line in the server config that is used for MLX cards in order to set the values to match on both
sides.
<div class=""><br class="">
</div>
<div class="">As for the actual settings, someone with more LNet IB experience should chime in on what is best to use. All I know is that they have to be the same on both sides or they get unhappy, and the usable values depend on the card type and MOFED/OFED
version. As a starting point I would just copy the client ko2iblnd options to the server and see if it works.</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Apr 11, 2024, at 12:02, Daniel Szkola <<a href="mailto:dszkola@fnal.gov" class="">dszkola@fnal.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">On the server node(s):<br class="">
<br class="">
options ko2iblnd-opa peer_credits=32 peer_credits_hiw=16 credits=1024 concurrent_sends=64 ntx=2048 map_on_demand=256 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4<br class="">
<br class="">
On clients:<br class="">
<br class="">
options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4<br class="">
<br class="">
My concern isn’t so much the mismatch because I know that’s an issue but rather what numbers we should settle on with a recent lustre build. I also see the ko2iblnd-opa in the server config, which means because the server is actually loading ko2iblnd that maybe
defaults are used?<br class="">
<br class="">
What made me look was we were seeing lots of:<br class="">
LNetError: 2961324:0:(o2iblnd_cb.c:2612:kiblnd_passive_connect()) Can't accept conn from xxx.xxx.xxx.xxx@o2ib2, queue depth too large: 42 (<=32 wanted)<br class="">
<br class="">
—<br class="">
Dan Szkola<br class="">
FNAL<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">On Apr 11, 2024, at 12:36 PM, Andreas Dilger <<a href="mailto:adilger@whamcloud.com" class="">adilger@whamcloud.com</a>> wrote:<br class="">
<br class="">
[EXTERNAL] – This message is from an external sender<br class="">
<br class="">
<br class="">
On Apr 11, 2024, at 09:56, Daniel Szkola via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:<br class="">
<blockquote type="cite" class=""><br class="">
Hello all,<br class="">
<br class="">
I recently discovered some mismatches in our /etc/modprobe.d/ko2iblnd.conf files between our clients and servers.<br class="">
<br class="">
Is it now recommended to keep the defaults on this module and run without a config file or are there recommended numbers for lustre-2.15.X?<br class="">
<br class="">
The only thing I’ve seen that provides any guidance is the Lustre wiki and an HP/Cray doc:<br class="">
<br class="">
<a href="https://www.hpe.com/psnow/resources/ebooks/a00113867en_us_v2/Lustre_Server_Recommended_Tuning_Parameters_4.x.html" class="">https://www.hpe.com/psnow/resources/ebooks/a00113867en_us_v2/Lustre_Server_Recommended_Tuning_Parameters_4.x.html</a><br class="">
<br class="">
Anyone have any sage advice on what the ko2iblnd.conf (and possibly ko2iblnd-opa.conf and hfi1.conf as well) on modern systems?<br class="">
</blockquote>
<br class="">
It would be useful to know what specific settings are mismatched. Definitely some of them need to be consistent between peers, others depend on your system.<br class="">
<br class="">
Cheers, Andreas<br class="">
--<br class="">
Andreas Dilger<br class="">
Lustre Principal Architect<br class="">
Whamcloud<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>