<div dir="ltr">Jesse,<div><br></div><div>Thank you very much for your answer. I tried your recommendation of putting map_on_demand=0, but it does not work. Anyway, as it is only a means to test lustre installation I can manage to use ksocklnd over ethernet, which it does work.</div><div><br></div><div>Best regards</div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Fri, 7 Feb 2025 at 19:24, Jesse Stroik <<a href="mailto:jesse.stroik@ssec.wisc.edu">jesse.stroik@ssec.wisc.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Ramiro,<br>
<br>
The invalid MR size looks like you're running into a limit with your cards setting up the RDMA (o2ib) LND when bringing up the network. There may be adjustments or workarounds for it possibly including setting map_on_demand=0 as an argument to the lnet module there.<br>
<br>
And since you are using older IB hardware on a newer OS, just a heads up: we recently ran into an issue with connectx-3 IB cards after upgrading our operating systems where we found RMDA communication to be unreliable possibly because they often would exceed the amount of connection queue pairs they could create. For us, the workaround was to use the ksocklnd instead of o2iblnd. If you have trouble getting the o2ib lustre network driver to work with this older hardware due to RDMA problems, that could be a workaround although it may not be feasible to implement depending on your networking setup.<br>
<br>
Best,<br>
Jesse<br>
<br>
<br>
________________________________________<br>
From: lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Ramiro Alba Queipo <<a href="mailto:ramiro.alba@upc.edu" target="_blank">ramiro.alba@upc.edu</a>><br>
Sent: Thursday, February 6, 2025 3:34 AM<br>
To: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
Subject: [lustre-discuss] Lnet not going up with InfiniHost III Lx HCA card<br>
<br>
<br>
Hi all,<br>
<br>
I am testing Ubuntu 24.04 (6.8.0-52-generic) client with Lustre 2.16.1 over Infiniband and using an old Mellanox DDR card (InfiniHost III Lx HCA).<br>
<br>
- # ip -br a<br>
<br>
options lnet networks=o2ib0(ib0)<br>
<br>
- # modprobe lnet<br>
- # lctl network up<br>
<br>
LNET configure error 100: Network is down<br>
<br>
- # tail -10 /var/log/kernel.log<br>
<br>
LNetError: 5071:0:(o2iblnd.c:2866:kiblnd_hdev_get_attr()) Invalid mr size: 0xffffffffffffffff<br>
LNetError: 5071:0:(o2iblnd.c:3103:kiblnd_dev_failover()) Can't get device attributes: -22<br>
LNetError: 5071:0:(o2iblnd.c:3831:kiblnd_startup()) ko2iblnd: Can't initialize device: rc = -22<br>
LNetError: Error -100 starting up LNI o2ib<br>
<br>
Lustre 2.15.0 and Ubuntu 20.04 (kernel 5.4.0-198-generic) is working fine with the same hardware<br>
<br>
Can anyone give me some advice or idea to make it work?<br>
<br>
Thans in advance<br>
Best regards<br>
<br>
--<br>
Ramiro Alba<br>
<br>
Centre Tecnològic de Tranferència de Calor<br>
<a href="http://www.cttc.upc.edu" rel="noreferrer" target="_blank">http://www.cttc.upc.edu</a><<a href="https://urldefense.com/v3/__http://www.cttc.upc.edu__;!!Mak6IKo!On9vgnDU5CEln4C9zazniBI1hEgioSxBPqr7Fd5blSIUQcojlPmtCAmRsP3OMqt4ZdEii93FRWH2FtVn8993JZ4Ixw$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__http://www.cttc.upc.edu__;!!Mak6IKo!On9vgnDU5CEln4C9zazniBI1hEgioSxBPqr7Fd5blSIUQcojlPmtCAmRsP3OMqt4ZdEii93FRWH2FtVn8993JZ4Ixw$</a>><br>
<br>
Escola Tècnica Superior d'Enginyeries<br>
Industrial i Aeronàutica de Terrassa<br>
Colom 11, E-08222, Terrassa, Barcelona, Spain<br>
Tel: (+34) 93 739 8928<br>
</blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr">Ramiro Alba <br> <br> Centre Tecnològic de Tranferència de Calor<br> <a href="http://www.cttc.upc.edu" rel="noreferrer" target="_blank">http://www.cttc.upc.edu</a><br> <br> Escola Tècnica Superior d'Enginyeries <br> Industrial i Aeronàutica de Terrassa <br> Colom 11, E-08222, Terrassa, Barcelona, Spain<br> Tel: (+34) 93 739 8928<br></div></div>