[lustre-discuss] max_frags incompatible

[Etienne Aujames]

Hi,

The CEA hit this kind of issue but with Lustre router (server 2.12 -> router 2.15). "lctl ping" from router to the server make the route go "up".

We solve this by setting map_on_demand=1 on the router.

Note here, map_on_demand=1 should be the default on a 2.15 to avoid this kind of issue: LU-15186 <https://jira.whamcloud.com/browse/LU-15186> Default ko2iblnd map_on_demand to 1

But, "lnetctl import" stills set map_on_demand=0 by default. This should be solve by LU-15538 <https://jira.whamcloud.com/browse/LU-15538>/LU-12452 <https://jira.whamcloud.com/browse/LU-12452>.

You can verify this with "lnetctl net show -v" on the server side.

In your case map_on_demand=1 on the client side (2.12) does not work because the 2.15 server asks for 257 frags but the 2.12 is not able to negotiate with this value (256 is the maximum for a 2.12 node).

Pinging from the server helps because a 2.15 node is able to negotiate frags (even with map_on_demand=0, LU-15094 <https://jira.whamcloud.com/browse/LU-15094>):

  * 2.15 node initiate the connection with max_frags=257
  * 2.12 node rejects the connection and requests max_frags=256
  * 2.15 node retries with max_frags=256 and save this value in memory for the remote peer
  * 2.12 node accepts the connection
  * 2.12 node initiate the connection with max_frags=256
  * 2.15 node accept the connection because it uses the save value of max_frags=256.

So, setting map_on_demand=1 on the server side (on the 2.15 nodes) should solve your issue.

Regards,

Etienne

On 7/24/25 22:23, Makia Minich via lustre-discuss wrote:
> Recently we upgraded our lustre servers to RHEL 8 with lustre 2.15.5 but due to scheduling the clients are still currently at RHEL 7 with lustre version 2.12.6. Infiniband is the interconnect.
>
> We've found that the client will fail to mount unless we run a "lnetctl ping" from the server side to the client. Once that happens then the client will ultimately mount. We've seen the following 
> error in the logs on the server side:
>
> lNet: 1920125:0:(o2iblnd_cd.c:2587:kiblnd_passive_connect()) Can’t accept conn from 172.16.19.6 at o2ib (version 12): max_frags 256 incompatible without FMR_pool (257 wanted)
>
> Attempting to set map_on_demand on the client side didn't help, resulting in the same error. Are there any parameters or configuration changes that may help the situation? At this time we aren't 
> able to upgrade the client side to RHEL 8, so we're ultimately limited on available versions, so looking for ideas on what to try next.
>
> Thanks.
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250725/b8ee569a/attachment.htm>