[lustre-discuss] Lustre RoCE v2 traffic issue

[VICTOR MANUEL MINJARES NERIZ]

Hello Serguei,
Thank you for your response, it worked!!!
I added your suggestion to the /etc/modprobe.d/lustre.conf file; this is its content:
options lnet networks="o2ib1(ens1f1np1)"
options ko2iblnd tos=0x69 peer_credits=128 credits=1024 concurrent_sends=128
options ksocklnd avoid_asym_router=1
Then I unloaded and reloaded Lustre, and it started using UC3 on the switch.
Thanks again for your help.
Warm regards,
Victor
________________________________
De: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> en nombre de Serguei Smirnov via lustre-discuss <lustre-discuss at lists.lustre.org>
Enviado: martes, 5 de mayo de 2026 10:11 a. m.
Para: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
Asunto: Re: [lustre-discuss] Lustre RoCE v2 traffic issue


Hi Victor,

You may want to try setting the o2iblnd module parameter "tos".

For example (in the conf file):
  options ko2iblnd tos=0x68

This controls the TOS value used by o2iblnd when establishing

RDMA CM connections, and may influence which SL/priority
queue is selected for RoCE v2 traffic.

Note that:

- This is applied at connection setup time, so existing
  connections would need to be re-established to pick up the change.

- The actual mapping to hardware queues depends on the NIC

Given that “ib_send_bw -x <gid>” works in your setup, it’s also possible
that the effective behavior is driven more by GID selection than by
TOS alone, and o2iblnd does not currently expose a way to control GID
index explicitly.

Still, setting "tos" is probably the first thing worth trying.

Cheers,

Serguei


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260506/c8fd85bc/attachment.htm>