<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hello Serguei,</div>
<div style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thank you for your response, it worked!!!<br>
I added your suggestion to the /etc/modprobe.d/lustre.conf file; this is its content:<br>
options lnet networks="o2ib1(ens1f1np1)"<br>
options ko2iblnd tos=0x69 peer_credits=128 credits=1024 concurrent_sends=128<br>
options ksocklnd avoid_asym_router=1</div>
<div style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Then I unloaded and reloaded Lustre, and it started using UC3 on the switch.</div>
<div style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thanks again for your help.</div>
<div style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Warm regards,<br>
Victor</div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" class="elementToProof">
<div style="direction: ltr; font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof">
<b>De:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> en nombre de Serguei Smirnov via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Enviado:</b> martes, 5 de mayo de 2026 10:11 a. m.<br>
<b>Para:</b> lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Asunto:</b> Re: [lustre-discuss] Lustre RoCE v2 traffic issue</div>
<div style="direction: ltr;" class="elementToProof"> </div>
</div>
<p style="margin: 0cm; font-family: "Calibri", sans-serif; font-size: 12pt;" class="elementToProof">
<span style="font-family: "Courier New"; font-size: 10pt;">Hi Victor,<br>
<br>
You may want to try setting the o2iblnd module parameter "tos".<br>
<br>
For example (in the conf file):<br>
options ko2iblnd tos=0x68<br>
<br>
This controls the TOS value used by o2iblnd when establishing</span></p>
<p style="margin: 0cm; font-family: "Calibri", sans-serif; font-size: 12pt;" class="elementToProof">
<span style="font-family: "Courier New"; font-size: 10pt;">RDMA CM connections, and may influence which SL/priority<br>
queue is selected for RoCE v2 traffic.<br>
<br>
Note that:<br>
<br>
- This is applied at connection setup time, so existing<br>
connections would need to be re-established to pick up the change.<br>
<br>
- The actual mapping to hardware queues depends on the NIC</span></p>
<p style="margin: 0cm; font-family: "Calibri", sans-serif; font-size: 12pt;" class="elementToProof">
<span style="font-family: "Courier New"; font-size: 10pt;"><br>
Given that ib_send_bw -x <gid> works in your setup, its also possible<br>
that the effective behavior is driven more by GID selection than by<br>
TOS alone, and o2iblnd does not currently expose a way to control GID<br>
index explicitly.<br>
<br>
Still, setting "tos" is probably the first thing worth trying.<br>
<br>
Cheers,</span></p>
<p style="margin: 0cm; font-family: "Calibri", sans-serif; font-size: 12pt;" class="elementToProof">
<span style="font-family: "Courier New"; font-size: 10pt;">Serguei</span></p>
<p style="margin: 0cm; font-family: "Calibri", sans-serif; font-size: 12pt;" class="elementToProof">
<span style="font-size: 11pt;"> </span></p>
</body>
</html>