<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hello Wangshuo,</div>
<div style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I apologize if I made a mistake with your name.</div>
<div style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I appreciate the response. However, I am a little bit afraid of changing and recompiling the drivers; there are other programs, and I do not know how it will affect them if I make that change. I would like to try other options first before recompiling.</div>
<div style="margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Warm regards,<br>
Victor</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>De:</b> wangshuobin <w14767780617@163.com><br>
<b>Enviado:</b> jueves, 30 de abril de 2026 12:15 a. m.<br>
<b>Para:</b> VICTOR MANUEL MINJARES NERIZ <victor.minjares@unison.mx><br>
<b>Asunto:</b> Re:[lustre-discuss] Lustre RoCE v2 traffic issue</font>
<div> </div>
</div>
<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; border:0!important; display:table!important; width:100%!important; table-layout:fixed!important; border-collapse:seperate!important; float:none!important; border-spacing:0px 0px!important">
<tbody style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; display:block!important">
<tr style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important">
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; padding:7px 2px 7px 2px!important; background-color:#A6A6A6!important; width:0px!important">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; width:100%!important; background-color:#EAEAEA!important; padding:7px 5px 7px 15px!important; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif!important; font-size:12px!important; font-weight:normal!important; color:#212121!important; text-align:left!important; word-wrap:break-word!important">
<div style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important">
No suele recibir correo electr¨®nico de w14767780617@163.com. <a href="https://aka.ms/LearnAboutSenderIdentification" style="background:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; opacity:revert!important; visibility:revert!important">
Por qu¨¦ es esto importante</a> </div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; width:75px!important; background-color:#EAEAEA!important; padding:7px 5px 7px 5px!important; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif!important; font-size:12px!important; font-weight:normal!important; color:#212121!important; text-align:left!important; word-wrap:break-word!important">
</td>
</tr>
</tbody>
</table>
<div>
<div data-ntes="ntes_mail_body_root" style="line-height:1.7; color:#000000; font-size:14px; font-family:Arial">
<div id="x_spnEditorContent">
<div style="margin:0">hello£¬</div>
<div style="margin:0"><br>
</div>
<div style="margin:0">
<p data-start="54" data-end="83">I¡¯ve noticed a similar issue:</p>
<p data-start="85" data-end="299">On the server side I¡¯m using an MLX NIC, and on the client side an Intel E810 NIC. When the RDMA connection is initiated by the E810 NIC, even if the priority is set to 3, the traffic still goes through priority 0.</p>
<p data-start="301" data-end="412">However, in the opposite case¡ªwhen the MLX NIC initiates the RDMA connection¡ªit can use priority 0 as expected.</p>
<p data-start="414" data-end="504">You can try modifying the kernel source file <code data-start="459" data-end="466">
cma.c</code> by commenting out the following code:</p>
<pre class="x_overflow-visible! x_px-0!" data-start="506" data-end="684"><div class="x_contain-inline-size x_rounded-2xl x_corner-superellipse/1.1 x_relative x_bg-token-sidebar-surface-primary"><div class="x_sticky x_top-[calc(var(--sticky-padding-top)+9*var(--spacing))]"><div class="x_absolute x_end-0 x_bottom-0 x_flex x_h-9 x_items-center x_pe-2"><div class="x_bg-token-bg-elevated-secondary x_text-token-text-secondary x_flex x_items-center x_gap-4 x_rounded-sm x_px-2 x_font-sans x_text-xs"></div></div></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c">static int get_vlan_ndev_tc(struct net_device *vlan_ndev, int prio)
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c">{
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> struct net_device *dev;
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> int tc;
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c">
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> dev = vlan_dev_real_dev(vlan_ndev);
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> pr_info("cma: get_vlan_ndev_tc: vlan_ndev=%s, real_dev=%s, prio=%d, num_tc=%d\n",
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> vlan_ndev->name, dev->name, prio, dev->num_tc);
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c">
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> //if (dev->num_tc) {
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> // tc = netdev_get_prio_tc_map(dev, prio);
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> // pr_info("cma: get_vlan_ndev_tc: path=netdev_get_prio_tc_map, tc=%d\n", tc);
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> // return tc;
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> //}
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c">
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> tc = (vlan_dev_get_egress_qos_mask(vlan_ndev, prio) &
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT;
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> pr_info("cma: new get_vlan_ndev_tc: path=vlan_egress_qos, egress_mask=0x%x, tc=%d\n",
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> vlan_dev_get_egress_qos_mask(vlan_ndev, prio), tc);
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"> return tc;
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c">}
</code></div><div class="x_overflow-y-auto x_p-4" dir="ltr"><code class="x_whitespace-pre! x_language-c"><br></code></div></div></pre>
<p data-start="686" data-end="776">Then recompile and reload it on the client where the E810 NIC is installed and test again.</p>
<p data-start="778" data-end="896">My NIC is an E810. Since you¡¯re using a Broadcom NIC, it might be a similar issue, so you can try this as a reference.</p>
</div>
<div style="margin:0"><br>
</div>
<div style="margin:0"><br>
</div>
<div style="margin:0">
<p data-start="39" data-end="131">The root cause is that the priority used by RDMA is determined by the CM connection request.</p>
<p data-start="133" data-end="431">When the E810 NIC initiates an RDMA connection, its
<code data-start="185" data-end="198">dev->num_tc</code> default value is 1 (instead of 0). As a result, in the
<code data-start="254" data-end="274">get_vlan_ndev_tc()</code> function, the code follows the
<code data-start="306" data-end="330">netdev_get_prio_tc_map</code> branch. This ultimately causes the PRIMARY_SL field in the transmitted CM REQ packet to be set to 0.</p>
<p data-start="433" data-end="547">The MLX responder then uses this value to set the priority, which results in the traffic going through priority 0.</p>
</div>
<div style="margin:0"><br>
</div>
<div style="margin:0"><br>
</div>
<div style="margin:0"><br>
</div>
</div>
<div style="zoom:1"></div>
<div id="x_divNeteaseMailCard"></div>
<div style="margin:0"><br>
</div>
<p>At 2026-04-30 05:52:26, "VICTOR MANUEL MINJARES NERIZ via lustre-discuss" <lustre-discuss@lists.lustre.org> wrote:</p>
<blockquote id="x_isReplyContent" style="padding-left:1ex; margin:0px 0px 0px 0.8ex; border-left:#ccc 1px solid">
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_elementToProof" style="font-family:Aptos,"Aptos_EmbeddedFont","Aptos_MSFontService",Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi everyone,</div>
<div class="x_elementToProof" style="font-family:Aptos,"Aptos_EmbeddedFont","Aptos_MSFontService",Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,"Aptos_EmbeddedFont","Aptos_MSFontService",Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I hope this message find you well.</div>
<div class="x_elementToProof" style="font-family:Aptos,"Aptos_EmbeddedFont","Aptos_MSFontService",Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,"Aptos_EmbeddedFont","Aptos_MSFontService",Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am unable to get Lustre RoCE v2 traffic to carry specific DSCP/TOS tags. While synthetic tests (ib_send_bw) successfully hit the desired hardware priority queues by selecting a specific GID index, Lustre traffic remains stuck at tos 0x1 (ECN enabled, DSCP
0), causing it to be mapped to the default Unicast queue (UC0) rather than the Lossless queue (UC3) on our SONiC switches.<br>
<br>
<b>System environment:</b><br>
<br>
OS: Rocky linux 9.6 (5.14.0-570.17.1.el9_6.x86_64)<br>
<br>
Lustre: 2.15.7<br>
<br>
NIC:<br>
driver: bnxt_en<br>
version: 1.10.3-233.0.198.0<br>
firmware-version: 233.0.196.0/pkg 23.31.18.10<br>
expansion-rom-version:<br>
bus-info: 0000:21:00.1<br>
supports-statistics: yes<br>
supports-test: yes<br>
supports-eeprom-access: yes<br>
supports-register-dump: yes<br>
supports-priv-flags: yes<br>
<br>
Switch:<br>
Software Version : 4.5.0a-Enterprise_Premium<br>
Product : Enterprise SONiC Distribution by Dell Technologies<br>
Distribution : Debian 11.11<br>
Kernel : 5.10.0-21-amd64<br>
Config DB Version : version_4_5_2<br>
<br>
lnet:<br>
net:<br>
- net type: lo<br>
local NI(s):<br>
- nid: 0@lo<br>
status: up<br>
statistics:<br>
send_count: 0<br>
recv_count: 0<br>
drop_count: 0<br>
tunables:<br>
peer_timeout: 0<br>
peer_credits: 0<br>
peer_buffer_credits: 0<br>
credits: 0<br>
lnd tunables:<br>
dev cpt: 0<br>
CPT: "[0,1,2,3,4,5,6,7]"<br>
- net type: o2ib1<br>
local NI(s):<br>
- nid: 172.16.7.13@o2ib1<br>
status: up<br>
interfaces:<br>
0: ens1f1np1<br>
statistics:<br>
send_count: 2314<br>
recv_count: 4361<br>
drop_count: 0<br>
tunables:<br>
peer_timeout: 180<br>
peer_credits: 128<br>
peer_buffer_credits: 0<br>
credits: 1024<br>
lnd tunables:<br>
peercredits_hiw: 64<br>
map_on_demand: 1<br>
concurrent_sends: 128<br>
fmr_pool_size: 512<br>
fmr_flush_trigger: 384<br>
fmr_cache: 1<br>
ntx: 512<br>
conns_per_peer: 1<br>
dev cpt: 2<br>
CPT: "[0,1,2,3,4,5,6,7]"<br>
<b><br>
Troubleshooting Steps Already Taken</b><br>
<br>
Manual TOS Overwrite: Attempted cma_roce_tos -d bnxt_re1 -t 104. Command returns successfully, but tcpdump confirms outgoing Lustre packets still carry tos 0x1.<br>
<br>
Kernel Mangle Bypass: Applied nftables (mangle table) rules to force DSCP 26 on UDP port 4791. Traffic remains 0x1, suggesting hardware offload bypasses the Linux network stack.<br>
<br>
Synthetic Success: Using ib_send_bw -x 3 (selecting GID Index 3) successfully changes the hardware queue and tagging. This proves the hardware is capable, but the Lustre kernel module isn't utilizing the correct GID index or TOS.<br>
<br>
Through all the investigation, I think Lustre (LNet and/or ko2iblnd) is not tagging the packets correctly, and I cannot find how to set it to use ToS 0x69. On the switch, it is still using UC0. I think the problem is with Lustre because if I use the ib_send_bw
-x 3 command, it does go through UC3.<br>
<br>
I would appreciate it if someone could give me some guidance to solve this problem.<br>
<br>
Thank you in advance.<br>
<br>
Warm regards,<br>
Victor</div>
<div class="x_elementToProof" style="font-family:Aptos,"Aptos_EmbeddedFont","Aptos_MSFontService",Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>