<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, "Aptos_EmbeddedFont", "Aptos_MSFontService", Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi everyone,</div>
<div class="elementToProof" style="font-family: Aptos, "Aptos_EmbeddedFont", "Aptos_MSFontService", Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, "Aptos_EmbeddedFont", "Aptos_MSFontService", Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I hope this message find you well.</div>
<div class="elementToProof" style="font-family: Aptos, "Aptos_EmbeddedFont", "Aptos_MSFontService", Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, "Aptos_EmbeddedFont", "Aptos_MSFontService", Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I am unable to get Lustre RoCE v2 traffic to carry specific DSCP/TOS tags. While synthetic tests (ib_send_bw) successfully hit the desired hardware priority queues by selecting a specific GID index, Lustre traffic remains stuck at tos 0x1 (ECN enabled, DSCP
0), causing it to be mapped to the default Unicast queue (UC0) rather than the Lossless queue (UC3) on our SONiC switches.<br>
<br>
<b>System environment:</b><br>
<br>
OS: Rocky linux 9.6 (5.14.0-570.17.1.el9_6.x86_64)<br>
<br>
Lustre: 2.15.7<br>
<br>
NIC:<br>
driver: bnxt_en<br>
version: 1.10.3-233.0.198.0<br>
firmware-version: 233.0.196.0/pkg 23.31.18.10<br>
expansion-rom-version:<br>
bus-info: 0000:21:00.1<br>
supports-statistics: yes<br>
supports-test: yes<br>
supports-eeprom-access: yes<br>
supports-register-dump: yes<br>
supports-priv-flags: yes<br>
<br>
Switch:<br>
Software Version : 4.5.0a-Enterprise_Premium<br>
Product : Enterprise SONiC Distribution by Dell Technologies<br>
Distribution : Debian 11.11<br>
Kernel : 5.10.0-21-amd64<br>
Config DB Version : version_4_5_2<br>
<br>
lnet:<br>
net:<br>
- net type: lo<br>
local NI(s):<br>
- nid: 0@lo<br>
status: up<br>
statistics:<br>
send_count: 0<br>
recv_count: 0<br>
drop_count: 0<br>
tunables:<br>
peer_timeout: 0<br>
peer_credits: 0<br>
peer_buffer_credits: 0<br>
credits: 0<br>
lnd tunables:<br>
dev cpt: 0<br>
CPT: "[0,1,2,3,4,5,6,7]"<br>
- net type: o2ib1<br>
local NI(s):<br>
- nid: 172.16.7.13@o2ib1<br>
status: up<br>
interfaces:<br>
0: ens1f1np1<br>
statistics:<br>
send_count: 2314<br>
recv_count: 4361<br>
drop_count: 0<br>
tunables:<br>
peer_timeout: 180<br>
peer_credits: 128<br>
peer_buffer_credits: 0<br>
credits: 1024<br>
lnd tunables:<br>
peercredits_hiw: 64<br>
map_on_demand: 1<br>
concurrent_sends: 128<br>
fmr_pool_size: 512<br>
fmr_flush_trigger: 384<br>
fmr_cache: 1<br>
ntx: 512<br>
conns_per_peer: 1<br>
dev cpt: 2<br>
CPT: "[0,1,2,3,4,5,6,7]"<br>
<b><br>
Troubleshooting Steps Already Taken</b><br>
<br>
Manual TOS Overwrite: Attempted cma_roce_tos -d bnxt_re1 -t 104. Command returns successfully, but tcpdump confirms outgoing Lustre packets still carry tos 0x1.<br>
<br>
Kernel Mangle Bypass: Applied nftables (mangle table) rules to force DSCP 26 on UDP port 4791. Traffic remains 0x1, suggesting hardware offload bypasses the Linux network stack.<br>
<br>
Synthetic Success: Using ib_send_bw -x 3 (selecting GID Index 3) successfully changes the hardware queue and tagging. This proves the hardware is capable, but the Lustre kernel module isn't utilizing the correct GID index or TOS.<br>
<br>
Through all the investigation, I think Lustre (LNet and/or ko2iblnd) is not tagging the packets correctly, and I cannot find how to set it to use ToS 0x69. On the switch, it is still using UC0. I think the problem is with Lustre because if I use the ib_send_bw
-x 3 command, it does go through UC3.<br>
<br>
I would appreciate it if someone could give me some guidance to solve this problem.<br>
<br>
Thank you in advance.<br>
<br>
Warm regards,<br>
Victor</div>
<div class="elementToProof" style="font-family: Aptos, "Aptos_EmbeddedFont", "Aptos_MSFontService", Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
</body>
</html>