<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
As a further troubleshooting step, I would suggest enabling neterror in the printk mask on the client and LNet routers:</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Consolas, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
lctl set_param printk=+neterror</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This may surface additional information around the routes going down.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Another thing you ought to try is checking connectivity between client and routers after the routes get marked down. Do pings over the LNet interface work?</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Consolas, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
ping -I <client_ip> <router_ip></div>
<div dir="ltr" style="font-family: Consolas, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
lnetctl ping --source <client_nid> <router_nid></div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
There were only a handful of LNet changes, so it is unlikely to be some regression in LNet.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> git -P le 2.15.7 ^2.15.6 lnet</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
17fc6dbcd6 LU-17784 build: improve wiretest for flexible arrays</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
8535cfe29a LU-18572 lnet: Uninitialized var in lnet_peer_add</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
c00bb50624 LU-18697 lnet: lnet_peer_del_nid refcount loss</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
9d8dbed27c LU-16594 build: get_random_u32_below, get_acl with dentry</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
247ae64877 LU-17081 build: compatibility for 6.5 kernels</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
></div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Chris Horn</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="mail-editor-reference-message-container" style="color: inherit; background-color: inherit;">
<div class="ms-outlook-mobile-reference-message skipProofing">
<meta name="Generator" content="Microsoft Exchange Server" style="color: inherit; background-color: inherit;">
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="text-align: left; padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Michael DiDomenico via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Date: </b>Thursday, October 30, 2025 at 2:08 PM<br>
<b>To: </b>lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[lustre-discuss] client failing off network<br>
<br>
</div>
<div class="PlainText" style="font-size: 11pt;">our network is running 2.15.6 everywhere on rhel9.5, we recently built<br>
a new machine using 2.15.7 on rhel9.6 and i'm seeing a strange<br>
problem. the client is ethernet connected to ten lnet routers which<br>
bridge ethernet to infiniband.<br>
<br>
i can mount the client just fine, read/write data, but then several<br>
hours later, the client marks all the routers offline. the only<br>
recovery is to lazy unmount, lustre_rmmod, and then restart the lustre<br>
mount<br>
<br>
nothing unusual comes out in the journal/dmesg logs. to lustre it<br>
"looks" like someone pulled the network cable, but there's no evidence<br>
that this has happened physically or even at the switch/software<br>
layers<br>
<br>
we upgraded two other machine to see if the problem replicates, but so<br>
far it hasn't. the only significant difference between the three<br>
machines is the one with the problem has heavy container (podman)<br>
usage, the others have zero. i'm not sure if this is an cause or just<br>
a red herring<br>
<br>
any suggestions?<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!NpxR!ibXFDE5f0Z10bD2MkR6l2DaJMCZpX6tzg8uJXOztC1mZt_r7Or5inWyefgVRAv10RUkPfLDg73fzg3o7ppoMYTibfHs2$" data-outlook-id="a47b3e90-af3d-4488-b404-38d71e4f98f9">https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!NpxR!ibXFDE5f0Z10bD2MkR6l2DaJMCZpX6tzg8uJXOztC1mZt_r7Or5inWyefgVRAv10RUkPfLDg73fzg3o7ppoMYTibfHs2$</a><br>
</div>
</div>
</body>
</html>