<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
What Lustre version are you using on clients and servers?</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Chris Horn</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="mail-editor-reference-message-container" style="color: inherit; background-color: inherit;">
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
</div>
<div style="padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor;">
<div class="ms-outlook-mobile-reference-message skipProofing" style="text-align: left; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Stepan Beskrovnyy via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Date: </b>Thursday, July 23, 2026 at 12:56\u202fPM<br>
<b>To: </b>lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[lustre-discuss] Lustre Multi-Rail Troubles<br>
<br>
</div>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing">Hey everyone ! I got a simple multi-rail configure without any routers, only peers with direct ROCEv2 connection. </div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
I got mdt/mds with 2 networks: ens30, ens60 </div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
I got OSS with same networks </div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
I got client with same networks</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
2 networks works in Multi-Rail and everything fine. </div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
Then I starting I/O benchmark on mounted fs on client. Then I disable ens30 on OSS with ip link set ens30 down. </div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
lnetctl net show marks ens30 as down and all of network bandwith goes through ens60(Thats good) </div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
Then I set ens30 up, by ip link set ens30</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
After a small timeout lnetctl marks ens30 as up, but all bandwidth still goes through ens60, and nothing goes through upped ens30. </div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
Is there any way to restore traffic on ens30 without stopping I/O and without reloading lnet kernel module ? I did any tricks with adding and deeting peers, but they do nothing. </div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
Any ideas? </div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
<br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
Thanks, </div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="direction: ltr;">
Stepan</div>
</div>
</body>
</html>