<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Hi Lustre Community <span style="font-size: 12pt;">,</span></p>
<p></p>
<div><br>
</div>
<div>Could you help me solve my luster perfomance problem. ? </div>
<div>I have just set up several Lustre gateways between two infiniband networks A and B.</div>
<div><br>
</div>
<div>Node A <--> Getway <--> Node B </div>
<div>lustre 2.12.2 Lustre 2.13.3 Lustre 2.12.3</div>
<div><br>
</div>
<div>I am using ConnectX 5 Mellanox cards on servers with AMD Epyc processors.</div>
<div><br>
</div>
<div>I have tested and validated all speeds at infiniband (ib_read_bw and ib_write_bw) and tcp / ip (iperf3) level.</div>
<div><br>
</div>
<div>From a lustre point of view (lnet selfttest the bit rates are correct between the nodes of A and the gateways. </div>
<div>On the other hand, the performance of the B nodes and the gateways are very very low: </div>
<div><br>
</div>
<div>
<div><span style="font-size: 12pt;">[LNet Rates of servers]</span><br>
</div>
<div>[R] Avg: 26 RPC/s Min: 26 RPC/s Max: 26 RPC/s</div>
<div>[W] Avg: 36 RPC/s Min: 36 RPC/s Max: 36 RPC/s</div>
<div>[LNet Bandwidth of servers]</div>
<div>[R] Avg: 8.61 MiB/s Min: 8.61 MiB/s Max: 8.61 MiB/s</div>
<div>[W] Avg: 9.59 MiB/s Min: 9.59 MiB/s Max: 9.59 MiB/s</div>
<div>[LNet Rates of servers]</div>
<div>[R] Avg: 33 RPC/s Min: 33 RPC/s Max: 33 RPC/s</div>
<div>[W] Avg: 42 RPC/s Min: 42 RPC/s Max: 42 RPC/s</div>
<div>[LNet Bandwidth of servers]</div>
<div>[R] Avg: 8.62 MiB/s Min: 8.62 MiB/s Max: 8.62 MiB/s</div>
<div>[W] Avg: 8.81 MiB/s Min: 8.81 MiB/s Max: 8.81 MiB/s</div>
<div><br>
</div>
<div><span>Do you have any idea on the causes of these ridiculous flow rates?</span><br>
</div>
<div><span><br>
</span></div>
<div>Regards,</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
</div>
<br>
<p></p>
</div>
</body>
</html>