[lustre-discuss] Bad performance lustre

Benjamin GALINA benjamin.galina at eolen.com
Mon Nov 30 06:54:14 PST 2020


Hi Lustre Community ,

Could you help me solve my luster perfomance problem. ?
I have just set up several Lustre gateways between two infiniband networks A and B.

Node A          <--> Getway             <--> Node B
lustre 2.12.2          Lustre 2.13.3           Lustre 2.12.3

I am using ConnectX 5 Mellanox cards on servers with AMD Epyc processors.

I have tested and validated all speeds at infiniband (ib_read_bw and ib_write_bw) and tcp / ip (iperf3) level.

>From a lustre point of view (lnet selfttest the bit rates are correct between the nodes of A and the gateways.
On the other hand, the performance of the B nodes and the gateways are very very low:

[LNet Rates of servers]
[R] Avg: 26       RPC/s Min: 26       RPC/s Max: 26       RPC/s
[W] Avg: 36       RPC/s Min: 36       RPC/s Max: 36       RPC/s
[LNet Bandwidth of servers]
[R] Avg: 8.61     MiB/s Min: 8.61     MiB/s Max: 8.61     MiB/s
[W] Avg: 9.59     MiB/s Min: 9.59     MiB/s Max: 9.59     MiB/s
[LNet Rates of servers]
[R] Avg: 33       RPC/s Min: 33       RPC/s Max: 33       RPC/s
[W] Avg: 42       RPC/s Min: 42       RPC/s Max: 42       RPC/s
[LNet Bandwidth of servers]
[R] Avg: 8.62     MiB/s Min: 8.62     MiB/s Max: 8.62     MiB/s
[W] Avg: 8.81     MiB/s Min: 8.81     MiB/s Max: 8.81     MiB/s

Do you have any idea on the causes of these ridiculous flow rates?

Regards,






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20201130/a9de4ac3/attachment.html>


More information about the lustre-discuss mailing list