[lustre-discuss] Terrible BW via lustre network

[Feng Zhang]

Is your lustre network set to be IB?

https://wiki.lustre.org/Starting_and_Stopping_LNet

Best,

Feng

On Tue, Jun 9, 2026 at 3:59 PM Ganji, Prasadreddy via lustre-discuss
<lustre-discuss at lists.lustre.org> wrote:
>
> Hello folks,
>
> I'm relatively new to Lustre and experimenting. I'm seeing significantly lower throughput than expected on the client side and would appreciate any suggestions.
>
>
>
> Environment:
>
> OS: Rocky 8.10
>
> Lustre: 2.15.6
>
> ZFS: 2.1.15-1
>
> Cluster Layout (3 nodes):
>
> IB HDR 200G connection b/w all 3 nodes are good.
>
> 1 Lustre server (combined MGS/MDS)
>
> 1 OSS (60 HDDs configured as a single dRAID3 pool, one vdev)
>
> 1 Client (using the same Lustre server image)
>
>
>
> Problem:
>
> Raw bandwidth per drive: ~250 MB/s
>
> ZFS pool raw throughput (local): 5–5.5 GB/s
>
> Lustre client mount throughput: <300 MB/s
>
> There's a massive gap between what the pool can deliver locally and what the client sees over Lustre. I'd appreciate any guidance on what might be limiting performance — whether it's network, OST count, stripe configuration, ZFS tuning, or something else entirely.
>
>
>
> Thanks,
>
> Prasad
>
>
>
>
>
>
> CONFIDENTIALITY NOTICE: This email transmission, and any documents, files or previous email messages attached to it, may contain confidential information. If you are not the intended recipient or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately destroy the original transmission and its attachments without reading them or saving them to disk. Email communications may be monitored by KLA as permitted by applicable law and regulations. For more information, read our Privacy Policy at www.kla.com/privacy-policy. Thank you.
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org