[lustre-discuss] [EXTERNAL]: Re: Terrible BW via lustre network

Wed Jun 10 13:43:24 UTC 2026

  given that configuration, assuming you have validated your client-to-server network speed and that you are using the appropriate lnd, then you for benchmarking, you might want to make sure you can generate many outstanding requests. Make sure you using several processes for the benchmark or multiple files simultaneously. You can adjust queue depth and use large I/O request sizes, say 16MB to help with getting multiple requests in flight.

kevin

________________________________________
From: Ganji, Prasadreddy <Prasadreddy.Ganji at kla.com>
Sent: Tuesday, June 9, 2026 3:19 PM
To: Harms, Kevin; lustre-discuss at lists.lustre.org
Subject: Re: [EXTERNAL]:  Re: Terrible BW via lustre network

Hi Kevin, Yes. Single OST -Prasad Get Outlook for iOS From: Harms, Kevin <harms@ alcf. anl. gov> Sent: Tuesday, June 9, 2026 4: 11: 12 PM To: lustre-discuss@ lists. lustre. org <lustre-discuss@ lists. lustre. org>; Ganji, Prasadreddy <Prasadreddy. Ganji@ kla. com>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd
Hi Kevin,
Yes. Single OST

-Prasad

Get Outlook for iOS<https://urldefense.us/v3/__https://aka.ms/o0ukef__;!!G_uCfscf7eWS!bDCTAE0Buvrn1CBLBPASKrMprnRNM4_2hFb71MyKJxLTGmtQKnffoHUnJurt_bkK42pUBff-iqB3KOwiEzMl9rEi83I$>
________________________________
From: Harms, Kevin <harms at alcf.anl.gov>
Sent: Tuesday, June 9, 2026 4:11:12 PM
To: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>; Ganji, Prasadreddy <Prasadreddy.Ganji at kla.com>
Subject: [EXTERNAL]: Re: Terrible BW via lustre network

  You said you have 1 OSS.. do you also have 1 OST? or more than one? if more than one, are you using more than one?

kevin

________________________________________
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of Ganji, Prasadreddy via lustre-discuss <lustre-discuss at lists.lustre.org>
Sent: Tuesday, June 9, 2026 2:40 PM
To: lustre-discuss at lists.lustre.org
Subject: [lustre-discuss] Terrible BW via lustre network

Hello folks, I'm relatively new to Lustre and experimenting. I'm seeing significantly lower throughput than expected on the client side and would appreciate any suggestions. Environment: OS: Rocky 8. 10 Lustre: 2. 15. 6 ZFS: 2. 1. 15-1 Cluster Layout

Hello folks,
I'm relatively new to Lustre and experimenting. I'm seeing significantly lower throughput than expected on the client side and would appreciate any suggestions.

Environment:
OS: Rocky 8.10
Lustre: 2.15.6
ZFS: 2.1.15-1
Cluster Layout (3 nodes):
IB HDR 200G connection b/w all 3 nodes are good.
1 Lustre server (combined MGS/MDS)
1 OSS (60 HDDs configured as a single dRAID3 pool, one vdev)
1 Client (using the same Lustre server image)

Problem:
Raw bandwidth per drive: ~250 MB/s
ZFS pool raw throughput (local): 5–5.5 GB/s
Lustre client mount throughput: <300 MB/s
There's a massive gap between what the pool can deliver locally and what the client sees over Lustre. I'd appreciate any guidance on what might be limiting performance — whether it's network, OST count, stripe configuration, ZFS tuning, or something else entirely.

Thanks,
Prasad

CONFIDENTIALITY NOTICE: This email transmission, and any documents, files or previous email messages attached to it, may contain confidential information. If you are not the intended recipient or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately destroy the original transmission and its attachments without reading them or saving them to disk. Email communications may be monitored by KLA as permitted by applicable law and regulations. For more information, read our Privacy Policy at http://www.kla.com/privacy-policy<https://urldefense.us/v3/__http://www.kla.com/privacy-policy__;!!G_uCfscf7eWS!bDCTAE0Buvrn1CBLBPASKrMprnRNM4_2hFb71MyKJxLTGmtQKnffoHUnJurt_bkK42pUBff-iqB3KOwiEzMlwtYyLAY$> .<https://urldefense.com/v3/__https://urldefense.us/v3/__http:/*www.kla.com/privacy-policy__;!!G_uCfscf7eWS!c2cCnzGauBUp6TMXT5GS9U73W03KeU_eLaL9rQIrfkF1N6URWr372eOoY77WbXCLhiE-jjW2ml1sLtOLp7Bnnbxj7P2rya_U$__;Lw!!LwW7MsFX3Kg!zKsxjH7l52l_yVr7mvgeI2CBygsZnunfQYQvSH29QMsmGikMIu4wOUcNV_icoTDHqFvkqHgAcQ7KPC-RHIcrsyo$<https://urldefense.com/v3/__https://urldefense.us/v3/__http:/*www.kla.com/privacy-policy__;!!G_uCfscf7eWS!c2cCnzGauBUp6TMXT5GS9U73W03KeU_eLaL9rQIrfkF1N6URWr372eOoY77WbXCLhiE-jjW2ml1sLtOLp7Bnnbxj7P2rya_U$__;Lw!!LwW7MsFX3Kg!zKsxjH7l52l_yVr7mvgeI2CBygsZnunfQYQvSH29QMsmGikMIu4wOUcNV_icoTDHqFvkqHgAcQ7KPC-RHIcrsyo$>> Thank you.

CONFIDENTIALITY NOTICE: This email transmission, and any documents, files or previous email messages attached to it, may contain confidential information. If you are not the intended recipient or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately destroy the original transmission and its attachments without reading them or saving them to disk. Email communications may be monitored by KLA as permitted by applicable law and regulations. For more information, read our Privacy Policy at http://www.kla.com/privacy-policy.<https://urldefense.us/v3/__http://www.kla.com/privacy-policy__;!!G_uCfscf7eWS!bDCTAE0Buvrn1CBLBPASKrMprnRNM4_2hFb71MyKJxLTGmtQKnffoHUnJurt_bkK42pUBff-iqB3KOwiEzMlwtYyLAY$> Thank you.