[lustre-devel] [External] : IOR MPIRUN read low performance and values are not consistent

[Patrick Farrell]

Muhammad,

Looking at your attached, both IOR and fio (mpirun is not the application, it's an execution framework) are configured to use 12 processes.  I encourage you to double check your CPU usage, what you're saying doesn't track unless something's very strangely configured.  It might be you've forced all the ranks to use 1 CPU or similar, but otherwise, it should be 12 CPUs.

You're using direct IO for IOR, so my best answer for your question about why read is much slower is you're perhaps seeing cache effects on your storage hardware.  Note FIO is using libaio, which is not the same as direct IO - it queues IOs rather than doing each one individually.  An iodepth of 128 with 256M block sizes is a total IO outstanding of 32 GB, and I think that's 32 GB per process.

In contrast, IOR is 12 * 128MB = 1.5 GB total IO outstanding.  Very different.

I'd encourage you to do some much simpler testing - single process with smaller amounts - and see what the results look like and build your understanding from that.

It would also be informative to not use Lustre and see what the IO behavior of your individual OSTs is when used as block devices.  (Obviously this requires reformatting)

Best of luck,
-Patrick
________________________________
From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of muhammad uzair jawaid <muhammaduzairjawaid at gmail.com>
Sent: Wednesday, October 9, 2024 10:44 AM
To: lustre-devel at lists.lustre.org <lustre-devel at lists.lustre.org>
Subject: [External] : [lustre-devel] IOR MPIRUN read low performance and values are not consistent

Hi,

I have 2 question

1) I run tests using fio and mpirun. The write speed for both are same but the read speed of mpirun is very low as compared to fio. My fio test gave around 8GB whereas my IOR test gave only 1.5GB.  I see on top utility mpirun read is only using 1 cpu whereas fio uses 12 cpus.

2) why the result in mpirun/fio read and write are not constant? I have isolated CPUS on numa 1 and my network interfaces are also on numa 1. On every run I see a different value which can deviate upto 4 GB .

I am attaching my test in a text file.
My setup is 2 lustre servers having 3 OSS and 1 MGS/MDS and 1 lustre client.  I am using rdma rocev2. 1 NIC card 100Gbps Connectx5 on each server.

Thanks for the help.
Best regards
Muhammad Uzair Jawaid
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20241009/741d2849/attachment.htm>