[Lustre-discuss] Lustre write performances 4 times faster after removing SD_IOSTATS from kernel

[Peter Grandi]

>> After performing some tests, I noticed that after removing
>> SD_IOSTATS functionality from the kernel, the write
>> performances of Lustre was 4 times faster. On a Lustre 1.8.7
>> client, I untared a tarfile of a directory containing about
>> 225,000 files (6.4GB).

> That is about 28kB/file, so it is causing a LOT of IO requests
> to the underlying disks.

Well, at least one per file, but I more suspicious of the MDS
when there is a lot of metadata traffic, and here there is file
creation and file closure (with update of the file size) every
28KiB. The size of each write IO is small, so the size of each
RPC, but that is reflected in the low overall transfer rate.

[ ... ]

>> With both servers, it took about 4 minutes to untar the
>> tarfile when running the kernel without SD_IOSTATS compared
>> to about 16 minutes when running the kernel with SD_IOSTATS.
>> Is that normal that SD_IOSTATS has a so big impact on Lustre
>> write performances?

> The sd_iostats code uses spin_lock_irqsave() to ensure
> consistent data structure access in interrupt context, so
> possibly this is causing the performance impact.  The impact
> might be more severe if there are many cores on the server.

It would be really unfortunate that it were severe to the point
that it takes 3 times as long to do that as to do 28KiB of IO.

Note the transfer rate: 6.4GB in 4 minutes is around 25-26MiB/s,
while in 16 minutes it reduces to around 6-7MiB/s.

Theese are not so good figures in absolute terms, and more
interestingly imply around ~1ms per file (~1,000 files/s) and
written per second) vs. 4ms per file (~250 files/s), with an
extra 3ms per file (allegedly) with 'SD_IOSTATS'.

Considering the lack of obviously useful information in the
original post, I am skeptical that 'SD_IOSTATS' directly
accounts for 3ms on top of 1ms of IO per file, that's why I
suspect at most some collateral effect.