[lustre-discuss] DIO/AIO performance at Lustre 2.15

[Landau Daniil]

Dear all,

I had a chance to try the Lustre DIO/AIO improvements appeared at 2.15 version (described here https://jira.whamcloud.com/browse/LU-13798 and here https://jira.whamcloud.com/browse/LU-13799 ). But It seems to me, that it does not fully work or I don’t understand how it should work.

I’ve two servers as OSSs with one RAID-on-NVMe-based OST on each and a client. MDS and MGT work at another server. All servers and client are interconnected with Infiniband HDR. All the settings during the filesystem creation and mount were default.
The test command is:

ior -w -r -t 256M -b 64G -e  -o /<dirpath>/iorfile --posix.odirect

I also applied the following tunning at the client:

lctl set_param osc.*.max_pages_per_rpc=4M osc.*.checksums=0 osc.*.max_rpcs_in_flight=16

So, when I run a test without any striping settings (and the test file appears only on one OST) the results are about:

Max Write: 4008.78 MiB/sec (4203.51 MB/sec)
Max Read:  7061.42 MiB/sec (7404.44 MB/sec)

When I set striping to “-S 2M -c 2” the results are about:

Max Write: 8796.89 MiB/sec (9224.21 MB/sec)
Max Read:  19043.23 MiB/sec (19968.27 MB/sec)

But when I disable caching at both OSSs with the following commands

/usr/sbin/lctl set_param osd-ldiskfs.*.read_cache_enable=0
/usr/sbin/lctl set_param osd-ldiskfs.*.writethrough_cache_enable=0

the striping results became about:

Max Write: 13054.52 MiB/sec (13688.65 MB/sec)
Max Read:  18323.93 MiB/sec (19214.03 MB/sec)

It looks like read and write caching on the OSS side works even if the test is launched with --posix.odirect flag, unless caching is manually disabled at OSS side. And cache logic calculations slow down the overall performance for writing (which is an expected behavior in case of NVMe-based storage). It looks strange for me, since as I got from some docs, when operations come with O_DIRECT flag, it should avoid any caching on both sides – on client side and on server side too. So, from my point of view, DIO just does not work or works incorrectly.

May I ask dear all enlighten me in this question: maybe my point of view is incorrect, or I missed some settings during the filesystem creation and mounting needed to let DIO work properly.

Thank you!

Daniel




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220819/877f0d5c/attachment-0001.htm>