[lustre-discuss] Improving file create performance with larger create_count

[Nathan Dauchy - NOAA Affiliate]

Greetings Lustre Experts!

I am looking for assistance on how to improve file create rate, as measured
with MDtest.

In particular, this is for filesystems with (4) MDTs that use progressive
file layouts (PFL) to place the first part of each file on one of the (2)
Flash OSTs, with the remainder of large files on HDD OSTs using increasing
stripe count (up to 32) as the files get larger.
MDtest file create performance of this configuration is significantly lower
(160k vs. 550k) than when using a simple stripe count of 1 on HDDs alone.
MDtest was run with 0-size files (not using "-w") so no data should
actually be written to the OSTs, and the IOPS and CPU and Network of the
Flash OSSs should be plenty to sustain higher performance.  Our vendor
pointed out that we are likely limited by the number of "precreated
objects" for the flash OSTs (since there are only two of them, handling all
files) and that it can be increased as an optimization.

I have found the "create_count" and "max_create_count" tunables, but the
Lustre manual only references those in the context of removing or disabling
an OST.  So either the manual is incomplete or I'm looking at the wrong
tunable.

If create_count is indeed the right parameter to adjust...
* What is the relation between create_count and max_create_count?  And why
would I never see create_count more than half of max?
MGS# lctl get_param osc.*OST0000-osc-MDT0000*.*create_count
osc.FS1-OST0000-osc-MDT0000.create_count=10000
osc.FS1-OST0000-osc-MDT0000.max_create_count=20000
* Is there a hard-coded limit to the max value?
MGS# lctl set_param osc.FS1-OST0000-osc-MDT0000.max_create_count=40000
error: set_param: setting
/sys/fs/lustre/osc/FS1-OST0000-osc-MDT0000/max_create_count=40000:
Numerical result out of range
* Is there a theoretical down side to pre-creating more objects?  (MDS or
OSS memory usage?  Longer mount times? slower e2fsck?)
* Are there other tunings we should be looking at to improve performance
with Progressive File Layouts and our particular balance of just 2 Flash
OSTs to 32 HDD OSTs?

Thanks everyone for any suggestions!

-Nathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20210107/197a12f9/attachment.html>