[lustre-discuss] Improving file create performance with larger create_count
Nathan Dauchy - NOAA Affiliate
nathan.dauchy at noaa.gov
Thu Jan 7 07:54:02 PST 2021
Greetings Lustre Experts!
I am looking for assistance on how to improve file create rate, as measured
In particular, this is for filesystems with (4) MDTs that use progressive
file layouts (PFL) to place the first part of each file on one of the (2)
Flash OSTs, with the remainder of large files on HDD OSTs using increasing
stripe count (up to 32) as the files get larger.
MDtest file create performance of this configuration is significantly lower
(160k vs. 550k) than when using a simple stripe count of 1 on HDDs alone.
MDtest was run with 0-size files (not using "-w") so no data should
actually be written to the OSTs, and the IOPS and CPU and Network of the
Flash OSSs should be plenty to sustain higher performance. Our vendor
pointed out that we are likely limited by the number of "precreated
objects" for the flash OSTs (since there are only two of them, handling all
files) and that it can be increased as an optimization.
I have found the "create_count" and "max_create_count" tunables, but the
Lustre manual only references those in the context of removing or disabling
an OST. So either the manual is incomplete or I'm looking at the wrong
If create_count is indeed the right parameter to adjust...
* What is the relation between create_count and max_create_count? And why
would I never see create_count more than half of max?
MGS# lctl get_param osc.*OST0000-osc-MDT0000*.*create_count
* Is there a hard-coded limit to the max value?
MGS# lctl set_param osc.FS1-OST0000-osc-MDT0000.max_create_count=40000
error: set_param: setting
Numerical result out of range
* Is there a theoretical down side to pre-creating more objects? (MDS or
OSS memory usage? Longer mount times? slower e2fsck?)
* Are there other tunings we should be looking at to improve performance
with Progressive File Layouts and our particular balance of just 2 Flash
OSTs to 32 HDD OSTs?
Thanks everyone for any suggestions!
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the lustre-discuss