<div dir="ltr"><div>Andreas, thanks for the insight and advice. Followup details inline below...</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 7, 2021 at 9:56 PM Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
On Jan 7, 2021, at 08:54, Nathan Dauchy - NOAA Affiliate wrote:<br>
<div>
<blockquote type="cite"><div dir="ltr"><div>I am looking for assistance on how to improve file create rate, as measured with MDtest.</div></div></blockquote>
<div>one thing to pay attention to is how large your PFL layout is. If it is 4 components and/or you have an MDT formatted with an older (pre 2.10) version of Lustre, then the amount of xattr space for the layout directly in the MDT inode is relatively small.
Limiting the PFL layout to 3 components and having an MDT inode size of 1KB should avoid the extra overhead of writing to a separate xattr block for each create.</div></div></div></blockquote><div><br></div><div>The PFL layout has 5 components, setup like this:</div><div>lfs setstripe -E 128K -c 1 -S 128K -p SSD -E 32M -S 1M -c 1 -p HDD -E 1G -c 4 -S 64M -E 32G -c 8 -E -1 -c 30<br></div><div><br></div><div>The MDT is brand new, formatted with 2.12+.</div><div>Using "dumpe2fs -h" it appears as though the inode size is indeed 1KB:</div><div>dumpe2fs 1.45.2.cr2 (09-Apr-2020)<br>Inode size: 1024<br></div><div><br></div><div>Does the following debugfs command confirm or refute whether file layouts created with that config are staying within the MDT inode? (I'm not sure if I can just do 32+24+632+47 = 735 < 1024, or if there is another constant value that needs to be added in.)</div><div><br></div><div>MDS3# debugfs -c -R 'stat /REMOTE_PARENT_DIR/0x2c000f630:0x1:0x0/Nathan.Dauchy/empty' /dev/md66 <br></div><div>Size of extra inode fields: 32<br>Extended attributes:<br> trusted.lma (24) = 00 00 00 00 00 00 00 00 33 f6 00 c0 02 00 00 00 02 00 00 00 00 00 00 00 <br> lma: fid=[0x2c000f633:0x2:0x0] compat=0 incompat=0<br> trusted.lov (632)<br> trusted.link (47)<br> trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <br></div><div><br></div><div>If I use a 3-component layout and an empty file like this:</div><div>lfs setstripe -E 32M -S 1M -c 1 -p HDD -E 1G -c 4 -S 64M -E -1 -c 8 3comp</div><div>...then the "trusted.lov" value is the only one that decreases:<br> trusted.lma (24) = 00 00 00 00 00 00 00 00 33 f6 00 c0 02 00 00 00 07 00 00 00 00 00 00 00 <br> lma: fid=[0x2c000f633:0x7:0x0] compat=0 incompat=0<br> trusted.link (47)<br> trusted.lov (392)<br> trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div>
<div>The other potential source of overhead is if the flash OSTs have a very large number of objects vs. the HDD OSTs (e.g. hundreds of millions if there is *always* a flash component for each file and they are never removed from large files during OST migration)
which make the OST object directories very large, and each create essentially goes into a separate directory block. The saving grace in your case is that the flash OSTs have higher IOPS, so they _should_ be able to keep up with the HDD OSTs even if they have
a higher workload.<br></div></div></div></blockquote><div><br></div><div>I don't believe this applies (to us at least) as it is a new and fairly empty filesystem. Thanks for the heads up for what to watch for in the future though!</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><div></div>
<div>You could try isolating whether the bottleneck is from the 2 flash OSTs to 32 HDD OSTs, or because of the large layout size. Try creating 1-stripe files in individual directories that are each setstripe to create only on a single OST. That should show
what the single OST create rate limit is. You could compare this against the no-stripe create rate for the MDS.<br></div>
<div><br>
</div>
<div>On the flip side, you could create files in a directory with a large layout that has many components, like:</div>
<div><br>
</div>
<div> lfs setstripe -E 16M -c 1 -i 1 -E 32M -i 2 -E 48M -i 3 -E eof -i 4 $MOUNT/ost1largedir</div>
<div><br>
</div>
<div>to see how this affects the create rate. The OST objects would be consumed 4x as fast, but from 4x OSTs (assume they are the HDD OSTs) so they _should_ be creating/consuming OST objects at the same rate as a 1-stripe file file, except the large layout
size will push this to a separate xattr block and put much more IOPS onto the MDS. You could try with the two flash OSTs, two objects on each of the two OSTs, which should get you 1/2 the create rate if the OST create rate is the limiting factor. If this
tanks performance then the issue is with the large layout xattr and not the OSTs.</div></div></div></blockquote><div><br></div><div>Additional benchmarking is in progress already. If we need more clues I'll add that to the list.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><blockquote type="cite"><div><div dir="ltr"><div>I have found the "create_count" and "max_create_count" tunables, but the Lustre manual only references those in the context of removing or disabling an OST. So either the manual is incomplete or I'm looking at the wrong tunable.</div></div></div></blockquote>
The important place to check the create_count/max_create_count on the MDS, since it is the one driving object creates to the OSTs.</div></blockquote><div><br></div><div>Understood, that "MGS" was my typo replacing a hostname. It's the "MDS".</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">
<blockquote type="cite">
<div dir="ltr">
<div>* Are there other tunings we should be looking at to improve performance with Progressive File Layouts and our particular balance of just 2 Flash OSTs to 32 HDD OSTs?<br></div></div></blockquote><div>
<br>
</div>
<div>Depends what the above results show. There is the "obdfilter.*.precreate_batch" tunable, which can help optimize OST creates if there is a lot of lock contention on the object directories for create vs. concurrent IO, but it is unlikely to be an issue
under normal usage. If the problem is that the OSTs have huge numbers of objects and large object directories there are other potential optimizations possible.</div></div></blockquote><div><br></div><div>With the simple mdtest 0-byte file test, it doesn't sound like contention would be the problem. Neither is it that the OSTs have huge number of objects (yet).</div><div><br></div><div>We will keep digging.</div><div><br></div><div>Thanks again,</div><div>Nathan</div><div> </div></div></div>