<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Jan 8, 2021, at 12:44, Nathan Rutman <<a href="mailto:nrutman@gmail.com" class="">nrutman@gmail.com</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<pre style="white-space: pre-wrap;" class=""><h1 style="white-space:normal" class=""><span style="font-weight:normal" class=""><font size="2" style="" class=""></font></span></h1></pre>
<pre style="white-space:pre-wrap" class=""><font style="background-color:rgb(255,255,255)" face="arial, sans-serif" class="">Riffing on something Andreas said in a lustre-discuss thread, I'm hoping someone can correct my understanding of how precreate works currently.</font></pre>
<pre style="white-space:pre-wrap" class=""><font style="background-color:rgb(255,255,255)" face="arial, sans-serif" class="">Olden days: </font></pre>
<pre style="white-space:pre-wrap" class=""><font style="background-color:rgb(255,255,255)" face="arial, sans-serif" class="">MDS would ask each OST for a set of precreated objects via a MDT->OST RPC. These have to be cleaned up during recovery, hence a cap. These were used up as MDS assigned them to layouts, and so MDS has to go back and get more, even for 0-length files.</font></pre>
<pre style="white-space:pre-wrap" class=""><font style="background-color:rgb(255,255,255)" face="arial, sans-serif" class="">Modern days, Lustre 2.5+:</font></pre>
<pre style="white-space:pre-wrap" class=""><span style="box-sizing:inherit;font-variant-ligatures:common-ligatures;white-space:normal;background-color:rgb(255,255,255)" class=""><font style="" face="arial, sans-serif" class="">MDT doesn't hold a pool of OST objects but instead takes an OST fid range from a FLD server instead. Each MD object has a mapping with an eventual OST object by this fid. The OST side just holds a small number of anonymous objects and assigns the fid to an object when any operation is executed without an existing FID->inode mapping on the OST.</font></span><span class="gmail-c-mrkdwn__br" style="box-sizing:inherit;display:block;height:8px;font-variant-ligatures:common-ligatures;white-space:normal"></span><font style="background-color:rgb(255,255,255)" face="arial, sans-serif" class=""><span style="box-sizing:inherit;font-variant-ligatures:common-ligatures;white-space:normal" class="">There is no more precreate RPC necessary, since OSTs maintain their own pool of anonymous objects and only use them up when data is actually written, and can create more when running low. There is no recovery cleanup needed on the OSTs. </span><br style="box-sizing:inherit;font-variant-ligatures:common-ligatures;white-space:normal" class=""><span style="box-sizing:inherit;font-variant-ligatures:common-ligatures;white-space:normal" class="">In this case, there should be no performance difference between create and mknod except for the FLD operation, and the number of OSTs should not matter for create rates.</span></font></pre>
<pre style="" class=""><font face="arial, sans-serif" class=""><span style="font-variant-ligatures:common-ligatures;white-space:normal" class="">Is my understanding wrong? It clearly must be, since Andreas is still talking OST_CREATE rpc and recovery implications, and we do see a performance difference with mknod and creating files with layouts.<br class=""></span></font></pre>
</div>
</div>
</blockquote>
</div>
<div>The precreate code still works the same as "the land before the time of FIDs". Actual objects are still precreated/destroyed on the OSTs. The only difference is that the FID sequences allocated to MDTs allow the OSTs to have different pools of objects
for each MDT so that they don't contend/conflict when those MDTs assign the objects to their own inodes.</div>
<div><br class="">
</div>
<div>Having multiple MDTs does "scale" the OST object space, in that there can be more object subdirectories (one per sequence), which improves both the concurrency and the maximum number of objects. There has also been work done to increase the maximum number
of files per directory in ldiskfs, but that doesn't really improve performance.</div>
<div><br class="">
</div>
<div>The patch <a href="https://review.whamcloud.com/38424" class="">https://review.whamcloud.com/38424</a> "<a href="https://jira.whamcloud.com/browse/LU-11912" class="">LU-11912</a><span class=""><span class=""> ofd: reduce LUSTRE_DATA_SEQ_MAX_WIDTH</span></span><span class="">"
would create smaller object directory trees, and allow "aging" of old objects to be in separate object directory trees from new objects. That allows old objects to drop out of cache (avoiding one-create-per-leaf as the size of the directory grows very lareg),
and keeps fewer "hot" objects densely packed in memory (allowing many new entries to be packed into a single leaf block).</span></div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>