<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Andreas,</div><div class="gmail_quote"><br></div><div class="gmail_quote">Thanks very much for your comments...</div><div class="gmail_quote"><br></div><div class="gmail_quote">On Wed, May 18, 2016 at 1:30 PM, Dilger, Andreas <span dir="ltr"><<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>
<div>
<div><br></div></div></div>
<span>
<div>
<div>On 2016/05/18, 11:22, "Nathan Dauchy - NOAA Affiliate" <<a href="mailto:nathan.dauchy@noaa.gov" target="_blank">nathan.dauchy@noaa.gov</a>> wrote:</div>
</div>
<blockquote style="BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5"><div><div><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div><div>I'm looking for your experience and perhaps some lively discussion regarding "best practices" for choosing a file stripe count. The Lustre manual has good tips on "Choosing a Stripe Size", and in practice the default 1M rarely causes problems on our systems.
Stripe Count on the other hand is far more difficult to chose a single value that is efficient for a general purpose and multi-use site-wide file system.<br></div>
<div>What do you all recommend as a reasonable rule of thumb that works for "most" user's needs, where stripe count can be determined based only on static data attributes (such as file size)?</div></div></div></div></div></div></div></blockquote></span>
<div>Using the log2() value seems reasonable.</div>
<div><br>
</div>
<span>
<blockquote style="BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5">
<div>
<div>
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">
<div>
<div>Ideally, I would like to have a tool to give the users and say "go restripe your directory with this command" and it will do the right thing in 90% of cases. See the rough patch to lfs_migrate (included below) which should help explain what I'm thinking.
Probably there are more efficient ways of doing things, but I have tested it lightly and it works as a proof-of-concept.</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>I'd welcome this as a patch submitted to Gerrit.</div>
<div><br></div></div></blockquote><div><br></div><div>A Jira ticket has been created:</div><div><a href="https://jira.hpdd.intel.com/browse/LU-8207">https://jira.hpdd.intel.com/browse/LU-8207</a></div><div><br></div><div>The draft patch is there, and probably needs a bit of work before pushing into Gerrit. If anyone wants to tackle that, assistance appreciated of course! :)</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif"><div>
</div>
<span>
<blockquote style="BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">
<div>
<div>With a good programmatic rule of thumb, we (as a Lustre community!) can eventually work with application developers to embed the stripe count selection into their code and get things at least closer to right up front. Even if trial and error is involved
to find the optimal setting, at least the rule of thumb can be a _starting_point_ for the users, and they can tweak it from there based on application, model, scale, dataset, etc.</div>
<div><br>
</div>
<div>Thinking farther down the road, with progressive file layout, what algorithm will be used as the default?</div>
</div>
</div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>To be clear, the PFL implementation does not currently have an algorithmic layout, rather a series of thresholds based on file size that will select different layouts (initially stripe counts, but could be anything including stripe size, OST pools, etc).
The PFL size thresholds and stripe counts _could_ be set up (manually) as as a geometric series, but they can also be totally arbitrary if you want.</div></div></blockquote><div><br></div><div>Understood. However, Lustre will still need to have some sort of default layout. I was thinking that it would be good to match that future code with current best-practice recommendations and whatever ends up in lfs_migrate for auto-striping.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div><br>
</div>
<span>
<blockquote style="BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">
<div>
<div>If Lustre gets to the point where it can rebalance OST capacity behind the scenes, could it also make some intelligent choice about restriping very large files to spread out load and better balance capacity? (Would that mean we need a bit set on the file
to flag whether the stripe info was set specifically by the user or automatically by Lustre tools or it was just using the system default?) Can the filesystem track concurrent access to a file, and perhaps migrate the file and adjust stripe count based on
number of active clients?</div>
</div>
</div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>I think this would be an interesting task for RobinHood, since it already has much of this information. It could find large files with low stripe counts and restripe them during OST rebalancing.</div></div></blockquote><div><br></div><div>Yes, the need to rebalance OSTs when adding new ones to the file system is in part what prompted this topic. We have only experimented with Robinhood as a low-priority task, but hope to use it more in the future.</div><div><br></div><div>I was picturing that the general rebalance process (without robinhood) would be something like:</div><div><br></div><div>* Identify the most full OSTs with something like:</div><div><span class="" style="white-space:pre"> </span># lfs df $FS | grep OST | sort -k 4 -n | head -n 4</div><div><br></div><div>* Search for singly-striped, large, and inactive files on those OSTs with:</div><div><span class="" style="white-space:pre"> </span># lfs find * -type f -mtime +30 -size +8G -c 1 -O A,B,N,X > filelist</div><div><br></div><div>* Restripe those files with:</div><div><span class="" style="white-space:pre"> </span># lfs_migrate -A -y < filelist</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif"><span><blockquote style="BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div><div><br></div></div></div></div></div></blockquote></span>
<div>One last comment on the patch below:</div>
<div>Instead of involving "bc", which is not guaranteed to be installed, why not just have a simple "divide by 2, increment stripe_count" loop after converting bytes to GiB? That would be a few cycles for huge files, but probably still faster than fork/exec
of an external binary as it could be at most 63 - 30 = 33 loops and usually many fewer.</div>
<div></div></div></blockquote></div><br></div><div class="gmail_extra">Good point. I made a note to that effect in the Jira ticket. In general, I would think that external commands in the "coreutils" package are OK (cut, wc, head, tr, comm) but others (bc, sed, awk, grep) should be avoided.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Cheers,</div><div class="gmail_extra">Nathan</div><div class="gmail_extra"><br></div></div>