<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
I just found this old email in my spam folder...<br class="">
<div><br class="">
</div>
<div>On Apr 21, 2020, at 14:54, Pinkesh Valdria <<a href="mailto:pinkesh.valdria@oracle.com" class="">pinkesh.valdria@oracle.com</a>> wrote:<br class="">
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Does lustre have mount options to mimic NFS mount option behavior , listed below?<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I know in most cases, Lustre would perform much better than NFS and can scale and support a lot of clients in parallel. I have a use case, where there are only few clients accessing the filesystem and the files are really small, but in millions and files
are very infrequently updated. The files are stored on an NFS server and its mounted on the clients with the below mount options, which results in caching of file attributes/metadata on the client and thus reduces # of calls to metadata and delivers better
performance. <span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><u class=""><o:p class=""><span style="text-decoration: none;" class=""> </span></o:p></u></b></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><u class="">NFS mount options<o:p class=""></o:p></u></b></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
type nfs (rw,<span style="color: red;" class="">nolock,nocto,actimeo=900</span>,nfsvers=3,proto=tcp)</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
Lustre will already cache file attributes and data on the client, since it is totally coherent, and doesn't depend on random timeouts like NFS to decide whether the client should cache the data or not.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
A custom proprietary application which compile (make command) on some of these files takes 20-24 seconds to run. The same command when ran on the same files stored in BeeGFS parallel filesystem takes 80-90 seconds (4x times slow), mainly because there is
no client caching in BeeGFS and client has to make a lot more metadata calls compared to NFS cache file attributes.</div>
</div>
</blockquote>
<blockquote type="cite" class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><u class="">Question<o:p class=""></o:p></u></b></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I already tried BeeGFS and I am asking this question to determine, if Lustre performance would be better than NFS for very small file workloads (50 bytes, 200 bytes, 2KB files) with 5 millions files spread across nested directories. Does lustre have mount
options to mimic NFS mount option behavior, listed below? Or is there some optional feature in Lustre to achieve this cache behavior?</div>
</div>
</blockquote>
<div><br class="">
</div>
Yes, Lustre will already/always have the desired caching behavior by default, no settings needed. Some tuning might be needed if the working set is so large (10,000s of files) that the locks protecting the data are cancelled because of the sheer volume of
data or because the files are unused for a long time (i.e. over 1h).</div>
<div><br class="">
</div>
<div>Since Lustre can be downloaded for free, you could always give your application workload a test, to see what the performance is.</div>
<div><br class="">
</div>
<div>For very small files, you might want to consider to use Data-on-MDT (DoM) by running "lfs setstripe -E 64k -L mdt -E64M -c 1 -Eeof -c -1 $dir" on the test directory (or on the root directory of the filesystem) to have it store these tiny files directly
on the MDT. You would in that case need enough free space on the MDT to hold all of the files.</div>
<div><br class="">
</div>
<div>Cheers, Andreas</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<a href="https://linux.die.net/man/5/nfs" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">https://linux.die.net/man/5/nfs</a><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-family: Verdana, sans-serif; color: rgb(68, 68, 68); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">ac</span></b><span style="font-family: Verdana, sans-serif; color: rgb(68, 68, 68); background-color: white; background-position: initial initial; background-repeat: initial initial;" class=""> / <b class="">noac</b></span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: Verdana, sans-serif; color: rgb(68, 68, 68); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">Selects whether the client may cache file attributes. If neither option is specified
(or if <b class="">ac</b> is specified), the client caches file attributes.</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: red;" class="">For my custom applications, cache of file attributes is fine (no negative impact) and it helps to improve performance of NFS. <span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<p style="background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">
<b class=""><span style="font-size: 12pt; font-family: Verdana, sans-serif; color: rgb(68, 68, 68);" class="">actimeo=</span></b><i class=""><span style="font-size: 12pt; font-family: Verdana, sans-serif; color: rgb(68, 68, 68);" class="">n</span></i><span style="font-size: 12pt; font-family: Verdana, sans-serif; color: rgb(68, 68, 68);" class=""><o:p class=""></o:p></span></p>
<p style="background-color: white; font-variant-ligatures: normal; font-variant-caps: normal; orphans: 2; text-align: start; widows: 2; -webkit-text-stroke-width: 0px; word-spacing: 0px; background-position: initial initial; background-repeat: initial initial;" class="">
<span style="font-size: 12pt; font-family: Verdana, sans-serif; color: rgb(68, 68, 68);" class="">Using <b class="">actimeo</b> sets all of <b class="">acregmin</b>, <b class="">acregmax</b>, <b class="">acdirmin</b>, and <b class="">acdirmax</b> to the same
value. If this option is not specified, the NFS client uses the defaults for each of these options listed above.<o:p class=""></o:p></span></p>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: red;" class="">For my applications, it’s okay to use cache file attributes/metadata for few mins (eg: 5mins) by setting this value, it can reduce # of metadata calls been made to the server and especially with filesystems storing lot of
small files, it’s a huge performance penalty, which can be avoided. <span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: red;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-family: Verdana, sans-serif; color: rgb(68, 68, 68); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">nolock<o:p class=""></o:p></span></b></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-family: Verdana, sans-serif; color: rgb(68, 68, 68); background-color: white; background-position: initial initial; background-repeat: initial initial;" class=""><o:p class=""> </o:p></span></b></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt; font-family: Verdana, sans-serif; color: rgb(68, 68, 68); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">When mounting servers that do not support the NLM protocol,
or when mounting an NFS server through a firewall that blocks the NLM service port, specify the <b class="">nolock</b> mount option.</span><span style="font-size: 12pt; font-family: Verdana, sans-serif; color: rgb(68, 68, 68); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">Specifying
the <b class="">nolock</b> option may also be advised to improve the performance of a proprietary application which runs on a single client and uses file locks extensively.</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Appreciate any guidance. <span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Thanks,<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
pinkesh valdria<o:p class=""></o:p></div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">lustre-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="mailto:lustre-discuss@lists.lustre.org" style="color: rgb(5, 99, 193); text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">lustre-discuss@lists.lustre.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="color: rgb(5, 99, 193); text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>