<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Oct 3, 2019, at 05:03, Hebenstreit, Michael <<a href="mailto:michael.hebenstreit@intel.com" class="">michael.hebenstreit@intel.com</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
So you are saying on a zfs based Lustre there is no way to increase the number of available inodes? I have 8TB MDT with roughly 17G inodes<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[root@elfsa1m1 ~]# df -h<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">Filesystem Size Used Avail Use% Mounted on<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">mdt0000 8.3T 256K 8.3T 1% /mdt0000<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[root@elfsa1m1 ~]# df -i<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">Filesystem Inodes IUsed IFree IUse% Mounted on<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">mdt0000 17678817874 6 17678817868 1% /mdt0000</span></div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<span style="font-size: 14px;" class="">For ZFS the only way to increase inodes on the *MDT* is to increase the size of the MDT, though more on that below. Note that the "number of inodes" reported by ZFS is an estimate based on the currently-allocated blocks
and inodes (i.e. bytes_per_inode_ratio = bytes_used / inodes_used, total inode estimate = bytes_free / inode_ratio + inodes_used), which becomes more accurate as the MDT becomes more full. With 17B inodes on a 8TB MDT that is an bytes-per-inode ratio of 497,
which is unrealistically low for Lustre since the MDT will always stores multiple xattrs on each inode. Note that the filesystem only has 6 inodes allocated, so the ZFS total inodes estimate is unrealistically high and will get better as more inodes are allocated
in the filesystem.</span></div>
<div>
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br class="">
</div>
</div>
</div>
<blockquote type="cite" class="">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Formating under Lustre 2.10.8<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">mkfs.lustre --mdt --backfstype=zfs --fsname=lfsarc01 --index=0 --mgsnid="36.101.92.22@tcp" --reformat mdt0000/mdt0000<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
this translates to only 948M inodes on the Lustre FS.<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[root@elfsa1m1 ~]# df -i<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">Filesystem Inodes IUsed IFree IUse% Mounted on<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">mdt0000 17678817874 6 17678817868 1% /mdt0000<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">mdt0000/mdt0000 948016092 263 948015829 1% /lfs/lfsarc01/mdt<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[root@elfsa1m1 ~]# df -h<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">Filesystem Size Used Avail Use% Mounted on<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">mdt0000 8.3T 256K 8.3T 1% /mdt0000<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">mdt0000/mdt0000 8.2T 24M 8.2T 1% /lfs/lfsarc01/mdt<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
and there is no reasonable option to provide more file entries except for adding another MDT?</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div><span style="font-size: 14px;" class="">The Lustre statfs code will weight in some initial estimates for the bytes-per-inode ratio when computing the total inode estimate for the filesystem. When the filesystem is nearly empty, as is the case here, then
those initial estimates will dominate, but once you've allocated a few thousand inodes in the filesystem the actual values will dominate and you will have a much more accurate number for the total inode count. This will probably be more in the range of 2B-4B
inodes in the end, unless you also use Data-on-MDT (Lustre 2.11 and later) to store small files directly on the MDT.</span></div>
<div><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div><span style="font-size: 14px;" class="">You've also excluded the OST lines from the above output? For the Lustre filesystem you (typically) also need at least one OST inode (object) for each file in the filesystem, possibly more than one, so "df" of the
Lustre filesystem may also be limited by the number of inodes reported by the OSTs (which may themselves depend on the average bytes-per-inode for files stored on the OST). </span><span style="font-size: 14px;" class="">If you use Data-on-MDT and only have
a small files, then no OST object is needed for small files, but you consume correspondingly more space on the MDT.</span></div>
<div><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div><span style="font-size: 14px;" class="">Cheers, Andreas</span></div>
<blockquote type="cite" class="">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</blockquote>
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span></div>
</div>
</div>
<blockquote type="cite" class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div class="">
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<a name="_____replyseparator" class=""></a><b class="">From:</b><span class="Apple-converted-space"> </span>Andreas Dilger <<a href="mailto:adilger@whamcloud.com" class="">adilger@whamcloud.com</a>><span class="Apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, October 02, 2019 18:49<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>Hebenstreit, Michael <<a href="mailto:michael.hebenstreit@intel.com" class="">michael.hebenstreit@intel.com</a>><br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span>Mohr Jr, Richard Frank <<a href="mailto:rmohr@utk.edu" class="">rmohr@utk.edu</a>>;
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [lustre-discuss] changing inode size on MDT<o:p class=""></o:p></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
There are several confusing/misleading comments on this thread that need to be clarified...<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Oct 2, 2019, at 13:45, Hebenstreit, Michael <<a href="mailto:michael.hebenstreit@intel.com" style="color: purple; text-decoration: underline;" class="">michael.hebenstreit@intel.com</a>> wrote:<o:p class=""></o:p></div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<a href="http://wiki.lustre.org/Lustre_Tuning#Number_of_Inodes_for_MDS" style="color: purple; text-decoration: underline;" class="">http://wiki.lustre.org/Lustre_Tuning#Number_of_Inodes_for_MDS</a><o:p class=""></o:p></div>
</div>
</div>
</blockquote>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Note that I've updated this page to reflect current defaults. The Lustre Operations Manual has a much better description of these parameters.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
</div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
and I'd like to use --mkfsoptions='-i 1024' to have more inodes in the MDT. We already run out of inodes on that FS (probably due to an ZFS bug in early IEEL version) - so I'd like to increase #inodes if possible.<o:p class=""></o:p></div>
</div>
</div>
</blockquote>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
The "-i 1024" option (bytes-per-inode ratio) is only needed for ldiskfs since it statically allocates the inodes at mkfs time, it is not relevant for ZFS since ZFS dynamically allocates inodes and blocks as needed.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Oct 2, 2019, at 14:00, Colin Faber <<a href="mailto:cfaber@gmail.com" style="color: purple; text-decoration: underline;" class="">cfaber@gmail.com</a>> wrote:<o:p class=""></o:p></div>
</div>
</div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
With 1K inodes you won't have space to accommodate new features, IIRC the current minimal limit on modern lustre is 2K now. If you're running out of MDT space you might consider DNE and multiple MDT's to accommodate that larger name space.<o:p class=""></o:p></div>
</div>
</div>
</div>
</blockquote>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
To clarify, since Lustre 2.10 any new ldiskfs MDT will allocate 1024 bytes for the inode itself (-I 1024). That allows enough space *within* the inode to efficiently store xattrs for more complex layouts (PFL, FLR, DoM). If xattrs do not fit inside the inode
itself then they will be stored in an external 4KB inode block.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
The MDT is formatted with a bytes-per-inode *ratio* of 2.5KB, which means (approximately) one inode will be created for every 2.5kB of the total MDT size. That 2.5KB of space includes the 1KB for the inode itself, plus space for a directory entry (or multiple
if hard-linked), extra xattrs, the journal (up to 4GB for large MDTs), Lustre recovery logs, ChangeLogs, etc. Each directory inode will have at least one 4KB block allocated.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
So, it is _possible_ to reduce the inode *ratio* below 2.5KB if you know what you are doing (e.g. 2KB/inode or 1.5KB/inode, this can be an arbitrary number of bytes, it doesn't have to be an even multiple of anything) but it definitely isn't possible to have
1KB inode size and 1KB per inode ratio, as there wouldn't be *any* space left for directories, log files, journal, etc.<o:p class=""></o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Cheers, Andreas<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">--<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Andreas Dilger<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Principal Lustre Architect<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Whamcloud</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>