<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Please note the differences between inodes on the ZFS and inodes on the mdt lustre. In the previous incarnation the file system run out of ionodes as reported on the Lustre, even though the mdt was only half filled and zfs backend still
reported free inodes.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"></a><b>From:</b> Shaun Tancheff <stancheff@cray.com>
<br>
<b>Sent:</b> Thursday, October 03, 2019 05:41<br>
<b>To:</b> Degremont, Aurelien <degremoa@amazon.com>; Hebenstreit, Michael <michael.hebenstreit@intel.com>; Andreas Dilger <adilger@whamcloud.com><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] changing inode size on MDT<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">A little pedantic but for ‘inodes’ don’t exist in a zfs pool per-se. So the code which attempts to report the number of inodes used/available guesses based on the average per-object utilization rate. If you have a many large files your
reported number of inodes goes down faster than if you have a many small files.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I think the primary take away is that for a zfs backend you will run out of _inodes_ at the same time as you run out of _<i>space</i>_, there simply is no distinction.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of
"Degremont, Aurelien" <<a href="mailto:degremoa@amazon.com">degremoa@amazon.com</a>><br>
<b>Date: </b>Thursday, October 3, 2019 at 6:11 PM<br>
<b>To: </b>"Hebenstreit, Michael" <<a href="mailto:michael.hebenstreit@intel.com">michael.hebenstreit@intel.com</a>>, Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>><br>
<b>Cc: </b>"<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>><br>
<b>Subject: </b>Re: [lustre-discuss] changing inode size on MDT<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">As Andreas said "<span style="color:black">it is not relevant for ZFS since ZFS dynamically allocates inodes and blocks as needed"</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">"as needed" is the important part. In your example, your MDT is almost empty, so 17G inodes for an empty MDT seems pretty sufficient.<o:p></o:p></p>
<p class="MsoNormal">As you will create new files and use these inodes, you will see the total number of inodes change.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">But I don't what is the maximum number of inodes you can estimate for a 8.3 TiB MDT…<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span style="font-size:12.0pt;color:black">De :
</span></b><span style="font-size:12.0pt;color:black">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a>> au nom de "Hebenstreit, Michael" <<a href="mailto:michael.hebenstreit@intel.com">michael.hebenstreit@intel.com</a>><br>
<b>Date : </b>jeudi 3 octobre 2019 à 13:04<br>
<b>À : </b>Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>><br>
<b>Cc : </b>"<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>><br>
<b>Objet : </b>Re: [lustre-discuss] changing inode size on MDT</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">So you are saying on a zfs based Lustre there is no way to increase the number of available inodes? I have 8TB MDT with roughly 17G inodes<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">[root@elfsa1m1 ~]# df -h</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">Filesystem Size Used Avail Use% Mounted on</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">mdt0000 8.3T 256K 8.3T 1% /mdt0000</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">[root@elfsa1m1 ~]# df -i</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">Filesystem Inodes IUsed IFree IUse% Mounted on</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">mdt0000 17678817874 6 17678817868 1% /mdt0000</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt">Formating under Lustre 2.10.8 <o:p>
</o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">mkfs.lustre --mdt --backfstype=zfs --fsname=lfsarc01 --index=0 --mgsnid="<a href="mailto:36.101.92.22@tcp">36.101.92.22@tcp</a>" --reformat mdt0000/mdt0000</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt">this translates to only 948M inodes on the Lustre FS.
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">[root@elfsa1m1 ~]# df -i</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">Filesystem Inodes IUsed IFree IUse% Mounted on</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">mdt0000 17678817874 6 17678817868 1% /mdt0000</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">mdt0000/mdt0000 948016092 263 948015829 1% /lfs/lfsarc01/mdt</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">[root@elfsa1m1 ~]# df -h</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">Filesystem Size Used Avail Use% Mounted on</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">mdt0000 8.3T 256K 8.3T 1% /mdt0000</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New"">mdt0000/mdt0000 8.2T 24M 8.2T 1% /lfs/lfsarc01/mdt</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt">and there is no reasonable option to provide more file entries except for adding another MDT?<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt">Thanks<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt">Michael<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:35.4pt"><b>From:</b> Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>>
<br>
<b>Sent:</b> Wednesday, October 02, 2019 18:49<br>
<b>To:</b> Hebenstreit, Michael <<a href="mailto:michael.hebenstreit@intel.com">michael.hebenstreit@intel.com</a>><br>
<b>Cc:</b> Mohr Jr, Richard Frank <<a href="mailto:rmohr@utk.edu">rmohr@utk.edu</a>>;
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> Re: [lustre-discuss] changing inode size on MDT<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">There are several confusing/misleading comments on this thread that need to be clarified...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">On Oct 2, 2019, at 13:45, Hebenstreit, Michael <<a href="mailto:michael.hebenstreit@intel.com">michael.hebenstreit@intel.com</a>> wrote:<o:p></o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><a href="http://wiki.lustre.org/Lustre_Tuning#Number_of_Inodes_for_MDS">http://wiki.lustre.org/Lustre_Tuning#Number_of_Inodes_for_MDS</a><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">Note that I've updated this page to reflect current defaults. The Lustre Operations Manual has a much better description of these parameters.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">and I'd like to use --mkfsoptions='-i 1024' to have more inodes in the MDT. We already run out of inodes on that FS (probably due to an ZFS bug in early IEEL version) - so I'd like to increase #inodes if possible.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">The "-i 1024" option (bytes-per-inode ratio) is only needed for ldiskfs since it statically allocates the inodes at mkfs time, it is not relevant for ZFS since ZFS dynamically allocates inodes and blocks as needed.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Oct 2, 2019, at 14:00, Colin Faber <<a href="mailto:cfaber@gmail.com">cfaber@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">With 1K inodes you won't have space to accommodate new features, IIRC the current minimal limit on modern lustre is 2K now. If you're running out of MDT space you might consider DNE and multiple MDT's to accommodate
that larger name space.<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">To clarify, since Lustre 2.10 any new ldiskfs MDT will allocate 1024 bytes for the inode itself (-I 1024). That allows enough space *within* the inode to efficiently store xattrs for more complex layouts (PFL,
FLR, DoM). If xattrs do not fit inside the inode itself then they will be stored in an external 4KB inode block.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">The MDT is formatted with a bytes-per-inode *ratio* of 2.5KB, which means (approximately) one inode will be created for every 2.5kB of the total MDT size. That 2.5KB of space includes the 1KB for the inode itself,
plus space for a directory entry (or multiple if hard-linked), extra xattrs, the journal (up to 4GB for large MDTs), Lustre recovery logs, ChangeLogs, etc. Each directory inode will have at least one 4KB block allocated.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">So, it is _possible_ to reduce the inode *ratio* below 2.5KB if you know what you are doing (e.g. 2KB/inode or 1.5KB/inode, this can be an arbitrary number of bytes, it doesn't have to be an even multiple of anything)
but it definitely isn't possible to have 1KB inode size and 1KB per inode ratio, as there wouldn't be *any* space left for directories, log files, journal, etc.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black">Cheers, Andreas</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black">--</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black">Andreas Dilger</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black">Principal Lustre Architect</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black">Whamcloud</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:35.4pt">
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
</div>
</body>
</html>