<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
The t0 filesystem OSTs are formatted for an average file size of 70TB / 300M inodes = 240KB/inode.
<div><br>
</div>
<div>The t1 filesystem OSTs are formatted for an average file size of 500TB / 65M inodes = 7.7MB/inode. </div>
<div><br>
</div>
<div>So not only are the t1 OSTs larger, but they have fewer inodes (by a factor of 32x). This must have been done with specific formatting options since the default inode ratio is 1MiB/inode for the OSTs.</div>
<div><br>
</div>
<div>There isn't any information about the actual space usage (eg. "lfs df"), so I can't calculate whether the default 1MiB/inode would be appropriate for your filesystem, but definitely it was formatted with the expectation that the average file size would
become larger as they were copied to t1 (eg. combined in a tarfile or something).</div>
<div><br>
</div>
<div>Unfortunately, there is no way to "fix" this in place, since the inode ratio for ldiskfs/ext4 filesystems is decided at format time. </div>
<div><br>
</div>
<div>One option is to use "lfs find" to find files on an OST (eg. OST0003 which is the least used), disable creates on that OST, and use "lfs migrate" to migrate all of the files to other OSTs, and then reformat the OST with more inodes and repeat this process
for all of the OSTs. </div>
<div><br>
</div>
<div>Unfortunately, the t1 filesystem only has 8.5M free inodes and there are 27M inodes on OST0003, so it couldn't be drained completely to perform this process. You would need to delete enough files from t1 to free up inodes to do the migration, or eg. tar
them up into larger files to reduce the inode count. </div>
<div><br>
</div>
<div>The OST migration/replacement process is described in the Lustre Operations Manual. </div>
<div><br>
<div dir="ltr">Cheers, Andreas</div>
<div dir="ltr"><br>
<blockquote type="cite">On Aug 11, 2023, at 01:17, Carlos Adean via lustre-discuss <lustre-discuss@lists.lustre.org> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">\ufeff
<div dir="ltr">
<div>Hello experts,<br>
</div>
<div><br>
</div>
<div>We have a Lustre with two tiers T0(SSD) and T1(HDD), the first with 70TB and the second one with ~500TB.</div>
<div><br>
</div>
<div>I'm experiencing a problem that the T1 has much less inodes than T0 and that is getting without inodes in the OSTs, so I'd like to understand the source of this and how to fix that.</div>
<div><br>
</div>
<div><br>
</div>
<div>Thanks in advance.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><span style="font-family:times new roman,serif">=== T0</span></div>
<div><br>
</div>
<div>$ <span style="font-family:times new roman,serif">lfs df -i /lustre/t0<br>
UUID Inodes IUsed IFree IUse% Mounted on<br>
t0-MDT0000_UUID 390627328 1499300 389128028 1% /lustre/t0[MDT:0]<br>
t0-OST0000_UUID 14651392 1097442 13553950 8% /lustre/t0[OST:0]<br>
t0-OST0001_UUID 14651392 1097492 13553900 8% /lustre/t0[OST:1]<br>
t0-OST0002_UUID 14651392 1097331 13554061 8% /lustre/t0[OST:2]<br>
t0-OST0003_UUID 14651392 1097563 13553829 8% /lustre/t0[OST:3]<br>
t0-OST0004_UUID 14651392 1097576 13553816 8% /lustre/t0[OST:4]<br>
t0-OST0005_UUID 14651392 1097505 13553887 8% /lustre/t0[OST:5]<br>
t0-OST0006_UUID 14651392 1097524 13553868 8% /lustre/t0[OST:6]<br>
t0-OST0007_UUID 14651392 1097596 13553796 8% /lustre/t0[OST:7]<br>
t0-OST0008_UUID 14651392 1097442 13553950 8% /lustre/t0[OST:8]<br>
t0-OST0009_UUID 14651392 1097563 13553829 8% /lustre/t0[OST:9]<br>
t0-OST000a_UUID 14651392 1097515 13553877 8% /lustre/t0[OST:10]<br>
t0-OST000b_UUID 14651392 1096524 13554868 8% /lustre/t0[OST:11]<br>
t0-OST000c_UUID 14651392 1096608 13554784 8% /lustre/t0[OST:12]<br>
t0-OST000d_UUID 14651392 1096524 13554868 8% /lustre/t0[OST:13]<br>
t0-OST000e_UUID 14651392 1096641 13554751 8% /lustre/t0[OST:14]<br>
t0-OST000f_UUID 14651392 1096647 13554745 8% /lustre/t0[OST:15]<br>
t0-OST0010_UUID 14651392 1096705 13554687 8% /lustre/t0[OST:16]<br>
t0-OST0011_UUID 14651392 1096616 13554776 8% /lustre/t0[OST:17]<br>
t0-OST0012_UUID 14651392 1096520 13554872 8% /lustre/t0[OST:18]<br>
t0-OST0013_UUID 14651392 1096598 13554794 8% /lustre/t0[OST:19]<br>
t0-OST0014_UUID 14651392 1096669 13554723 8% /lustre/t0[OST:20]<br>
t0-OST0015_UUID 14651392 1096570 13554822 8% /lustre/t0[OST:21]<br>
<br>
filesystem_summary: 299694753 1499300 298195453 1% /lustre/t0</span></div>
<div><span style="font-family:times new roman,serif"><br>
</span></div>
<div><span style="font-family:times new roman,serif"><br>
</span></div>
<div><span style="font-family:times new roman,serif">=== T1</span></div>
<div>
<pre class="gmail-wiki">$ lfs df -i /lustre/t1<br>UUID Inodes IUsed IFree IUse% Mounted on<br>t1-MDT0000_UUID 1478721536 56448788 1422272748 4% /lustre/t1[MDT:0]<br>t1-OST0000_UUID 30492032 30491899 133 100% /lustre/t1[OST:0]<br>t1-OST0001_UUID 30492032 30491990 42 100% /lustre/t1[OST:1]<br>t1-OST0002_UUID 30492032 30491916 116 100% /lustre/t1[OST:2]<br>t1-OST0003_UUID 30492032 27471050 3020982 91% /lustre/t1[OST:3]<br>t1-OST0004_UUID 30492032 30491989 43 100% /lustre/t1[OST:4]<br>t1-OST0005_UUID 30492032 30491960 72 100% /lustre/t1[OST:5]<br>t1-OST0006_UUID 30492032 30491948 84 100% /lustre/t1[OST:6]<br>t1-OST0007_UUID 30492032 30491939 93 100% /lustre/t1[OST:7]<br>t1-OST0008_UUID 30492032 29811803 680229 98% /lustre/t1[OST:8]<br>t1-OST0009_UUID 30492032 29808261 683771 98% /lustre/t1[OST:9]<br>t1-OST000a_UUID 30492032 29809919 682113 98% /lustre/t1[OST:10]<br>t1-OST000b_UUID 30492032 29807585 684447 98% /lustre/t1[OST:11]<br>t1-OST000c_UUID 30492032 29809171 682861 98% /lustre/t1[OST:12]<br>t1-OST000d_UUID 30492032 29804206 687826 98% /lustre/t1[OST:13]<br>t1-OST000e_UUID 30492032 29806399 685633 98% /lustre/t1[OST:14]<br>t1-OST000f_UUID 30492032 29802857 689175 98% /lustre/t1[OST:15]<br><br>filesystem_summary: 64946408 56448788 8497620 87% /lustre/t1<br></pre>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div></div>
<div>--<br>
</div>
<div>Carlos Adean</div>
<div></div>
<div><a href="http://www.linea.org.br" target="_blank">www.linea.org.br</a></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
<br>
<p style="color:rgb(0,0,0);font-family:"segoe ui","lucida sans",sans-serif;background-color:rgb(253,253,253)">
<span style="font-weight:bold;line-height:13.6px;font-family:arial,sans-serif;color:rgb(123,127,127)"><a href="https://www.linea.org.br/" rel="nofollow%20nofollow%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer nofollow noopener noreferrer" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:x-small;font-family:verdana,sans-serif">\u200b</span><font size="2">Site</font></a><font size="2"> | <a href="https://twitter.com/LIneA_org" rel="nofollow%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer nofollow noopener noreferrer" style="color:rgb(17,85,204)" target="_blank">Twitter</a> | <a href="https://www.instagram.com/linea_org/" rel="nofollow%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer nofollow noopener noreferrer" style="color:rgb(17,85,204)" target="_blank">Instagram</a> | <a href="https://www.facebook.com/linea.org" rel="nofollow%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20nofollow%20noopener%20noreferrer nofollow noopener noreferrer" style="color:rgb(17,85,204)" target="_blank"><span></span><wbr>Facebook</a> </font></span><span style="line-height:12.8px"><font size="2"><br>
<font color="#233a68" face="arial"><b>Laboratório Interinstitucional de e-Astronomia </b></font></font></span></p>
<span>_______________________________________________</span><br>
<span>lustre-discuss mailing list</span><br>
<span>lustre-discuss@lists.lustre.org</span><br>
<span>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</span><br>
</div>
</blockquote>
</div>
</body>
</html>