[lustre-discuss] [EXTERNAL] getting without inodes

Tue Aug 15 11:06:31 PDT 2023

Carlos,

One thing you should look at is the stripe counts for your files.  For T1, the mdt has used about 56M inodes and each of the 16 osts has used about 30M inodes.  This means that the average stripe count for a file is around 8.  If there are some very small files, it's possible that those files have inodes allocated on osts for objects that don’t contain any data.  I don't know what the space usage looks like for your osts, but if there is room, you could try using "lfs migrate" to restripe files to stripe count 1.  If you were able to do this for all files on T1, then each ost would only be using about 3M to 4M inodes instead of 30M.  I don't know how T1 is being used, what the file size distribution looks like, etc. so you will have to decide if that is a viable option.  T0 looks like it might be on a similar trajectory as T1.  There are about 1.5M inodes used on the mdt and about 1M used on each of the 22 osts.  This means the average stripe count for a file looks to be around 14-15.  At that rate, there could be a similar imbalance in inode usage between the mdt and osts at some point.

Do your users customize the stripe counts for their files?  Or are they like most users and just use whatever default stripe count is set on the file system?  You might need to do some digging to see if files are being striped appropriately based on their size.  If you find lots of small files with large stripe counts, you could consider changing the default stripe count or maybe look into using PFL to define a default file layout that increases in stripe count as the file size increases.

--Rick

On 8/10/23, 6:17 PM, "lustre-discuss on behalf of Carlos Adean via lustre-discuss" <lustre-discuss-bounces at lists.lustre.org <mailto:lustre-discuss-bounces at lists.lustre.org> on behalf of lustre-discuss at lists.lustre.org <mailto:lustre-discuss at lists.lustre.org>> wrote:

Hello experts,

We have a Lustre with two tiers T0(SSD) and T1(HDD), the first with 70TB and the second one with ~500TB.
I'm experiencing a problem that the T1 has much less inodes than T0 and that is getting without inodes in the OSTs, so I'd like to understand the source of this and how to fix that.

Thanks in advance.

=== T0

$ lfs df -i /lustre/t0
UUID Inodes IUsed IFree IUse% Mounted on
t0-MDT0000_UUID 390627328 1499300 389128028 1% /lustre/t0[MDT:0]
t0-OST0000_UUID 14651392 1097442 13553950 8% /lustre/t0[OST:0]
t0-OST0001_UUID 14651392 1097492 13553900 8% /lustre/t0[OST:1]
t0-OST0002_UUID 14651392 1097331 13554061 8% /lustre/t0[OST:2]
t0-OST0003_UUID 14651392 1097563 13553829 8% /lustre/t0[OST:3]
t0-OST0004_UUID 14651392 1097576 13553816 8% /lustre/t0[OST:4]
t0-OST0005_UUID 14651392 1097505 13553887 8% /lustre/t0[OST:5]
t0-OST0006_UUID 14651392 1097524 13553868 8% /lustre/t0[OST:6]
t0-OST0007_UUID 14651392 1097596 13553796 8% /lustre/t0[OST:7]
t0-OST0008_UUID 14651392 1097442 13553950 8% /lustre/t0[OST:8]
t0-OST0009_UUID 14651392 1097563 13553829 8% /lustre/t0[OST:9]
t0-OST000a_UUID 14651392 1097515 13553877 8% /lustre/t0[OST:10]
t0-OST000b_UUID 14651392 1096524 13554868 8% /lustre/t0[OST:11]
t0-OST000c_UUID 14651392 1096608 13554784 8% /lustre/t0[OST:12]
t0-OST000d_UUID 14651392 1096524 13554868 8% /lustre/t0[OST:13]
t0-OST000e_UUID 14651392 1096641 13554751 8% /lustre/t0[OST:14]
t0-OST000f_UUID 14651392 1096647 13554745 8% /lustre/t0[OST:15]
t0-OST0010_UUID 14651392 1096705 13554687 8% /lustre/t0[OST:16]
t0-OST0011_UUID 14651392 1096616 13554776 8% /lustre/t0[OST:17]
t0-OST0012_UUID 14651392 1096520 13554872 8% /lustre/t0[OST:18]
t0-OST0013_UUID 14651392 1096598 13554794 8% /lustre/t0[OST:19]
t0-OST0014_UUID 14651392 1096669 13554723 8% /lustre/t0[OST:20]
t0-OST0015_UUID 14651392 1096570 13554822 8% /lustre/t0[OST:21]

filesystem_summary: 299694753 1499300 298195453 1% /lustre/t0

=== T1
$ lfs df -i /lustre/t1
UUID Inodes IUsed IFree IUse% Mounted on
t1-MDT0000_UUID 1478721536 56448788 1422272748 4% /lustre/t1[MDT:0]
t1-OST0000_UUID 30492032 30491899 133 100% /lustre/t1[OST:0]
t1-OST0001_UUID 30492032 30491990 42 100% /lustre/t1[OST:1]
t1-OST0002_UUID 30492032 30491916 116 100% /lustre/t1[OST:2]
t1-OST0003_UUID 30492032 27471050 3020982 91% /lustre/t1[OST:3]
t1-OST0004_UUID 30492032 30491989 43 100% /lustre/t1[OST:4]
t1-OST0005_UUID 30492032 30491960 72 100% /lustre/t1[OST:5]
t1-OST0006_UUID 30492032 30491948 84 100% /lustre/t1[OST:6]
t1-OST0007_UUID 30492032 30491939 93 100% /lustre/t1[OST:7]
t1-OST0008_UUID 30492032 29811803 680229 98% /lustre/t1[OST:8]
t1-OST0009_UUID 30492032 29808261 683771 98% /lustre/t1[OST:9]
t1-OST000a_UUID 30492032 29809919 682113 98% /lustre/t1[OST:10]
t1-OST000b_UUID 30492032 29807585 684447 98% /lustre/t1[OST:11]
t1-OST000c_UUID 30492032 29809171 682861 98% /lustre/t1[OST:12]
t1-OST000d_UUID 30492032 29804206 687826 98% /lustre/t1[OST:13]
t1-OST000e_UUID 30492032 29806399 685633 98% /lustre/t1[OST:14]
t1-OST000f_UUID 30492032 29802857 689175 98% /lustre/t1[OST:15]

filesystem_summary: 64946408 56448788 8497620 87% /lustre/t1