[lustre-discuss] [EXTERNAL] inodes distribution

[Mohr, Rick]

Ihsan,

Roughly speaking, every file/dir in lustre will consume one inode on the mdt that hosts it, and each file will also consume one inode on each ost that has a stripe allocated to that file.  The exact inode usage can be complicated with more advanced features like DNE, PFL, etc. but that is a simple estimate on how inodes are used.

Now, how the inode usage is presented is a bit tricky.  In your case, the mdts have 156.7M and 127.7M inodes used for a combined total of 284.4M inodes.  Since inode usage for filesystems is usually an indication of how many files/dirs exist on the filesystem, the sum of the mdt inode usage is reported as the overall filesystem inode usage.  (Because even though a file with stripe_count=4 might consume 1 inode on an mdt and 4 inodes on 4 different osts, it still only counts as 1 file.  So it only adds 1 to the total inode usage and not 5.)

Free inodes are calculated differently.  In the simplest case, a file with stripe_count=1 would consume 1 mdt inode and 1 ost inode.  Since your filesystem has a lot more mdt inodes than ost inodes, lustre assumes that the number of ost inodes is the limiting factor, so it uses the sum of all the free ost inodes as the total number of free inodes remaining.  If you add up 17.9M+16.6M+..., you will get 173.6M which basically matches the number of free inodes.  (The total number of filesystem inodes is then the sum of the used inodes and free inodes.). Of course, the calculation of free inodes can be off depending on the circumstance.  If you use DoM, then it is possible to have a small file that consumes an inode on a mdt but doesn't consume any ost inodes which means your filesystem could accommodate more additional files than the 173.7M indicated by the number of free inodes.  On the other hand, if you created files with stripe_count=10, you would only be able to create about 16.4M files.  Since the total inode usage on your mdts is 284.4M, but the total inode usage on all your osts is around 125M, I'm guessing maybe you are using DoM for a bunch of small files.

The above explanation assumes you are using ldiskfs for the backend which formats the mdts and osts with a fixed number of inodes.  If you are using zfs for the backend, then I think the inode values for each mdt/ost are merely estimates anyway since zfs doesn't have fixed inodes like ldiskfs does.

Hope that helps.

--Rick


On 11/10/25, 6:30 AM, "lustre-discuss on behalf of Ihsan Ur Rahman via lustre-discuss" <lustre-discuss-bounces at lists.lustre.org> wrote:

Hello lustre folks,

In the Lustre file system who is responsible for giving inodes. As per my understanding, it is MDS/MGS who is giving inodes. 
Below is the output of the inodes distribution in our lustre file system. Is this correct? Because the ost is also giving the inodes and most of the used more than 40%.

lfs df -ih /mnt/lust-das
UUID Inodes IUsed IFree IUse% Mounted on
lust-das-MDT0000_UUID 745.2M 156.7M 588.5M 22% /mnt/lust-das[MDT:0]
lust-das-MDT0001_UUID 745.2M 127.7M 617.6M 18% /mnt/lust-das[MDT:1]
lust-das-OST0000_UUID 30.4M 12.5M 17.9M 42% /mnt/lust-das[OST:0]
lust-das-OST0001_UUID 29.1M 12.5M 16.6M 43% /mnt/lust-das[OST:1]
lust-das-OST0002_UUID 30.2M 12.5M 17.7M 42% /mnt/lust-das[OST:2]
lust-das-OST0003_UUID 30.7M 12.5M 18.2M 41% /mnt/lust-das[OST:3]
lust-das-OST0004_UUID 29.7M 12.5M 17.3M 42% /mnt/lust-das[OST:4]
lust-das-OST0005_UUID 29.8M 12.5M 17.3M 42% /mnt/lust-das[OST:5]
lust-das-OST0006_UUID 29.9M 12.5M 17.4M 42% /mnt/lust-das[OST:6]
lust-das-OST0007_UUID 29.8M 12.5M 17.3M 42% /mnt/lust-das[OST:7]
lust-das-OST0008_UUID 28.8M 12.5M 16.4M 44% /mnt/lust-das[OST:8]
lust-das-OST0009_UUID 30.0M 12.5M 17.5M 42% /mnt/lust-das[OST:9]

filesystem_summary: 458.1M 284.4M 173.7M 63% /mnt/lust-das

Regards,
Ihsan