[lustre-discuss] inodes not adding up
Andreas Dilger
adilger at whamcloud.com
Thu Apr 18 02:52:17 PDT 2019
On Apr 15, 2019, at 12:56, Mohr Jr, Richard Frank (Rick Mohr) <rmohr at utk.edu> wrote:
>
>
>> On Apr 13, 2019, at 4:57 AM, Youssef Eldakar <youssefeldakar at gmail.com> wrote:
>>
>> For one Lustre filesystem, inode count in the summary is notably less than what the individual OST inode counts would add up to:
>
> The first thing to understand is that every Lustre file will consume one inode on the MDT, and this inode uses attributes to store information about which OSTs the file is striped over. Then for each file stripe, there will also be an inode consumed on the corresponding OSTs. For example, a file with stripe_count=4 will consume one inode on the MDT and four inodes on OSTs (one inode on each OST the file is striped over).
>
>> # lfs df -i /lfs01
>> UUID Inodes IUsed IFree IUse% Mounted on
>> lustrefs-MDT0000_UUID 2402287616 46560885 2355726731 2% /share/lfs01[MDT:0]
>> lustrefs-OST0001_UUID 24117248 22883788 1233460 95% /share/lfs01[OST:1]
>> lustrefs-OST0003_UUID 24117248 22903308 1213940 95% /share/lfs01[OST:3]
>> lustrefs-OST0004_UUID 24117248 22895442 1221806 95% /share/lfs01[OST:4]
>> lustrefs-OST0006_UUID 24117248 22890201 1227047 95% /share/lfs01[OST:6]
>>
>> filesystem_summary: 51457138 46560885 4896253 90% /share/lfs01
>
> On this file system, there are already 46,560,885 files which also consume the same number of inodes on the MDT (so IUsed=46560885). However, even though the MDT has over 2 billion inodes free, every file created in the future will use at least one inode on an OST. If you add up all the free inodes on all the OSTs, you get 4896253. So at best, there is only space for 4,896,253 more files. That is why IFree=4896253. Then, Inodes = IUsed + IFree = 46,560,885 + 4,896,253 = 51,457,138.
>
>> On another filesystem, this is not the case:
>>
>> # lfs df -i /lfs02
>> UUID Inodes IUsed IFree IUse% Mounted on
>> lustrefs-MDT0000_UUID 1288503296 19222318 1269280978 1% /share/lfs02[MDT:0]
>> lustrefs-OST0001_UUID 24117248 5942156 18175092 25% /share/lfs02[OST:1]
>> lustrefs-OST0002_UUID 24117248 5816469 18300779 24% /share/lfs02[OST:2]
>> lustrefs-OST0003_UUID 24117248 5982962 18134286 25% /share/lfs02[OST:3]
>>
>> filesystem_summary: 73832475 19222318 54610157 26% /share/lfs02
>
> Again, there are already 19,222,318 files on the file system, so IUsed=19222318. All the OSTs together only have 18,175,092 + 18,300,779 + 18,134,286 = 54,610,157 inodes available, so IFree=54610157. And Inodes = IUsed + IFree = 73832475.
Thanks to Rick for the good explanation here. One thing to add is that it appears that
the /lfs01 filesystem has a default stripe_count=2, since there are 46560885 inodes used
on MDT0000 and 91572739 total objects used on the four OSTs, and 91572739/46560885 = 1.96
OST objects per MDT inode.
If you have a large number of small files, you don't need a high stripe count.
Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud
More information about the lustre-discuss
mailing list