[lustre-discuss] inodes not adding up

[Mohr Jr, Richard Frank (Rick Mohr)]

> On Apr 13, 2019, at 4:57 AM, Youssef Eldakar <youssefeldakar at gmail.com> wrote:
> 
> For one Lustre filesystem, inode count in the summary is notably less than what the individual OST inode counts would add up to:

The first thing to understand is that every Lustre file will consume one inode on the MDT, and this inode uses attributes to store information about which OSTs the file is striped over.  Then for each file stripe, there will also be an inode consumed on the corresponding OSTs.  For example, a file with stripe_count=4 will consume one inode on the MDT and four inodes on OSTs (one inode on each OST the file is striped over).

> # lfs df -i /lfs01
> UUID                      Inodes       IUsed       IFree IUse% Mounted on
> lustrefs-MDT0000_UUID  2402287616    46560885  2355726731   2% /share/lfs01[MDT:0]
> lustrefs-OST0001_UUID    24117248    22883788     1233460  95% /share/lfs01[OST:1]
> lustrefs-OST0003_UUID    24117248    22903308     1213940  95% /share/lfs01[OST:3]
> lustrefs-OST0004_UUID    24117248    22895442     1221806  95% /share/lfs01[OST:4]
> lustrefs-OST0006_UUID    24117248    22890201     1227047  95% /share/lfs01[OST:6]
> 
> filesystem_summary:     51457138    46560885     4896253  90% /share/lfs01

On this file system, there are already 46,560,885 files which also consume the same number of inodes on the MDT (so IUsed=46560885).  However, even though the MDT has over 2 billion inodes free, every file created in the future will use at least one inode on an OST.  If you add up all the free inodes on all the OSTs, you get 4896253.  So at best, there is only space for 4,896,253 more files.  That is why IFree=4896253.  Then, Inodes = IUsed + IFree = 46,560,885 + 4,896,253 = 51,457,138.

> On another filesystem, this is not the case:
> 
> # lfs df -i /lfs02
> UUID                      Inodes       IUsed       IFree IUse% Mounted on
> lustrefs-MDT0000_UUID  1288503296    19222318  1269280978   1% /share/lfs02[MDT:0]
> lustrefs-OST0001_UUID    24117248     5942156    18175092  25% /share/lfs02[OST:1]
> lustrefs-OST0002_UUID    24117248     5816469    18300779  24% /share/lfs02[OST:2]
> lustrefs-OST0003_UUID    24117248     5982962    18134286  25% /share/lfs02[OST:3]
> 
> filesystem_summary:     73832475    19222318    54610157  26% /share/lfs02

Again, there are already 19,222,318 files on the file system, so IUsed=19222318.   All the OSTs together only have 18,175,092 + 18,300,779 + 18,134,286 = 54,610,157 inodes available, so IFree=54610157.  And Inodes = IUsed + IFree = 73832475.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu