[Lustre-discuss] Understanding lfs output

[Andreas Dilger]

On Aug 19, 2008  20:27 -0400, Mag Gam wrote:
> Yes, I have looked thru the lists but I could not really get this
> question answered. tune2fs is a ext2/3 setting, and I though each file
> I create on a Lustre filesystem an inode gets created on the MDS and
> the OST as an object.
> 
> But, I don't understand how there is that much of a space discrepancy.

The reason for this discrepancy is because of "safety".  It is possible
that the striping is too large to fit into the inode, so it needs an
external block.  If that is needed for every remaining inode on the MDS
then the limit on the number of files created is equal to the number of
free blocks on the MDS.  In order to avoid over-stating the number of
possible files that can be created (sometimes important to e.g. RPM
or regression tests), "df" and "lfs df" will return min(free inodes,
free blocks) for the inode count.

If (as usually happens) inodes are created that do not need external blocks
to store the striping information then more inodes "appear" (i.e. as
"IUsed" increases so does "Inodes" and "IFree" stays constant).

> Also, how does striping account for in inodes? Lets say I stipe a file
> on 4 OSTs, I suppose 5 (1 for MDS, 4 for OSTs) inodes are occupied,
> correct?

Yes, though each of the inodes are in different filesystems.

> On Tue, Aug 19, 2008 at 4:30 PM, Brian J. Murrell <Brian.Murrell at sun.com> wrote:
> > On Sun, 2008-08-17 at 15:50 -0700, Mag Gam wrote:
> >> My departments Lustre's lfs df -i looks like this
> >>
> >> UUID                    Inodes     IUsed     IFree IUse% Mounted on
> >> xfs001-MDT0000_UUID   82565322  19073586  63491736   23%
> >> /xfs/engine1/xfs001[MDT:0]
> >> xfs001-OST0000_UUID   41943040   5076445  36866595   12%
> >> /xfs/engine1/xfs001[OST:0]
> >> xfs001-OST0001_UUID   41943040   3735715  38207325    8%
> >> /xfs/engine1/xfs001[OST:1]
> >> xfs001-OST0002_UUID   41943040   4784447  37158593   11%
> >> /xfs/engine1/xfs001[OST:2]
> >> xfs001-OST0003_UUID   41943040   4556627  37386413   10%
> >> /xfs/engine1/xfs001[OST:3]
> >>
> >> filesystem summary:   82565322  19073586  63491736   23% /xfs/engine1/xfs001
> >>
> >>
> >> Is it safe to say, "
> >> My MDT has "19073586" 19 million inodes free? or should I consider.
> >>
> >> However, when I check my inode count on my mds using tune2fs it reports this.
> >> Inode count:              524288000
> >> Free inodes:              505214414
> >>
> >> Am I missing something?
> >
> > Yes.  lfs df does not simply report the free inodes on the MDT.  It's
> > more complicated than that.  This has been discussed either here or in a
> > bugzilla bug (don't recall which).  I don't recall the bug or when the
> > lustre-discussion was, offhand, but a search should not be terribly
> > difficult to find the answer.
> >
> > b.
> >
> >
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss at lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >
> >
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.