[Lustre-discuss] Understanding lfs output

[Andreas Dilger]

On Aug 20, 2008  06:45 -0400, Mag Gam wrote:
> Is this documented anywhere? We are facing inode shortage problems on
> our filesystem and I would like to show this to my professors. The
> reason why this is  occurring is..

Well, according to the "lfs df -i" output below there are lots of inodes
free.  Depending on the striping you should be able to create at least
63M more files in the filesystem, possibly as many as 144M if you use
only a single stripe per file at which point you run out of inodes
for the OST objects.

> I checked the operations manual and Lustre "inter galactic" file
> system but no luck of this issue.

I thought it was in the "Knowledge Base" or "FAQ" section, maybe not.
Our trusty Documentation team should be able to add this if it isn't
there already.

> On Wed, Aug 20, 2008 at 1:55 AM, Andreas Dilger <adilger at sun.com> wrote:
> > On Aug 19, 2008  20:27 -0400, Mag Gam wrote:
> >> Yes, I have looked thru the lists but I could not really get this
> >> question answered. tune2fs is a ext2/3 setting, and I though each file
> >> I create on a Lustre filesystem an inode gets created on the MDS and
> >> the OST as an object.
> >>
> >> But, I don't understand how there is that much of a space discrepancy.
> >
> > The reason for this discrepancy is because of "safety".  It is possible
> > that the striping is too large to fit into the inode, so it needs an
> > external block.  If that is needed for every remaining inode on the MDS
> > then the limit on the number of files created is equal to the number of
> > free blocks on the MDS.  In order to avoid over-stating the number of
> > possible files that can be created (sometimes important to e.g. RPM
> > or regression tests), "df" and "lfs df" will return min(free inodes,
> > free blocks) for the inode count.
> >
> > If (as usually happens) inodes are created that do not need external blocks
> > to store the striping information then more inodes "appear" (i.e. as
> > "IUsed" increases so does "Inodes" and "IFree" stays constant).
> >
> >> Also, how does striping account for in inodes? Lets say I stipe a file
> >> on 4 OSTs, I suppose 5 (1 for MDS, 4 for OSTs) inodes are occupied,
> >> correct?
> >
> > Yes, though each of the inodes are in different filesystems.
> >
> >> On Tue, Aug 19, 2008 at 4:30 PM, Brian J. Murrell <Brian.Murrell at sun.com> wrote:
> >> > On Sun, 2008-08-17 at 15:50 -0700, Mag Gam wrote:
> >> >> My departments Lustre's lfs df -i looks like this
> >> >>
> >> >> UUID                    Inodes     IUsed     IFree IUse% Mounted on
> >> >> xfs001-MDT0000_UUID   82565322  19073586  63491736   23%
> >> >> /xfs/engine1/xfs001[MDT:0]
> >> >> xfs001-OST0000_UUID   41943040   5076445  36866595   12%
> >> >> /xfs/engine1/xfs001[OST:0]
> >> >> xfs001-OST0001_UUID   41943040   3735715  38207325    8%
> >> >> /xfs/engine1/xfs001[OST:1]
> >> >> xfs001-OST0002_UUID   41943040   4784447  37158593   11%
> >> >> /xfs/engine1/xfs001[OST:2]
> >> >> xfs001-OST0003_UUID   41943040   4556627  37386413   10%
> >> >> /xfs/engine1/xfs001[OST:3]
> >> >>
> >> >> filesystem summary:   82565322  19073586  63491736   23% /xfs/engine1/xfs001
> >> >>
> >> >>
> >> >> Is it safe to say, "
> >> >> My MDT has "19073586" 19 million inodes free? or should I consider.
> >> >>
> >> >> However, when I check my inode count on my mds using tune2fs it reports this.
> >> >> Inode count:              524288000
> >> >> Free inodes:              505214414
> >> >>
> >> >> Am I missing something?
> >> >
> >> > Yes.  lfs df does not simply report the free inodes on the MDT.  It's
> >> > more complicated than that.  This has been discussed either here or in a
> >> > bugzilla bug (don't recall which).  I don't recall the bug or when the
> >> > lustre-discussion was, offhand, but a search should not be terribly
> >> > difficult to find the answer.
> >> >
> >> > b.
> >> >
> >> >
> >> > _______________________________________________
> >> > Lustre-discuss mailing list
> >> > Lustre-discuss at lists.lustre.org
> >> > http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >> >
> >> >
> >> _______________________________________________
> >> Lustre-discuss mailing list
> >> Lustre-discuss at lists.lustre.org
> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >
> > Cheers, Andreas
> > --
> > Andreas Dilger
> > Sr. Staff Engineer, Lustre Group
> > Sun Microsystems of Canada, Inc.
> >
> >

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.