[Lustre-discuss] Performance issues in directories after large number of I/O operations on small files.

[Andreas Dilger]

On Dec 10, 2008  12:18 -0800, Sebastian Gutierrez wrote:
> If he moves delete creates files in a directory on his luster working
> directory the directory eventually gets unmanageably slow.
> He is working with a large number of small files.
> Here is a example of a copied directory that shows the directory
> metadata much smaller but has the exact same data in it.
> 
> drwxr-sr-x  3 user group     4096 Dec  3 17:26 StatisticsQualScore
> drwxr-sr-x  3 user group 12623872 Dec  3 15:59 StatisticsQualScore_old

A 12MB directory would hold in the neighbourhood of 250k files.
It should take about 150MB to cache all of the files in this directory,
which isn't that much memory these days.

The "new" directory appears to be virtually or completely empty as a
4096-byte directory is the minimum possible size, holding only about
100 files in it.

Can you be more specific about what is "unmanageably slow" means?
Is that the create/delete speed, or the "ls -l" speed, or something else?

Is it "more slow" after a number of create/delete cycles, or only slower
than after the directory was first created?  Is it "more slow" after the
files have been deleted than when it the files are still in the directory?

> After a copy of the directory the directory speeds improve.
> 
> Is this happening because of a LRU sizing issue or could there be
> something else that could be causing this?  We currently have a work
> around but I would like to have a better understanding of this and
> possible solution or option I can implement.

I don't think this is related to the lock LRU size.  Rather, the directory
size itself is growing because of many files being created in it, but
deleting files from the directory does not shrink the directory again.
This is how ext3/4/ldiskfs works, on the assumption that the directory
will again grow to this size so freeing the blocks again is unnecessary
work.

The problem is that after deleting 99% of the files in the directory
the remaining ones are spread around the disk, so instead of having all
of them read/written with only a few blocks of data this requires many
more blocks to be read/written.

If the directory will grow to contain 250k files again then there isn't
much that can be done because it will again grow to the large size.
The ability to shrink ext2/3/ldiskfs directories has been discussed in
the past, but hasn't really gotten much attention because workloads
that do this are rare and the workaround is straight forward.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.