[Lustre-discuss] Small files performance

[Andreas Dilger]

On 2009-10-30, at 05:21, Arne Wiebalck wrote:
> We're currently experimenting with Lustre and small files (<=1MB)
> and some questions came to mind:
>
> - will writing files that are smaller than the stripe_size and that
>  are written to a fully striped directory create objects on more
>  than 1 OST?

The striping is set at file creation time, before any information is
known about how much data will go into the file.

> - what would be the main options to tune in order get the best out
>  of Lustre for small files? (If the answer to the 1st question is
>  'yes', then 'avoid striping' would be a first candidate, I guess ;)


Right - wide striping should only be used when it is known that the
file size will be very large and/or accessed by many nodes at one time.

Secondly, using RAID-1+0 OSTs instead of RAID-5/6 will definitely also
improve performance, since there is RAID parity overhead for writing
small chunks of data to disk.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.