[lustre-discuss] stripe count recommendation, and proposal for auto-stripe tool

[Chris Hunter]

Hi Nathan,
Although not useful short-term, there is a plan to address this issue
via feature "progressive file layout":
http://lustre.ornl.gov/ecosystem-2016/documents/tech/Mohr-UTK-PFL.pdf

Generally the more users, the more headache is file striping.

A possible hack is to:
1) set small default stripecount 
2) use robinhood to periodically generate a list of (inactive) large
files and restripe them with lfs_migrate script.

regards,
chris hunter
chuntera at gmail.com


> Greetings All,
> 
> I'm looking for your experience and perhaps some lively discussion
> regarding "best practices" for choosing a file stripe count.  The
> Lustre
> manual has good tips on "Choosing a Stripe Size", and in practice the
> default 1M rarely causes problems on our systems. Stripe Count on the
> other
> hand is far more difficult to chose a single value that is efficient
> for a
> general purpose and multi-use site-wide file system.
> 
> Since there is the "increased overhead" of striping, and weather
> applications do unfortunately write MANY tiny files, we usually keep
> the
> filesystem default stripe count at 1.  Unfortunately, there are
> several
> users who then write very large and shared-access files with that
> default.
> I would like to be able to tell them to restripe... but without
> digging
> into the specific application and access pattern it is hard to know
> what
> count to recommend.  Plus there is the "stripe these but not those"
> confusion... it is common for users to have a few very large data
> files and
> many small log or output image files in the SAME directory.
>