[Lustre-discuss] Small files

[Nikolay Kvetsinski]

Thanks mate, what you say makes sense. Anyway I`m dealing with different
file sizes, varying from KBs to GBs. Luckily some of the files are grouped
in folders, for example in folder /fs/X all files will be 9MB, which I`ll
not stripe. In /fs/Y all files will be GBs, which I`ll stripe among all
OSTs. I guess the problem now is to figure out the optimal stripe
size/stripe count for the files in the range from a couple of hundreds of
MBs to GB .... on top of that, what I said about files with similar size
being put in their own directory might not always be true .....
Unfortunately I cant "teach"  users to think before they generate their
files.

Cheers,


On Wed, Jul 3, 2013 at 12:18 PM, Kevin Van Maren <KVanMaren at fusionio.com>wrote:

> With a 1MB default stripe size, anything that is <1 MB is a "small" file.
>  In general, I would not stripe 10MB files across more than one OST (each).
>
> Kevin
>
>
> On Jul 3, 2013, at 3:11 AM, Nikolay Kvetsinski wrote:
>
> > Hello again guys,
> >
> > I constantly read about that Lustre is not very good with "small files".
> However, there is no definition of small file in Lustre point of view.
> Would you be able to draw some borders on what is considered to be a small
> file ,  for example is a 10 MB considered a small file, and should a
> directory holding such files be striped on few OSTs, and should it be
> striped at all.
> >
> > Cheers,
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss at lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
> This e-mail (and any attachments) is confidential and may be privileged.
>  Any unauthorized use, copying, disclosure or dissemination of this
> communication is prohibited.  If you are not the intended recipient,
>  please notify the sender immediately and delete all copies of the message
> and its attachments.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130703/d66ed772/attachment.htm>