[Lustre-discuss] Limit of OSTs per OSS?

[Andreas Dilger]

On Aug 20, 2009  09:27 +0200, Arne Wiebalck wrote:
>> Is there a reason to do this instead of, say, two 5TB OSTs using MD RAID-0?
>> Or for that matter one 8+2 8TB OST with MD RAID-6?  That will give
>> better space utilization if you have large files, otherwise you will
>
> What is a 'large' file for you?

Well, this is relative to the size of the OST itself, and the striping.

For example, if you have 1TB OSTs and stripecount=1, but your files are 
500GB in size, then you might only be able to fit one onto each OST
(due to fs-internal overhead), leaving about 50% of each OST unusable.
If you have 8TB OSTs then you could store 15 files/OST, leaving about
6% of each OST unusable.  This is "external" free space fragmentation.

Similarly, if you have, say, 5% of space free in each OST, then for an
8TB OST the free space would likely be in 8x larger chunks of space,
vs a 1TB OST.

Other benefits of fewer, larger OSTs:
- less configuration
- more bandwidth per stripe
- more space per file (if you have > 160 OSTs)

The reasons for NOT going to a larger single OST are:
- more resource contention
- larger point of failure
- longer e2fsck time

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.