[Lustre-discuss] Unbalanced OSTs

[Andreas Dilger]

On 2010-02-18, at 10:19, syed haider wrote:
> 1 combined MGS/MDT, 4 OSS's attached to DDN controller/SAN
> We have 32 1Tb OSTs (8 on each OSS) and because of the nature of our  
> jobs, the files written to lustre vary greatly (5MB ~ 400GB size  
> files).
> we have very high inconsistencies in disk space usage. One OST maybe  
> only 40% full while another 94% full.

If your files are up to 40% of the OST size, then only 1 + fraction  
files
are needed to hit a 50% imbalance between OSTs.  I would really  
suggest to
use 8TB OSTs to minimize this effect.

> The only solution I see is increasing each OST from 1TB to maybe 2  
> or 3TB? So this lead me to my second question - what type of  
> performance degradation (if any) am I looking at if increase the  
> size of the OSTs? and is there a limit to the size of OSTs? Any  
> input is appreciated!

Also, with the DDN it doesn't make any sense to split an 8TB tier of  
disks into 8 separate 1TB LUNs.  That is just causing extra contention  
on the disks shared between these 8 LUNs, and hurting your  
performance.  Most DDN installations use the 8TB LUN size, which is  
the maximum size until 1.8.2 where it goes to 16TB (enough for a tier  
of 2TB disks).

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.