[Lustre-discuss] Unbalanced OST--for discussion purposes

[Andreas Dilger]

On 2010-03-02, at 13:45, Ms. Megan Larko wrote:
> I logged directly into the OSS (OSS4) and just ran a df (along with a
> periodic check of the log files).  I last looked about two weeks ago
> (I know it was after 17 Feb).   Anyway, the OST0007 is more full than
> any of the other OSTs.  The default lustre stripe (I believe that is
> set to 1) is used.    Can just one file shift the size used of one OST
> that significantly?

Sure, this is easy if the size of a single file is be a large fraction  
of the OST size.  This is one reason why we recommend people use  
larger OSTs (up to 16TB in 1.8.2 with RHEL5.4) instead of e.g. 1TB or  
less that is sometimes reported here.

> What other reasonable explanation for a difference on one OST in  
> comparison with the others?  Could this cause a lustre performance  
> hit at this point?

It is possible, if the filesystem is getting very full and it causes  
more seeking to do IO.  At the 84% you report below it is starting to  
get into that range - I wouldn't recommend running the filesystem  
beyond 90% full unless you are more concerned with space usage than  
performance.

You can find the file(s) that are abnormally large on that particular  
OST by running (preferably on a client mountpoint on the MDS):

   lfs find --obd crew8-OST0006_UUID -size +10G /mnt/lustre

>   [root at oss4 ~]# df -h
> Filesystem            Size  Used Avail Use% Mounted on
>
> /dev/sdb1             6.3T  3.6T  2.5T  60% /srv/lustre/OST/crew8- 
> OST0000
> /dev/sdb2             6.3T  4.1T  1.9T  69% /srv/lustre/OST/crew8- 
> OST0001
> /dev/sdc1             6.3T  3.3T  2.8T  55% /srv/lustre/OST/crew8- 
> OST0002
> /dev/sdc2             6.3T  3.3T  2.7T  56% /srv/lustre/OST/crew8- 
> OST0003
> /dev/sdd1             6.3T  3.5T  2.6T  58% /srv/lustre/OST/crew8- 
> OST0004
> /dev/sdd2             6.3T  4.1T  1.9T  69% /srv/lustre/OST/crew8- 
> OST0005
> /dev/sdi1             6.3T  3.9T  2.2T  65% /srv/lustre/OST/crew8- 
> OST0006
> /dev/sdi2             6.3T  5.0T 1015G  84%
> /srv/lustre/OST/crew8-OST0007     <----
> /dev/sdj1             6.3T  3.4T  2.7T  56% /srv/lustre/OST/crew8- 
> OST0008
> /dev/sdj2             6.3T  3.3T  2.7T  56% /srv/lustre/OST/crew8- 
> OST0009
> /dev/sdk1             6.3T  3.4T  2.7T  56% /srv/lustre/OST/crew8- 
> OST0010
> /dev/sdk2             6.3T  3.8T  2.2T  64% /srv/lustre/OST/crew8- 
> OST0011

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.