[Lustre-discuss] Bad distribution of files among OSTs

[Andreas Dilger]

On 2009-10-30, at 12:07, Thomas Roth wrote:
> in our 196 OST - Cluster, the previously perfect distribution of files
> among the OSTs is not working anymore, since ~ 2 weeks.
> The filling for most OSTs is between 57% and 62%, but some (~10)  have
> risen up to 94%. I'm trying to fix that by having these OSTs  
> deactivated
> on the MDT and finding and migrating away data from them, but it seems
> I'm not fast enough and it's a ongoing problem - I've just deactivated
> another OST with threatening 67%.

Is this correlated to some upgrade of Lustre?  What version are you  
using?


> Our qos_prio_free is at the default 90%.
>
> Our OST's sizes are between 2.3TB and 4.5TB. We use striping level  
> 1, so
> it would be possible to fill up an OST by just creating a 2TB file.
> However, I'm not aware of any such gigafiles (using robinhood to get a
> picture of our file system).

To fill the smallest OST from 60% to 90% would only need a few file that
total 0.3 * 2.3TB, or 690GB.  One way to find such files is to mount the
full OSTs with ldiskfs and do "find /mnt/ost/O/0 -size +100G" to list  
the
object IDs that are very large, and then in bug 21244 I've written a  
small
program that dumps the MDS inode number from the specified objects.  You
can then use "debugfs -c -R "ncheck {list of inode numbers} /dev/$ 
{mdsdev}"
on the MDS to find the pathnames of those files.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.