[Lustre-discuss] Bad distribution of files among OSTs

[Andreas Dilger]

On 2009-11-01, at 02:03, Thomas Roth wrote:
> Could this situation, 10 full OSTs out of 200,
> lead to a significant drop in performance?
> Before, we could usually get the full 110MB/s
> or so over the 1Gbit/s ethernet lines of the clients.
> That had dropped to about 50%, but we did not
> find any other odd thing than the filling levels of
> the OSTs.

Yes, this is entirely possible.  If the OST is very
full, then it takes longer to find free blocks.

> Andreas Dilger wrote:
>> On 2009-10-30, at 12:07, Thomas Roth wrote:
>>> in our 196 OST - Cluster, the previously perfect distribution of  
>>> files
>>> among the OSTs is not working anymore, since ~ 2 weeks.
>>> The filling for most OSTs is between 57% and 62%, but some (~10)   
>>> have
>>> risen up to 94%. I'm trying to fix that by having these OSTs  
>>> deactivated
>>> on the MDT and finding and migrating away data from them, but it  
>>> seems
>>> I'm not fast enough and it's a ongoing problem - I've just  
>>> deactivated
>>> another OST with threatening 67%.
>>
>> Is this correlated to some upgrade of Lustre?  What version are you  
>> using?
>>
>>
>>> Our qos_prio_free is at the default 90%.
>>>
>>> Our OST's sizes are between 2.3TB and 4.5TB. We use striping level  
>>> 1, so
>>> it would be possible to fill up an OST by just creating a 2TB file.
>>> However, I'm not aware of any such gigafiles (using robinhood to  
>>> get a
>>> picture of our file system).
>>
>> To fill the smallest OST from 60% to 90% would only need a few file  
>> that
>> total 0.3 * 2.3TB, or 690GB.  One way to find such files is to  
>> mount the
>> full OSTs with ldiskfs and do "find /mnt/ost/O/0 -size +100G" to  
>> list the
>> object IDs that are very large, and then in bug 21244 I've written  
>> a small
>> program that dumps the MDS inode number from the specified  
>> objects.  You
>> can then use "debugfs -c -R "ncheck {list of inode numbers} /dev/$ 
>> {mdsdev}"
>> on the MDS to find the pathnames of those files.
>>
>> Cheers, Andreas
>> -- 
>> Andreas Dilger
>> Sr. Staff Engineer, Lustre Group
>> Sun Microsystems of Canada, Inc.
>>
>
> -- 
> --------------------------------------------------------------------
> Thomas Roth
> Gesellschaft für Schwerionenforschung
> Planckstr. 1                -         64291 Darmstadt, Germany
> Department: Informationstechnologie
> Location: SB3 1.262
> Phone: +49-6159-71 1453  Fax: +49-6159-71 2986
>
> <t_roth.vcf>


Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.