[Lustre-discuss] Bad distribution of files among OSTs

[Jerome, Ron]

Another question I had with regards to this is how long have your OSS's been running without a reboot? 

Mine have been up for 148 days which is probably longer than ever before.  And now that I've said this, it just occurred to me that one of them was rebooted about a three weeks ago and all the others have been up for almost 6 months.  

I don't know if this has any relevance, but it's the only thing I can think of that's different.  

Ron.


-----Original Message-----
From: lustre-discuss-bounces at lists.lustre.org on behalf of Thomas Roth
Sent: Sun 11/1/2009 4:03 AM
To: Andreas Dilger
Cc: lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] Bad distribution of files among OSTs
 
Another question:
Could this situation, 10 full OSTs out of 200, lead to a significant drop in performance?
Before, we could usually get the full 110MB/s or so over the 1Gbit/s ethernet lines of the clients.
That had dropped to about 50%, but we did not find any other odd thing than the filling levels of
the OSTs.

Regards,
Thomas

Andreas Dilger wrote:
> On 2009-10-30, at 12:07, Thomas Roth wrote:
>> in our 196 OST - Cluster, the previously perfect distribution of files
>> among the OSTs is not working anymore, since ~ 2 weeks.
>> The filling for most OSTs is between 57% and 62%, but some (~10)  have
>> risen up to 94%. I'm trying to fix that by having these OSTs deactivated
>> on the MDT and finding and migrating away data from them, but it seems
>> I'm not fast enough and it's a ongoing problem - I've just deactivated
>> another OST with threatening 67%.
> 
> Is this correlated to some upgrade of Lustre?  What version are you using?
> 
> 
>> Our qos_prio_free is at the default 90%.
>>
>> Our OST's sizes are between 2.3TB and 4.5TB. We use striping level 1, so
>> it would be possible to fill up an OST by just creating a 2TB file.
>> However, I'm not aware of any such gigafiles (using robinhood to get a
>> picture of our file system).
> 
> To fill the smallest OST from 60% to 90% would only need a few file that
> total 0.3 * 2.3TB, or 690GB.  One way to find such files is to mount the
> full OSTs with ldiskfs and do "find /mnt/ost/O/0 -size +100G" to list the
> object IDs that are very large, and then in bug 21244 I've written a small
> program that dumps the MDS inode number from the specified objects.  You
> can then use "debugfs -c -R "ncheck {list of inode numbers} /dev/${mdsdev}"
> on the MDS to find the pathnames of those files.
> 
> Cheers, Andreas
> -- 
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
> 

-- 
--------------------------------------------------------------------
Thomas Roth
Gesellschaft für Schwerionenforschung
Planckstr. 1                -         64291 Darmstadt, Germany
Department: Informationstechnologie
Location: SB3 1.262
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20091101/92915c56/attachment.htm>