<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7654.12">
<TITLE>RE: [Lustre-discuss] Bad distribution of files among OSTs</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/plain format -->
<P><FONT SIZE=2>Another question I had with regards to this is how long have your OSS's been running without a reboot?<BR>
<BR>
Mine have been up for 148 days which is probably longer than ever before. And now that I've said this, it just occurred to me that one of them was rebooted about a three weeks ago and all the others have been up for almost 6 months. <BR>
<BR>
I don't know if this has any relevance, but it's the only thing I can think of that's different. <BR>
<BR>
Ron.<BR>
<BR>
<BR>
-----Original Message-----<BR>
From: lustre-discuss-bounces@lists.lustre.org on behalf of Thomas Roth<BR>
Sent: Sun 11/1/2009 4:03 AM<BR>
To: Andreas Dilger<BR>
Cc: lustre-discuss@lists.lustre.org<BR>
Subject: Re: [Lustre-discuss] Bad distribution of files among OSTs<BR>
<BR>
Another question:<BR>
Could this situation, 10 full OSTs out of 200, lead to a significant drop in performance?<BR>
Before, we could usually get the full 110MB/s or so over the 1Gbit/s ethernet lines of the clients.<BR>
That had dropped to about 50%, but we did not find any other odd thing than the filling levels of<BR>
the OSTs.<BR>
<BR>
Regards,<BR>
Thomas<BR>
<BR>
Andreas Dilger wrote:<BR>
> On 2009-10-30, at 12:07, Thomas Roth wrote:<BR>
>> in our 196 OST - Cluster, the previously perfect distribution of files<BR>
>> among the OSTs is not working anymore, since ~ 2 weeks.<BR>
>> The filling for most OSTs is between 57% and 62%, but some (~10) have<BR>
>> risen up to 94%. I'm trying to fix that by having these OSTs deactivated<BR>
>> on the MDT and finding and migrating away data from them, but it seems<BR>
>> I'm not fast enough and it's a ongoing problem - I've just deactivated<BR>
>> another OST with threatening 67%.<BR>
><BR>
> Is this correlated to some upgrade of Lustre? What version are you using?<BR>
><BR>
><BR>
>> Our qos_prio_free is at the default 90%.<BR>
>><BR>
>> Our OST's sizes are between 2.3TB and 4.5TB. We use striping level 1, so<BR>
>> it would be possible to fill up an OST by just creating a 2TB file.<BR>
>> However, I'm not aware of any such gigafiles (using robinhood to get a<BR>
>> picture of our file system).<BR>
><BR>
> To fill the smallest OST from 60% to 90% would only need a few file that<BR>
> total 0.3 * 2.3TB, or 690GB. One way to find such files is to mount the<BR>
> full OSTs with ldiskfs and do "find /mnt/ost/O/0 -size +100G" to list the<BR>
> object IDs that are very large, and then in bug 21244 I've written a small<BR>
> program that dumps the MDS inode number from the specified objects. You<BR>
> can then use "debugfs -c -R "ncheck {list of inode numbers} /dev/${mdsdev}"<BR>
> on the MDS to find the pathnames of those files.<BR>
><BR>
> Cheers, Andreas<BR>
> --<BR>
> Andreas Dilger<BR>
> Sr. Staff Engineer, Lustre Group<BR>
> Sun Microsystems of Canada, Inc.<BR>
><BR>
<BR>
--<BR>
--------------------------------------------------------------------<BR>
Thomas Roth<BR>
Gesellschaft fr Schwerionenforschung<BR>
Planckstr. 1 - 64291 Darmstadt, Germany<BR>
Department: Informationstechnologie<BR>
Location: SB3 1.262<BR>
Phone: +49-6159-71 1453 Fax: +49-6159-71 2986<BR>
<BR>
<BR>
</FONT>
</P>
</BODY>
</HTML>