[Lustre-discuss] Bad distribution of files among OSTs

Thomas Roth t.roth at gsi.de
Sun Nov 1 01:03:04 PST 2009


Another question:
Could this situation, 10 full OSTs out of 200, lead to a significant drop in performance?
Before, we could usually get the full 110MB/s or so over the 1Gbit/s ethernet lines of the clients.
That had dropped to about 50%, but we did not find any other odd thing than the filling levels of
the OSTs.

Regards,
Thomas

Andreas Dilger wrote:
> On 2009-10-30, at 12:07, Thomas Roth wrote:
>> in our 196 OST - Cluster, the previously perfect distribution of files
>> among the OSTs is not working anymore, since ~ 2 weeks.
>> The filling for most OSTs is between 57% and 62%, but some (~10)  have
>> risen up to 94%. I'm trying to fix that by having these OSTs deactivated
>> on the MDT and finding and migrating away data from them, but it seems
>> I'm not fast enough and it's a ongoing problem - I've just deactivated
>> another OST with threatening 67%.
> 
> Is this correlated to some upgrade of Lustre?  What version are you using?
> 
> 
>> Our qos_prio_free is at the default 90%.
>>
>> Our OST's sizes are between 2.3TB and 4.5TB. We use striping level 1, so
>> it would be possible to fill up an OST by just creating a 2TB file.
>> However, I'm not aware of any such gigafiles (using robinhood to get a
>> picture of our file system).
> 
> To fill the smallest OST from 60% to 90% would only need a few file that
> total 0.3 * 2.3TB, or 690GB.  One way to find such files is to mount the
> full OSTs with ldiskfs and do "find /mnt/ost/O/0 -size +100G" to list the
> object IDs that are very large, and then in bug 21244 I've written a small
> program that dumps the MDS inode number from the specified objects.  You
> can then use "debugfs -c -R "ncheck {list of inode numbers} /dev/${mdsdev}"
> on the MDS to find the pathnames of those files.
> 
> Cheers, Andreas
> -- 
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
> 

-- 
--------------------------------------------------------------------
Thomas Roth
Gesellschaft für Schwerionenforschung
Planckstr. 1                -         64291 Darmstadt, Germany
Department: Informationstechnologie
Location: SB3 1.262
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

-------------- next part --------------
A non-text attachment was scrubbed...
Name: t_roth.vcf
Type: text/x-vcard
Size: 298 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20091101/6059fd0f/attachment.vcf>


More information about the lustre-discuss mailing list