[Lustre-discuss] Bad distribution of files among OSTs
Thomas Roth
t.roth at gsi.de
Sun Nov 1 01:03:04 PST 2009
Another question:
Could this situation, 10 full OSTs out of 200, lead to a significant drop in performance?
Before, we could usually get the full 110MB/s or so over the 1Gbit/s ethernet lines of the clients.
That had dropped to about 50%, but we did not find any other odd thing than the filling levels of
the OSTs.
Regards,
Thomas
Andreas Dilger wrote:
> On 2009-10-30, at 12:07, Thomas Roth wrote:
>> in our 196 OST - Cluster, the previously perfect distribution of files
>> among the OSTs is not working anymore, since ~ 2 weeks.
>> The filling for most OSTs is between 57% and 62%, but some (~10) have
>> risen up to 94%. I'm trying to fix that by having these OSTs deactivated
>> on the MDT and finding and migrating away data from them, but it seems
>> I'm not fast enough and it's a ongoing problem - I've just deactivated
>> another OST with threatening 67%.
>
> Is this correlated to some upgrade of Lustre? What version are you using?
>
>
>> Our qos_prio_free is at the default 90%.
>>
>> Our OST's sizes are between 2.3TB and 4.5TB. We use striping level 1, so
>> it would be possible to fill up an OST by just creating a 2TB file.
>> However, I'm not aware of any such gigafiles (using robinhood to get a
>> picture of our file system).
>
> To fill the smallest OST from 60% to 90% would only need a few file that
> total 0.3 * 2.3TB, or 690GB. One way to find such files is to mount the
> full OSTs with ldiskfs and do "find /mnt/ost/O/0 -size +100G" to list the
> object IDs that are very large, and then in bug 21244 I've written a small
> program that dumps the MDS inode number from the specified objects. You
> can then use "debugfs -c -R "ncheck {list of inode numbers} /dev/${mdsdev}"
> on the MDS to find the pathnames of those files.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
--
--------------------------------------------------------------------
Thomas Roth
Gesellschaft für Schwerionenforschung
Planckstr. 1 - 64291 Darmstadt, Germany
Department: Informationstechnologie
Location: SB3 1.262
Phone: +49-6159-71 1453 Fax: +49-6159-71 2986
-------------- next part --------------
A non-text attachment was scrubbed...
Name: t_roth.vcf
Type: text/x-vcard
Size: 298 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20091101/6059fd0f/attachment.vcf>
More information about the lustre-discuss
mailing list