[Lustre-discuss] Bad distribution of files among OSTs

[Jerome, Ron]

Strangely (although I'm sure it's not related) I have seen the exact same behavior on my Lustre cluster in the last month or so. I have also never seen this before, and to the best of my knowledge there is no change in usage patterns.

I'm running 1.6.7.2 on the servers.

Ron Jerome
National Research Council Canada.


-----Original Message-----
From: lustre-discuss-bounces at lists.lustre.org on behalf of Thomas Roth
Sent: Fri 10/30/2009 2:07 PM
To: lustre-discuss at lists.lustre.org
Subject: [Lustre-discuss] Bad distribution of files among OSTs
 
Hi all,

in our 196 OST - Cluster, the previously perfect distribution of files
among the OSTs is not working anymore, since ~ 2 weeks.
The filling for most OSTs is between 57% and 62%, but some (~10)  have
risen up to 94%. I'm trying to fix that by having these OSTs deactivated
on the MDT and finding and migrating away data from them, but it seems
I'm not fast enough and it's a ongoing problem - I've just deactivated
another OST with threatening 67%.

Our qos_prio_free is at the default 90%.

Our OST's sizes are between 2.3TB and 4.5TB. We use striping level 1, so
it would be possible to fill up an OST by just creating a 2TB file.
However, I'm not aware of any such gigafiles (using robinhood to get a
picture of our file system).

In addition, our user's behavior should not have changed recently. In
August, the entire cluster had filled up to almost 80% in a neatly even
distribution among the OSTs, so we extended the cluster by more OSTs,
migrating data to even the filling between old and new ones. This also
succeeded, and up to October there was no indication of something not
working.

There are no error message in the logs that would point to some OSTs
being favored ;-)

So, what could be the cause of this misdistribution?

Regards,
Thomas


_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20091030/43c5f8cb/attachment.htm>