<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7654.12">
<TITLE>RE: [Lustre-discuss] Bad distribution of files among OSTs</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/plain format -->
<BR>
<P><FONT SIZE=2>Strangely (although I'm sure it's not related) I have seen the exact same behavior on my Lustre cluster in the last month or so. I have also never seen this before, and to the best of my knowledge there is no change in usage patterns.<BR>
<BR>
I'm running 1.6.7.2 on the servers.<BR>
<BR>
Ron Jerome<BR>
National Research Council Canada.<BR>
<BR>
<BR>
-----Original Message-----<BR>
From: lustre-discuss-bounces@lists.lustre.org on behalf of Thomas Roth<BR>
Sent: Fri 10/30/2009 2:07 PM<BR>
To: lustre-discuss@lists.lustre.org<BR>
Subject: [Lustre-discuss] Bad distribution of files among OSTs<BR>
<BR>
Hi all,<BR>
<BR>
in our 196 OST - Cluster, the previously perfect distribution of files<BR>
among the OSTs is not working anymore, since ~ 2 weeks.<BR>
The filling for most OSTs is between 57% and 62%, but some (~10) have<BR>
risen up to 94%. I'm trying to fix that by having these OSTs deactivated<BR>
on the MDT and finding and migrating away data from them, but it seems<BR>
I'm not fast enough and it's a ongoing problem - I've just deactivated<BR>
another OST with threatening 67%.<BR>
<BR>
Our qos_prio_free is at the default 90%.<BR>
<BR>
Our OST's sizes are between 2.3TB and 4.5TB. We use striping level 1, so<BR>
it would be possible to fill up an OST by just creating a 2TB file.<BR>
However, I'm not aware of any such gigafiles (using robinhood to get a<BR>
picture of our file system).<BR>
<BR>
In addition, our user's behavior should not have changed recently. In<BR>
August, the entire cluster had filled up to almost 80% in a neatly even<BR>
distribution among the OSTs, so we extended the cluster by more OSTs,<BR>
migrating data to even the filling between old and new ones. This also<BR>
succeeded, and up to October there was no indication of something not<BR>
working.<BR>
<BR>
There are no error message in the logs that would point to some OSTs<BR>
being favored ;-)<BR>
<BR>
So, what could be the cause of this misdistribution?<BR>
<BR>
Regards,<BR>
Thomas<BR>
<BR>
<BR>
_______________________________________________<BR>
Lustre-discuss mailing list<BR>
Lustre-discuss@lists.lustre.org<BR>
<A HREF="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</A><BR>
<BR>
</FONT>
</P>
</BODY>
</HTML>