[Lustre-discuss] e2scan for cleaning scratch space

[Jim Garlick]

Hi Brock,

We do this on our scratch file systems as well.  Our policy is to remove
files that have a time of last use of 60 days or older, where time of last
use is defined as the greatest of [acm]time.

E2scan seems like an evil kludge to me, at least if you don't quiesce
your servers first, which is impractical for us to do.  It is especially
painful if you have to correlate data taken seperately from OST and MDT
which I guess you need not do if a) you have a release with trustable MDS
times (not 1.6.6), or b) you plan to stat(2) the MDS-generated list on a
client before purging them.

See bug 16942 which describes an MDS resident "purge thread" that
continually walks the file system implementing the policy.  This is how
we thought purging ought to be optimized and we hope to have this in place
by the time we put 1.8 in production.

Meanwhile, we are walking the file system from a client.  Note that for this

  lfs find --type f --atime +60 --mtime +60 --ctime +60 /mnt/lustre >list

beats

  find /mnt/lustre -type f -atime +60 -mtime +60 -ctime +60 >list

by a wide margin since most of the time it does not have to contact
the OST's, which stat(2) will always do for the foreseeable future 
(until size-on-MDS) to get st_size.

Jim

On Wed, Mar 04, 2009 at 01:48:10PM -0500, Brock Palen wrote:
> e2scan will show me all the files that have changed from a date, but  
> I want to know all the files that have not changed sense some date.
> 
> The goal is to make a system for purging scratch spaces that is fast,  
> and minimum wear on the filesystem.
> How are groups doing this now?  Are you using e2scan?
> Is there a way to have e2scan not only list the file but also the  
> mtime/ctime in the log file, so that we can sort oldest to newest?
> 
> Thank  you!
> 
> 
> Brock Palen
> www. umich.edu/~brockp
> Center for Advanced Computing
> brockp at umich.edu
> (734)936-1985
> 
> 
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http:// lists.lustre.org/mailman/listinfo/lustre-discuss