[Lustre-discuss] Large scale delete results in lag on clients

[Arden Wiebe]

--- On Thu, 8/6/09, Andreas Dilger <adilger at sun.com> wrote:

> From: Andreas Dilger <adilger at sun.com>
> Subject: Re: [Lustre-discuss] Large scale delete results in lag on clients
> To: "Jim McCusker" <james.mccusker at yale.edu>
> Cc: "lustre-discuss" <lustre-discuss at lists.lustre.org>
> Date: Thursday, August 6, 2009, 1:27 PM
> On Aug 06, 2009  15:08 -0400,
> Jim McCusker wrote:
> > We have a 15 TB luster volume across 4 OSTs and we
> recently deleted over 4
> > million files from it in order to free up the 80 GB
> MDT/MDS (going from 100%
> > capacity on it to 81%. As a result, after the rm
> completed, there is
> > significant lag on most file system operations (but
> fast access once it
> > occurs) even after the two servers that host the
> targets were rebooted. It
> > seems to clear up for a little while after reboot, but
> comes back after some
> > time.
> > 
> > Any ideas?
> 
> The Lustre unlink processing is somewhat asynchronous, so
> you may still be
> catching up with unlinks.  You can check this by
> looking at the OSS service
> RPC stats file to see if there are still object destroys
> being processed
> by the OSTs.  You could also just check the system
> load/io on the OSTs to
> see how busy they are in a "no load" situation.
> 
> 
> > For the curious, we host a large image archive (almost
> 400k images) and do
> > research on processing them. We had a lot of
> intermediate files that we
> > needed to clean up:
> > 
> >  http://krauthammerlab.med.yale.edu/imagefinder
> (currently laggy and
> > unresponsive due to this problem)
> > 

Jim, from the web side perspective it seems responsive.  Are you actually serving the images from the lustre cluster?  I have ran a few searches looking for "Purified HIV Electron Microscope" and your project returns 15 pages of results with great links to full abstracts almost instantly but obviously none with real purified HIV electron microscope images similar to a real pathogenic virus like http://krauthammerlab.med.yale.edu/imagefinder/Figure.external?sp=62982&state:Figure=BrO0ABXcRAAAAAQAACmRvY3VtZW50SWRzcgARamF2YS5sYW5nLkludGVnZXIS4qCk94GHOAIAAUkABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAD2Cg%3D%3D 

Again though, not surprisingly some of the same proteins in this virus are present in molecular clones of HIV. I'll have to agree more now with http://www.karymullis.com that using PCR to detect viral infection is a bad idea lacking proper viral isolation of HIV that is still overlooked after 25 years.  No doubt http://ThePerthGroup.com are probably correct in their views but enough curiosity.  

Have you physically separated your MDS/MDT from the MGS portion on different servers?  I somehow doubt you overlooked this but if you didn't for some reason this could be a cause of unresponsiveness on the client side.  Again if your serving up the images from the cluster I find it works great.

http://krauthammerlab.med.yale.edu/imagefinder/Figure.external?sp=62982&state:Figure=BrO0ABXcRAAAAAQAACmRvY3VtZW50SWRzcgARamF2YS5sYW5nLkludGVnZXIS4qCk94GHOAIAAUkABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAD2Cg%3D%3D

> > Thanks,
> > Jim
> > --
> > Jim McCusker
> > Programmer Analyst
> > Krauthammer Lab, Pathology Informatics
> > Yale School of Medicine
> > james.mccusker at yale.edu
> | (203) 785-6330
> > http://krauthammerlab.med.yale.edu
> 
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss at lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>