[lustre-discuss] Removing large directory tree

[Sreedhar]

Hi,

Just like Malcolm suggested, before I implemented Robinhood, I built the
list and removed files (that were not accessed in more than 31 days). I
found it much faster than all other methods.

     /usr/bin/lfs find /scratch/${folder} -A +31 -p -t f | xargs -0 -n 10
-P 8 /bin/rm

10 and 8 for flags n and p worked well for me on our system.

Sreedhar.
New York University.


On Friday, July 10, 2015, Cowe, Malcolm J <malcolm.j.cowe at intel.com
<javascript:_e(%7B%7D,'cvml','malcolm.j.cowe at intel.com');>> wrote:

>  There’s something in the rm command that makes recursive deletes really
> expensive, although I don’t know why. I’ve found in the past that even
> running a regular find ... –exec rm {} \; has been quicker. Running lfs
> find to build the file list would presumably be quicker still.
>
>
>
> Malcolm.
>
>
>
>
>
> *From:* lustre-discuss [mailto:lustre-discuss-bounces at lists.lustre.org] *On
> Behalf Of *Andrus, Brian Contractor
> *Sent:* Saturday, July 11, 2015 8:05 AM
> *To:* lustre-discuss at lists.lustre.org
> *Subject:* [lustre-discuss] Removing large directory tree
>
>
>
> All,
>
>
>
> I understand that doing recursive file operations can be taxing on lustre.
>
> So, I wonder if there is a preferred performance-minded way to remove an
> entire directory tree that is several TB in size.
>
> The standard rm -rf ./dir seems to spike the cpu usage on my OSSes where
> it sits and sometimes causes clients to be evicted.
>
>
>
> Brian Andrus
>
> ITACS/Research Computing
>
> Naval Postgraduate School
>
> Monterey, California
>
> voice: 831-656-6238
>
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>

-- 
Sreedhar Manchu
HPC Systems Administrator
eSystems & Research Services
New York University, New York 10012
*http://unixoperator.blogspot.com <http://unixoperator.blogspot.com>*
*https://wikis.nyu.edu/display/~sm4082
<https://wikis.nyu.edu/display/~sm4082>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20150710/94cb92ed/attachment.htm>