<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
FWIW, last September in the context of reducing the memory usage on
the mgs, Andreas Dilger had this to say<br>
<br>
<pre wrap="">"ls" is itself fairly inefficient at directory traversal, because all of
the GNU file utilities are bloated and do much more work than necessary
(e.g. "rm" will stat() every file before unlinking it). Using something
like:
find $dir -type f -print0 | xargs -0 munlink
find $dir -type d -print0 | xargs rmdir
will probably be most efficient. "munlink" is a Lustre-specific tool that
just unlinks the filenames passed without stat() on each one. If that is
not available, you can also use "xargs -0 -n 1 unlink" but it will fork a
new unlink process for every file which also adds some overhead.</pre>
bob<br>
<br>
<div class="moz-cite-prefix">On 7/10/2015 7:24 PM, Sreedhar wrote:<br>
</div>
<blockquote
cite="mid:CAAQYKDBxsr4i7p8tANVyH5==cMZx9NKi+Wb5Qw8XMmPku88J6g@mail.gmail.com"
type="cite">Hi,
<div><br>
</div>
Just like Malcolm suggested, before I implemented Robinhood, I
built the list and removed files (that were not accessed in more
than 31 days). I found it much faster than all other methods.
<div><br>
</div>
<div><font size="2"><span
style="background-color:rgba(255,255,255,0)">
/usr/bin/lfs find /scratch/${folder} -A +31 -p -t f | xargs
-0 -n 10 -P 8 /bin/rm</span></font></div>
<div><font size="2"><span
style="background-color:rgba(255,255,255,0)"><br>
</span></font></div>
<div><font size="2"><span
style="background-color:rgba(255,255,255,0)">10 and 8 for
flags n and p worked well for me on our system. </span></font></div>
<div>
<div><br>
</div>
<div>Sreedhar. </div>
<div>New York University. </div>
<div><span></span><font size="2"><span
style="background-color:rgba(255,255,255,0)"> </span></font><br>
<br>
On Friday, July 10, 2015, Cowe, Malcolm J <<a
moz-do-not-send="true"
href="javascript:_e(%7B%7D,'cvml','malcolm.j.cowe@intel.com');"
target="_blank">malcolm.j.cowe@intel.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-GB">
<div>
<p class="MsoNormal">There’s something in the rm command
that makes recursive deletes really expensive,
although I don’t know why. I’ve found in the past that
even running a regular find ... –exec rm {} \; has
been quicker. Running lfs find to build the file list
would presumably be quicker still.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Malcolm.</p>
<p class="MsoNormal"><span style="font-size:10.0pt"
lang="EN-US"> </span></p>
<p class="MsoNormal"> </p>
<div>
<div style="border:none;border-top:solid #b5c4df
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""
lang="EN-US">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""
lang="EN-US"> lustre-discuss [mailto:<a
moz-do-not-send="true">lustre-discuss-bounces@lists.lustre.org</a>]
<b>On Behalf Of </b>Andrus, Brian Contractor<br>
<b>Sent:</b> Saturday, July 11, 2015 8:05 AM<br>
<b>To:</b> <a moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> [lustre-discuss] Removing large
directory tree</span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"> </p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
lang="EN-US">All,</span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
lang="EN-US"> </span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
lang="EN-US">I understand that doing recursive file
operations can be taxing on lustre.</span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
lang="EN-US">So, I wonder if there is a preferred
performance-minded way to remove an entire directory
tree that is several TB in size.</span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
lang="EN-US">The standard rm -rf ./dir seems to
spike the cpu usage on my OSSes where it sits and
sometimes causes clients to be evicted.</span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
lang="EN-US"> </span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="color:#1f497d" lang="EN-US">Brian Andrus</span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="color:#1f497d" lang="EN-US">ITACS/Research
Computing</span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="color:#1f497d" lang="EN-US">Naval
Postgraduate School</span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="color:#1f497d" lang="EN-US">Monterey,
California</span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="color:#1f497d" lang="EN-US">voice:
831-656-6238</span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
lang="EN-US"> </span></p>
</div>
</div>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br>
<a moz-do-not-send="true"
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
</blockquote>
</div>
</div>
<br>
<br>
-- <br>
<div dir="ltr"><font face="arial narrow, sans-serif" size="2">Sreedhar
Manchu<br>
HPC Systems Administrator</font>
<div><font face="arial narrow, sans-serif" size="2">eSystems
& Research Services<br>
New York University, New York 10012</font></div>
<div><font face="arial narrow, sans-serif"><b><a
moz-do-not-send="true"
href="http://unixoperator.blogspot.com" target="_blank">http://unixoperator.blogspot.com</a></b></font></div>
<div><font face="arial narrow, sans-serif"><b><a
moz-do-not-send="true"
href="https://wikis.nyu.edu/display/%7Esm4082"
target="_blank">https://wikis.nyu.edu/display/~sm4082</a></b></font></div>
</div>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<br>
</body>
</html>