[lustre-discuss] Why certain commands should not be used on Lustre file system

[Dilger, Andreas]

On 2016/02/10, 00:33, "Prakrati.Agrawal at shell.com"
<Prakrati.Agrawal at shell.com> wrote:

>Hi,
>
>Then rm -rf * should not be used in any kind of file system. Why only
>Lustre file system' best practices have this as a pointer.

Because Lustre is more likely to have tens of thousands or millions of
files in a single directory than most local filesystems.  This same
problem exists for local filesystems as well, for example:

http://www.cyberciti.biz/faq/argument-list-too-long-error-solution/
http://stackoverflow.com/questions/11475221/why-do-i-get-bin-sh-argument-li
st-too-long-when-passing-quoted-arguments

Cheers, Andreas


>-----Original Message-----
>From: Dilger, Andreas [mailto:andreas.dilger at intel.com]
>Sent: Wednesday, February 10, 2016 11:22 AM
>To: Agrawal, Prakrati PTIN-PTT/ICOE; lustre-discuss at lists.lustre.org
>Subject: Re: [lustre-discuss] Why certain commands should not be used on
>Lustre file system
>
>On 2016/02/09, 21:16, "lustre-discuss on behalf of
>Prakrati.Agrawal at shell.com<mailto:Prakrati.Agrawal at shell.com>"
><lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lis
>ts.lustre.org> on behalf of
>Prakrati.Agrawal at shell.com<mailto:Prakrati.Agrawal at shell.com>> wrote:
>
>I read on Lustre best practices that ls -U should be used instead of ls
>-l . I understand that ls -l makes MDS contact all OSS to get all
>information about all files and hence loads it. But, what does ls -U do
>to avoid it?
>
>       -U     do not sort; list entries in directory order
>
>This is more important for very large directories, since "ls" will read
>all of the entries and stat them before printing anything.  That said,
>GNU ls will still read all of the entries before printing them, so for
>very large directories "find <directory> -ls" is a lot faster to start
>printing entries.
>
>Also, it is said that rm-rf * should not be used. Please can someone
>explain the reason for that.
>
>It is also said that instead lfs find  <directory path> --type f -print0
>| xargs -0 rm -f should be used. Please explain the reason for this also.
>
>"rm -rf *" will expand "*" onto the command line (done by bash) and if
>there are too many files in the directory (more than about 8MB IIRC) then
>bash will fail to execute the command.  Running "lfs find" (or just plain
>"find") will only print the filenames onto the output and xargs will
>process them in chunks that fit onto a command-line.
>
>Cheers, Andreas
>--
>Andreas Dilger
>Lustre Principal Architect
>Intel High Performance Data Division
>
>_______________________________________________
>lustre-discuss mailing list
>lustre-discuss at lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>


Cheers, Andreas
-- 
Andreas Dilger

Lustre Principal Architect
Intel High Performance Data Division