[lustre-discuss] Why certain commands should not be used on Lustre file system

Dilger, Andreas andreas.dilger at intel.com
Thu Feb 11 03:41:56 PST 2016

On 2016/02/10, 00:33, "Prakrati.Agrawal at shell.com"
<Prakrati.Agrawal at shell.com> wrote:

>Then rm -rf * should not be used in any kind of file system. Why only
>Lustre file system' best practices have this as a pointer.

Because Lustre is more likely to have tens of thousands or millions of
files in a single directory than most local filesystems.  This same
problem exists for local filesystems as well, for example:


Cheers, Andreas

>-----Original Message-----
>From: Dilger, Andreas [mailto:andreas.dilger at intel.com]
>Sent: Wednesday, February 10, 2016 11:22 AM
>To: Agrawal, Prakrati PTIN-PTT/ICOE; lustre-discuss at lists.lustre.org
>Subject: Re: [lustre-discuss] Why certain commands should not be used on
>Lustre file system
>On 2016/02/09, 21:16, "lustre-discuss on behalf of
>Prakrati.Agrawal at shell.com<mailto:Prakrati.Agrawal at shell.com>"
><lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lis
>ts.lustre.org> on behalf of
>Prakrati.Agrawal at shell.com<mailto:Prakrati.Agrawal at shell.com>> wrote:
>I read on Lustre best practices that ls -U should be used instead of ls
>-l . I understand that ls -l makes MDS contact all OSS to get all
>information about all files and hence loads it. But, what does ls -U do
>to avoid it?
>       -U     do not sort; list entries in directory order
>This is more important for very large directories, since "ls" will read
>all of the entries and stat them before printing anything.  That said,
>GNU ls will still read all of the entries before printing them, so for
>very large directories "find <directory> -ls" is a lot faster to start
>printing entries.
>Also, it is said that rm-rf * should not be used. Please can someone
>explain the reason for that.
>It is also said that instead lfs find  <directory path> --type f -print0
>| xargs -0 rm -f should be used. Please explain the reason for this also.
>"rm -rf *" will expand "*" onto the command line (done by bash) and if
>there are too many files in the directory (more than about 8MB IIRC) then
>bash will fail to execute the command.  Running "lfs find" (or just plain
>"find") will only print the filenames onto the output and xargs will
>process them in chunks that fit onto a command-line.
>Cheers, Andreas
>Andreas Dilger
>Lustre Principal Architect
>Intel High Performance Data Division
>lustre-discuss mailing list
>lustre-discuss at lists.lustre.org

Cheers, Andreas
Andreas Dilger

Lustre Principal Architect
Intel High Performance Data Division

More information about the lustre-discuss mailing list