[lustre-discuss] Why certain commands should not be used on Lustre file system
sdm900 at gmail.com
Tue Feb 9 23:50:09 PST 2016
We actually use
find <dir> -type f -print0 | xargs -n 100 -P 32 -0 -- rm -f
which will parallelise the rm... which runs a fair bit faster.
On Wed, Feb 10, 2016 at 3:33 PM, <Prakrati.Agrawal at shell.com> wrote:
> Then rm -rf * should not be used in any kind of file system. Why only Lustre file system' best practices have this as a pointer.
> Thanks and Regards,
> -----Original Message-----
> From: Dilger, Andreas [mailto:andreas.dilger at intel.com]
> Sent: Wednesday, February 10, 2016 11:22 AM
> To: Agrawal, Prakrati PTIN-PTT/ICOE; lustre-discuss at lists.lustre.org
> Subject: Re: [lustre-discuss] Why certain commands should not be used on Lustre file system
> On 2016/02/09, 21:16, "lustre-discuss on behalf of Prakrati.Agrawal at shell.com<mailto:Prakrati.Agrawal at shell.com>" <lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org> on behalf of Prakrati.Agrawal at shell.com<mailto:Prakrati.Agrawal at shell.com>> wrote:
> I read on Lustre best practices that ls -U should be used instead of ls -l . I understand that ls -l makes MDS contact all OSS to get all information about all files and hence loads it. But, what does ls -U do to avoid it?
> -U do not sort; list entries in directory order
> This is more important for very large directories, since "ls" will read all of the entries and stat them before printing anything. That said, GNU ls will still read all of the entries before printing them, so for very large directories "find <directory> -ls" is a lot faster to start printing entries.
> Also, it is said that rm-rf * should not be used. Please can someone explain the reason for that.
> It is also said that instead lfs find <directory path> --type f -print0 | xargs -0 rm -f should be used. Please explain the reason for this also.
> "rm -rf *" will expand "*" onto the command line (done by bash) and if there are too many files in the directory (more than about 8MB IIRC) then bash will fail to execute the command. Running "lfs find" (or just plain "find") will only print the filenames onto the output and xargs will process them in chunks that fit onto a command-line.
> Cheers, Andreas
> Andreas Dilger
> Lustre Principal Architect
> Intel High Performance Data Division
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
Dr Stuart Midgley
sdm900 at sdm900.com
More information about the lustre-discuss