[Lustre-discuss] How efficient is 'lfs find'?

[Andreas Dilger]

On 2010-07-15, at 15:46, "Adesanya, Adeyemi" <yemi at slac.stanford.edu> wrote:
> We are working on coming up with a backup plan for our Lustre filesystem in case we ever lose an OST in the future. I like the idea of backing up the filesystem at the client level and then identifying what files were stored on a missing OST using 'lfs find'.

Note that there us a bug in 'lfs find' that prevents it working when an OST is down, because it sometimes tries to get the object sizes from he OSTs. However, it is possible to use 'lfs getstripe -q' to do the same thing. 

> There are also some useful 'lfs find' commands that may assist us in migrating/purging old data such as the use of the '--atime' and '--mtime' options. Can someone comment on the efficiency of 'lfs find'? Does it only query the MDT or will it also contact the OSTs?

It depends on your query. If you are asking only for MDS informaion (uid, gid, name, type, osts) it only queries the MDS.  If you ask for size it must query the OSTs. For timestamos it depends in the query. If the MDS timestamp is enough to decide then no OST RPC is sent (e.g. --mtime -1 and the MDS inode has an mtime newer than that), but this depends on the exact query and the inode timestamps. 

Note that you may want to look at e2scan, or ne2scan (modified version of e2scan) since it does much of thus already. Also of interest would be CEA's RobinHood policy manager. 

Cheers, Andreas