[Lustre-discuss] Copying data from one Lustre system to an other lustre system

[Daire Byrne]

Mertol,

You probably want to check out the "e2scan" work that is being done atm. There's a bugzilla entry for it (NERSC). As this was not available to us a couple of months ago we came up with a way to snapshot both MDSs and then run "rsync --itemize-changes" between them. This builds a filelist that we can then split into deletes and copies. For the copies you can split it N times and use multiple "rsync --files-from=" instance across many clients.

The filelist generation takes around 6-8 hours and the copies around 5 hours for our 72Tb filesystem. Fast enough for us to get a backup through every 24 hours. But like I said a better/faster longterm solution is to use e2scan.

Daire


----- "Mertol Ozyoney" <Mertol.Ozyoney at Sun.COM> wrote:
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss