[lustre-discuss] rsync not appropriate for lustre

[Åke Sandgren]

I think this might be of some interest
https://github.com/hpc/mpifileutils

On 1/23/20 4:33 PM, Bernd Melchers wrote:
> Hi All,
> we are copying large data sets within our lustre filesystem and between
> lustre and an external nfs server. In both cases the performance is
> unexpected low and the reason seems that rsync is reading and writing in
> 32 kB Blocks, whereas our lustre would be more happy with 4 MB
> Blocks.
> rsync has an --block-size=SIZE Parameter but this adjusts only the
> checksum block size (and the maximum is 131072), not the i/o block size.
> Is there a solution to accelerate rsync? 
> 
> Mit freundlichen Grüßen
> Bernd Melchers
> 

-- 
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: ake at hpc2n.umu.se   Phone: +46 90 7866134 Fax: +46 90-580 14
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se