[Lustre-discuss] rsync --sparse actually work on RHEL5?

Peter Grandi pg_lus at lus.for.sabi.co.UK
Sun Oct 25 15:46:34 PDT 2009


> Given that I want to transplant to a new device on the same
> MDS, I can use rsync so that the EAs are transplanted in
> tandem with the files rather than as an after-thought with
> getfattr.

> Tar requires the more clunky getfattr afterwards because this
> is RHEL5 and RHEL5's tar doesn't appear to have EA support in
> it.

Schily TAR ('star') has extended attr (and ACL support) too if
'exustar' format is used, and seems available for RHEL5 (not by
coincidence I guess).

> But, I seem to be finding that rsync's --sparse option is not
> quite as good as tar's.  Witness:

> pwd
> /tmp/mdt1_snapshot/ROOT
> rsync --sparse brian/lustre/OFED-1.3.tgz /tmp/mdt1_new/ROOT/brian/lustre/
> ls -ls /tmp/mdt1_{snapshot,new}/ROOT/brian/lustre/OFED-1.3.tgz
> 12 -rw-r--r-- 1 root root 59918537 Oct 23 23:34 /tmp/mdt1_new/ROOT/brian/lustre/OFED-1.3.tgz
>  0 -rw-r--r-- 1 1001 1001 59918537 Mar 27  2008 /tmp/mdt1_snapshot/ROOT/brian/lustre/OFED-1.3.tgz

> As you can see, rsync allocated actual disk blocks for the
> copy of the file where it's clearly sparse on the source.

That '12' looks like 3 blocks, that is 2 metadata (indirect?)
blocks and 1 data block, therefore probably the last block.
I suspect that RSYNC refuses to write wholly empty files.

> Now tar doing the same operation:

> rm /tmp/mdt1_new/ROOT/brian/lustre/OFED-1.3.tgz
> tar -cSvf - brian/lustre/OFED-1.3.tgz | tar -xSvf - -C /tmp/mdt1_new/ROOT/
> brian/lustre/OFED-1.3.tgz
> brian/lustre/OFED-1.3.tgz
> ls -ls /tmp/mdt1_{snapshot,new}/ROOT/brian/lustre/OFED-1.3.tgz
> 0 -rw-r--r-- 1 1001 1001 59918537 Mar 27  2008 /tmp/mdt1_new/ROOT/brian/lustre/OFED-1.3.tgz
> 0 -rw-r--r-- 1 1001 1001 59918537 Mar 27  2008 /tmp/mdt1_snapshot/ROOT/brian/lustre/OFED-1.3.tgz

> Looks much better.  But I'd rather use rsync.  Is this rsync
> --sparse issue on RHEL5 known to anyone else?

My suspicion was that for some reason RSYNC writes at least one
byte to the file, at the end, and 'strace' confirms (on a JFS
filesystem with 4KiB minimum extents):

  $ rm -f /var/tmp/empty /var/tmp/empty2
  $ dd bs=1 count=0 if=/dev/zero seek=1048576 of=/var/tmp/empty
  0+0 records in
  0+0 records out
  0 bytes (0 B) copied, 1.2341e-05 seconds, 0.0 kB/s
  $ strace -f -e trace=file,write,_llseek rsync --no-whole-file -S /var/tmp/empty /var/tmp/empty2
  ....
  [pid 15619] _llseek(1, 1024, [1048576], SEEK_CUR) = 0
  [pid 15619] _llseek(1, 1, [1048577], SEEK_CUR) = 0
  [pid 15619] _llseek(1, -1, [1048576], SEEK_CUR) = 0
  [pid 15619] write(1, "\0", 1)           = 1
  [pid 15619] lstat64(".empty2.bYDneE", {st_mode=S_IFREG|0600, st_size=1048577, ...}) = 0
  [pid 15619] rename(".empty2.bYDneE", "empty2") = 0
  ....
  $ ls -s /var/tmp/empty{,2}
  0 /var/tmp/empty  4 /var/tmp/empty2

Note that in the above RSYNC explicitly seeks back to the
next-to-last byte ad writes it; the most likely reason is
portability, that is probably RSYNC does not assume that the
target system has got 'ftruncate' (while the 'dd' above uses
it). In Perl the same effect can be obtained with

  perl -e 'truncate(STDOUT,1048576);' >| /var/tmp/empty

There are a couple of other issues with RSYNC and sparse files:

  http://lists.samba.org/archive/rsync/2003-August/007000.html
  http://arighi.blogspot.com/2008/03/rsync-and-sparse-files.html
  http://lists.samba.org/archive/rsync/2008-February/020112.html

I regard these as largely unavoidable given its extreme
requirements for cross platform portability and the way it
operates. But it would be nice to add to it checking whether
'ftruncate' is available. Recent versions of RSYNC indeed
seem to imply it into the new "--preallocate" option.

BTW, this is unrelated, these are the options I recommend for
backups:

  rsync -axHAX [-u] --delete [--sparse|--inplace] [--partial] [-z] [-i]



More information about the lustre-discuss mailing list