[lustre-discuss] Lustre optimize for spares data files ?

[Andreas Dilger]

On Sep 8, 2020, at 9:13 PM, Tung-Han Hsieh <thhsieh at twcp1.phys.ntu.edu.tw> wrote:
> 
> I would like to ask whether Lustre file system has implemented the
> function to optimize for large sparse data files ?
> 
> For example, a 3GB data file but with more than 80% bytes zero, can
> Lustre file system optimize the storage not actually taking the whole
> 3GB of disk space ?

Could you please explain your usage further?  Lustre definitely has
support for sparse files - if they are written by an application with
"seek" or by multiple threads in parallel, then only the blocks that
are written will use space on the OST.

For ldiskfs the block size is 4KB.  For ZFS the OST block size is up
to 1MB, if the file size is 1MB or larger.  That is why compression
on ZFS can help reduce space usage on the OST, because it can effectively
compress the 1MB blocks that are nearly full of zeroes, if your sparse
writes are smaller than the blocksize.

If you are *copying* a sparse file, that depends on the tool that is
doing the copy.  For example, "cp --sparse=always" will generate a
sparse file.  We are also working on adding SEEK_HOLE and SEEK_DATA,
which will help tools to copy sparse files.

Cheers, Andreas

> I know that some file systems (e.g., ZFS) has this function. If Lustre
> does not have it, is there a roadmap to implement it in the future ?
> 
> Thanks for your reply in advance.
> 
> Best Regards,
> 
> T.H.Hsieh
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


Cheers, Andreas





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 873 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200909/e140cc98/attachment.sig>