[lustre-discuss] Lustre optimize for spares data files ?

[Robert Redl]

Dear Tung-Han,

Am 09.09.20 um 09:49 schrieb Tung-Han Hsieh:
> Dear Robert,
>
> Thank you very much for your prompt reply.
>
> I have additional questions. Since we have a large Lustre file
> system pool, some are OST with ldiskfs backend, and some are OST
> with ZFS backend. Can I just enable compression on the ZFS backend
> which is going to be joined into the Lustre file system without
> affecting the remaining parts ?
Maybe someone else can answer that. Our system runs exclusively on ZFS
backends (OST+MDT). I think it is not a problem to mix backends, but I'm
not sure and I never tried.
>
> If my OST with ZFS has run for a while without compression enabled,
> could I enable it without hurting the existing data ?

Yes. If you set compression=on, then new data will be compressed (with
LZ4), old data remains unchanged. If you turn compression off again,
then new data will not be compressed anymore. But data already in the
system remains compressed. This is the normal ZFS behavior. Moving data
with lfs migrate to an OST with compression enabled will result in
compressed data.

Cheers,
Robert

>
> Thank you very much.
>
> T.H.Hsieh
>
>
> On Wed, Sep 09, 2020 at 08:32:30AM +0200, Robert Redl wrote:
>> Dear Tung-Han,
>>
>> you can use Lustre with ZFS backend and enabled compression. That has
>> the effect you are looking for and works very well.
>>
>> Cheers,
>> Robert
>>
>> Am 09.09.20 um 05:13 schrieb Tung-Han Hsieh:
>>> Dear All,
>>>
>>> I would like to ask whether Lustre file system has implemented the
>>> function to optimize for large sparse data files ?
>>>
>>> For example, a 3GB data file but with more than 80% bytes zero, can
>>> Lustre file system optimize the storage not actually taking the whole
>>> 3GB of disk space ?
>>>
>>> I know that some file systems (e.g., ZFS) has this function. If Lustre
>>> does not have it, is there a roadmap to implement it in the future ?
>>>
>>> Thanks for your reply in advance.
>>>
>>> Best Regards,
>>>
>>> T.H.Hsieh
>>> _______________________________________________
>>> lustre-discuss mailing list
>>> lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-- 

Dr. Robert Redl
Scientific Programmer, "Waves to Weather" (SFB/TRR165)
Meteorologisches Institut
Ludwig-Maximilians-Universität München
Theresienstr. 37, 80333 München, Germany
Tel.: +49 89 2180 4569

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200909/94ef39dd/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200909/94ef39dd/attachment-0001.sig>