[lustre-discuss] [EXTERNAL] Re: Joining files

[Sven Willner]

Thank you all for the very useful advice! I will give these options a try.

best,
Sven

On Fri, Mar 31, 2023 at 08:36:22AM +0000, Gael.DELBARY at CEA.FR wrote:
>Hi,
>
>Yes squashfs is a pretty good alternative and rocks if you have a read-only workload on many tiny files. 2 advantages :
>1 -  like Andreas has said, for Lustre it is "light", only one file to "manage" (metadata load will be low)
>2 -  like you have said userspace tools already exist in common distro (easy to manage on client side)
>
>But I think ext4 loopback is a more versatile solution but you need to deal with "root privilege requirements" and find a solution to mount the loopback file on demand for yours clients.
>The big advantage with ext4 loopback is that you benefit of all ext4 debugging tools, which can be pretty useful on some use case (loopback file corruption for example). More you can also write inside the loopback image and it is pretty efficient in kernel post CentOS 7. The grail is the CCI feature but I don't know what is the status?
>
>Best regards.
>
>Gael  
>
>-----Message d'origine-----
>De : lustre-discuss [[1]] De la part de Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss Envoyé : Thursday, March 30, 2023 10:37 PM À : Andreas Dilger <adilger at whamcloud.com>; Sven Willner <sven.willner at mpimet.mpg.de> Cc : lustre-discuss at lists.lustre.org Objet : Re: [lustre-discuss] [EXTERNAL] Re: Joining files
>
>> Instead, my recommendation would be to use an ext4 filesystem image to hold the many small files (during create, if from a single client, or aggregated after they are created). Later, this filesystem image could be mounted read-only on multiple clients for access. Also, the whole image file can be archived to tape efficiently (taking all small files with it, instead of keeping a stub in Lustre for each file).
>>
>> The use of loopback mounting image files from Lustre already works 
>> today, but needs userspace help to create and mount/unmount them. 
>> There was some proposal "Client Container Image (CCI)" on how this 
>> could be integrated directly into Lustre. Please see my LUG 
>> presentation for details (maybe 2019 or so?)
>
>Would squashfs files be a good alternative to this?  The user space tools already exist.  We have a couple of workflows here that create a lot of small files and we are using squashfs to aggregate those files for the purposes of archival and to reduce the metadata burden on our lustre filesystem.  
>
>_______________________________________________
>lustre-discuss mailing list
>lustre-discuss at lists.lustre.org
>[2]
>
>
>----------------------
>
>Links:
>
>[1] mailto:lustre-discuss-bounces at lists.lustre.org
>[2] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

-- 
Dr. Sven Willner
Scientific Computing Lab (SCLab)
Max Planck Institute for Meteorology
Bundesstraße 53, D-20146 Hamburg, Germany
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5471 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20230331/b30bb286/attachment-0001.bin>