[lustre-discuss] [EXTERNAL] Re: Joining files

[Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]]

> Instead, my recommendation would be to use an ext4 filesystem image to hold the many small files (during create, if from a single client, or aggregated after they are created). Later, this filesystem image could be mounted read-only on multiple clients for access. Also, the whole image file can be archived to tape efficiently (taking all small files with it, instead of keeping a stub in Lustre for each file).
>
> The use of loopback mounting image files from Lustre already works today, but needs userspace help to create and mount/unmount them. There was some proposal "Client Container Image (CCI)" on how this could be integrated directly into Lustre. Please see my LUG presentation for details (maybe 2019 or so?)

Would squashfs files be a good alternative to this?  The user space tools already exist.  We have a couple of workflows here that create a lot of small files and we are using squashfs to aggregate those files for the purposes of archival and to reduce the metadata burden on our lustre filesystem.