[lustre-discuss] overly zealous Data-on-Metadata/

[Peter Grandi]

So because of external constraints I have configured a Lustre
instance as:

* MDT: ZFS with recordsize 8KiB, no compression.
* OSTs: ZFS with LZ4 compression.
* Default striping: first 64KiB as Data-on-Metadata.

The workload has a prevalence of either tiny files (less than
2KiB) or large files that are extremely compressible. That's
something that I am pushing hard to change, but for now tough.

Then I was surprised by how quickly the MDT filled up, so I
looked at it and I saw something unexpected: a lot of files with
logical length of over 1MiB and a physical size of less than
64KiB ('du --apparent' vs. 'du').

That means that the DoM code puts in the MDT first chunks whose
_physical size_ is less than 64KiB, that is it switches chunk
based on chunk physical size rather than logical length, which
is sort of understandable.

What is more surprising is that the DoM chunks seem compressed
on an uncompressed ZFS filesystem (I checked its properties).

The files don't look sparse, so I am curious. Is there any
explanation for this?