[lustre-discuss] Data stored in OST

[Peter Grandi]

>>> The usual practice is to use RAUD10 for the MDT(s) on
>>> "enterprise" high-endurance SSD, and RAID6 for the OST on
>>> "professional" mixed-load SSDs or "small" (1-2TB at most)
>>> "datacenter" HDDs, fronted by failover-servers.

> What would be the problem with large 'datacenter' type HDD's
> for an OST (in raid10 for instance)?

Very, very low IOPS-per-TB, leading to terrifyingly low speed
under combined user and maintenance load. Consider for example
18TB HDDs capable of multistream transfer rates of around 2-3MBs
each. I have seen some such setup where there were not enough
IOPS for the maintenance load (scrubbing and resilvering,
checking, migrating, etc.), never mind for the user load.
Especially if there is a non trivial percentage of "small" (less
than several MB) files.

https://www.sabi.co.uk/blog/13-two.html?131227#131227
"The issue with disk drives with multi-TB capacities"

That applies to every filesystem, but even more so to Lustre
which is mostly targeted at highly parallel HPC user loads.