[lustre-discuss] Mixed size OST's

[Steve Thompson]

Lustre newbie here (1 month). Lustre 2.10.3, CentOS 7.4, ZFS 0.7.5. All 
networking is 10 GbE.

I am building a test Lustre filesystem. So far, I have two OSS's, each 
with 30 disks of 2 TB each, all in a single zpool per OSS. Everything 
works well, and was suprisingly easy to build. Thus, two OST's of 60 TB 
each. File types are comprised of home directories. Clients number about 
225 HPC systems (about 2400 cores).

In about a month, I will have a third OSS available, and about a month 
after that, a fourth. Each of these two systems has 48 disks of 4 TB each. 
I am looking for advice on how best to configure this. If I go with one 
OST per system (one zpool comprising 8 x 6 RAIDZ2 vdevs), I will have a 
lustre f/s comprised of two 60 TB OST's and two 192 TB OST's (minus RAIDZ2 
overhead). This is obviously a big mismatch between OST sizes. I have not 
encountered any discussion of the effect of mixing disparate OST sizes. I 
could instead format two 96 TB OST's on each system (two zpools of 4 x 6 
RAIDZ2 vdevs), or three 64 TB OST's, and so on. More OST's means more 
striping possibilities, but less vdev's per zpool impacts ZFS performance 
negatively. More OST's per OSS does not help with network bandwidth to the 
OSS. How would you go about this?

TIA,
Steve