[lustre-discuss] ZFS-OST layout, number of OSTs

[Thomas Roth]

Hi all,

I have done some "fio" benchmarking, amongst other things to test the proposition that to get more iops, the number of disks per raidz should be less. 
I was happy I could reproduce that: one server with 30 disks in one raidz2 (=one zpool = one OST) is indeed slower than one with 30 disks in three 
raidz2 (one zpool, one OST).
I ran fio also on a third server were the 30 disks make up 3 raidz2 = 3 zpools = 3 OSTs, that one is faster still.

Now I seem to remember a warning not to have too many OSTs in one Lustre, because each OST eats some memory on the client. I haven't found that 
reference, and I would like to ask what the critical numbers might be? How much RAM are we talking about? Is there any other "wise" limit on the OST 
number?
Currently our clients are equipped with 128 or 256 GB RAM.  We have 550 OSTs in the system, but the next cluster could easily grow much larger here if 
we stick to the small OSTs.

Regards,
Thomas