[lustre-discuss] limit on number of oss/ost's?

[Patrick Farrell]

The 160 limit has been raised.  I don't know what the new one is, but it is *quite* large.  I'm pretty sure it's beyond practical interest today.

There are a few issues with having extremely large numbers of OSTs, especially if you are explicitly trading off 1 vs many OSTs.

There are no particular scaling issues with number of OSTs of an OSS, so if you took the same storage and subdivided it to create more OSTs, there's no particular concern there.  But that assumes you're taking the same storage and deciding how to subdivide it - Obviously, a given amount of CPU/RAM/network on the OSS can only "feed" so much storage, so if you're just *adding* storage, you will quickly exhaust your OSS resources.  (Generally speaking one tries to match the two, so one does not have too much CPU/RAM/network bandwidth OR too much storage.)

The two main problems I see with "many" OSTs are:
1. They can get rather small, and so they can fill up relatively easily.  If your OSTs are really small and a few of the files assigned to that OST become large (so, they're assigned there when the OST is mostly empty, and then grow large), you'll run out of space on that OST and will no longer be able to write to files striped there.
2. As file stripe counts go up, the file layout - basically, the mapping from the byte range as seen in userspace to the actual objects on the OSTs - can become large enough that sending it around is a performance bottleneck.  Opening a single file with hundreds of stripes from thousands of clients - like a large supercomputer center might do - can take a significant period of time.

That second is the only scaling issue with OST *count* that I'm aware of, other than that there is a bit of memory overhead for tracking each OST - so 10 OSTs instead of 1 OST will use marginally more memory on servers and clients.  This is pretty small, though.

So in general, I would say you'd be happier with fewer & faster, rather than more & slower, especially when talking about very large OST counts.  There are some performance issues with multiple writers to single files with low stripe counts, so it doesn't hold in extremis.  This is all to say you'd be much better served with 10 OSTs than with *1*, but 100 is probably not a better idea than 10.

- Patrick

On 10/11/18, 1:07 PM, "lustre-discuss on behalf of Michael Di Domenico" <lustre-discuss-bounces at lists.lustre.org on behalf of mdidomenico4 at gmail.com> wrote:

    Is there a limit on the number of oss servers you can have in a single
    filesystem?  is there one for ost's?
    
    I'm curious of the performance implications between two different
    configurations (this is just theory mind you)...
    
    1000 oss with 1 ost each
    vs
    100 oss with 10 ost each
    
    one could scale this up further 2000, 5000, 10000 oss's with 1 single ost each
    
    i did note two references one from 2012 by Oleg Drokin that tested
    1300 OST's at ornl which "mostly worked" and a note from Andreas last
    year that quoted 2000 OST's before scaling issues.
    
    I'm curious if there's a fundamental issue with scaling lustre, which
    might be based on the presumption that oss's are typically fatter
    nodes (getting fatter everyday) rather than large quantities of skinny
    nodes
    
    I'm also curious if there is still a 160 OST limit for file striping as well.
    _______________________________________________
    lustre-discuss mailing list
    lustre-discuss at lists.lustre.org
    http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org