[Lustre-discuss] Limits for o2ib lnet network numbers

[Rick Mohr]

On Wed, 2012-08-08 at 22:38 +0800, Liang Zhen wrote:
> LNet reserved 32 bits for network number, so you can choose a very
> large network number if only have a few networks

Thanks.  That was exactly what I was looking for.

>  but really create many networks will have some issues:
> - o2iblnd will pre-allocate memory resources for each network, so it will consume a lot of memory

I am mainly looking at using network numbers to keep things better
organized.  For example, instead of having two different clusters each
use o2ib1 for their internal networks, I could assign o2ib101 to one
cluster and o2ib102 to the second (reserving o2ib[0-100] for other
purposes).

Any given client would probably only know about a few networks (maybe
2-3), but the lustre servers would obviously need to have more (maybe
10-20).  Is there an estimate on how much memory is consumed for each
network?  Also, if a node has o2ib0 and o2ib5 configured, will it just
allocate memory for those networks, or will it also allocate memory for
o2ib[1-4] even if they are unused?  (I wouldn't expect it to allocate
memory, but better to find it out now than discover my mistake later.)

> - Main stream LNet will have performance issue if there're many
> networks, for example, hundreds, although it's not difficult to fix
> this.

I don't expect to have hundreds of networks, but I am curious how I
would fix it if I ever did.

Thanks.

-- 
Rick Mohr
HPC Systems Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu/