[Lustre-devel] faking LNET scale
Zhen.Liang at Sun.COM
Fri Apr 17 11:33:30 PDT 2009
It's very late night for me now, my head is not clear enough for me to
make sure whether I'm saying something crazy, :)
LNet always thinks target is remote network(needs router) if it can't
find a NI with same network ID, for example, if local NI is (ptl0) and
caller wants to send message to (ptl1), then LNet will:
1. Try to find local NI for ptl1, and failed then:
2. try to find if ptl1 is a remote network and whether there is router
for this network (ptl1)
So if you want your server has only one NI instance and can talk with a
set of different networks, and at the same time, it can talk with other
remote networks via routers, I would suggest:
1. create a new command, for example: lctl add_local_net ptl0 ptl[1-N],
which means LNet should allow NI(ptl0) accessing networks( ptl[1-N] as
2. add a new structure in LNet, i.e:
struct list_head ln_list;
As you see, it's very like current structure lnet_remotenet_t, which is
pending on lnet_t::ln_remote_nets; we can create a lnet_locallnet_t
object and add it to global list (i.e: lnet_t::ln_local_nets) by the
command we mentioned above: lctl add_local_net
3. once upper layer caller sending message, lnet_send() should check
lnet_t::ln_local_nets firstly (before thinking it's a remote network and
checking on lnet_t::ln_remote_nets), if it is on
lnet_t::ln_local_netsthen we can take the local NI. on
4. We need add a new flag for LND, only LND with the flag can support
command lctl add_local_net.
5. make the LND wouldn't reject messages from different networks.
again, hope I'm answering what you are asking, :)
Nicholas Henke wrote:
> Greetings -
> I was looking into ways to simulate scale at the LNET level. It would allow us
> to test the LNDs better with less hardware, not to mention things like LNet
> SelfTest and friends.
> With the work in bug 15332 to add multiple nets per NIC, it seemed fairly close
> that we could use that to generate multiple LND connections from a single NIC.
> Ideally we'd have a server or router that would have just one LND instance
> (ptl0) and the client nodes with multiple interfaces (ptl1, ptl2, ...). This
> would increase the load on those server nodes to something interesting.
> However, to do this either hacking up lnet_ptlcompat_matchXXX to look at another
> flag besides the_lnet.ln_ptlcompat or some other way of allowing a server with a
> single NET (ptl0) to accept requests from a variety of nets (ptl1, ptl2, etc).
> One cannot use multiple interfaces for the same net type with ln_ptlcompat enabled.
> Is there a better way to do this ? What would be the least abusive of th e rules ?
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
More information about the lustre-devel