[Lustre-discuss] Help with lustre routing

[Isaac Huang]

On Mon, Nov 09, 2009 at 10:55:34PM -0800, Eric Adint wrote:
> OK i have read the manual and i have read the boards and done as much  
> research as i can, but i cant seem to bend my head around this, what i  
> want to do is create a router so that i can keep my OST and MGS/MDT on  
> the IB network and have another computer handle the ib to ip routing.
> here is the configuration.
> 
> nas-0-0,  MGS/MDT , OST1 - o2ib 192.168.1.2, tcp0 10.1.254.3
> nas-0-1, OST2   		- o2ib 192.168.1.3, tcp0 10.1.254.4
> nas-0-2, OST3   		- o2ib 192.168.1.4, tcp0 10.1.254.5
> compute-0-0 		- o2ib 192.168.1.10, tcp0 10.1.254.10
> compute-0-1 		- o2ib 192.168.1.11, tcp0 10.1.254.11
> compute-0-*			-o2ib 192.168.1.*,tcp0 10.1.254.*
> compute-1-*			-tcp0 10.1.253.*

It appears that all nodes reside in the tcp0 network, and thus no
router would be needed for them to communicate with each other over
@tcp0 NIDs.

> What i think i need is
> in modprob.conf
> options lnet 'ip2nets="o2ib0(ib0) 192.168.*.*; tcp(eth0) 10.1.*.*"' \
> 'routes="tcp 192.168.1.[3-5]@o2ib0; o2ib 10.1.254.[10-11]@tcp0"

The 'routes' would be ignored by all node groups except 'compute-1-*'
because 'tcp' and 'o2ib' are local networks to them. The nodes in
'compute-1-*' would use routers 10.1.254.[10-11]@tcp0 to reach the
'o2ib' network, however servers in the 'o2ib' network would NOT try to
use any of the routers in order to talk to clients in 'tcp0' network
because they are already connected to the 'tcp0' network - this
probably is not what you wanted.

If the 10.1.254.* network is separate from the 10.1.253.* network, you
might consider assigning two 'tcp' networks to them.

Thanks,
Isaac