[Lustre-discuss] New lustre 1.8.5 over IB problem

Gary Molenkamp gary at sharcnet.ca
Mon Dec 20 06:36:11 PST 2010


                                                           ^^^^^
>> Why are you assigning two different network types to the same physical
>> device?
> 
> Hello Colin,
> 
> Thanks for the reply. In answer to your question:
> 
> The same physical device has access to two different lustre filesystems 
> using different protocols.
> 
> One lustre filesystem is locally available via the native ib interface 
> o2ib0(ib0).
> 
> The other lustre filesystem is remotely available (via a IB to 10Gb 
> switch/gateway in the local IB fabric) on the same local IB device but 
> only via the tcp/ip (IPoIB) protocol, tcp0(ib0).
> 
> (not sure how good this ASCII diagram will look)
> 
>  				     ---------------------
>      |-------------|	|------------| local lustre setup|
> ib0 |		-----------	     ---------------------
> --------	|ib fabric|
> |client|	-----------
> --------	 |
>  		--------------
>  		|ib to 10Gb gw|
>  		--------------
>  		 |		   eth0	--------------------
>  		 |---------------------| remote lustre setup|
>  					--------------------
> 
> Is this possible?
> 
> -k


I did manage to get this to work properly under the following conditions:
   remote lustre setup uses tcp(eth0)
   local lustre setup uses  o2ib(ib0)

   on the ib client
     lnet o2ib(ib0),tcp(ib0)

With this configuration,  all lustre servers are active and reachable.
If the client ordering is reversed, then the OSSs on the local lustre
always reports as temporarily unreachable.



-- 
Gary Molenkamp			SHARCNET
Systems Administrator		University of Western Ontario
Compute/Calcul Canada		http://www.computecanada.org
gary at sharcnet.ca		http://www.sharcnet.ca
(519) 661-2111 x88429		(519) 661-4000



More information about the lustre-discuss mailing list