[Lustre-discuss] New lustre 1.8.5 over IB problem
Gary Molenkamp
gary at sharcnet.ca
Mon Dec 20 06:36:11 PST 2010
^^^^^
>> Why are you assigning two different network types to the same physical
>> device?
>
> Hello Colin,
>
> Thanks for the reply. In answer to your question:
>
> The same physical device has access to two different lustre filesystems
> using different protocols.
>
> One lustre filesystem is locally available via the native ib interface
> o2ib0(ib0).
>
> The other lustre filesystem is remotely available (via a IB to 10Gb
> switch/gateway in the local IB fabric) on the same local IB device but
> only via the tcp/ip (IPoIB) protocol, tcp0(ib0).
>
> (not sure how good this ASCII diagram will look)
>
> ---------------------
> |-------------| |------------| local lustre setup|
> ib0 | ----------- ---------------------
> -------- |ib fabric|
> |client| -----------
> -------- |
> --------------
> |ib to 10Gb gw|
> --------------
> | eth0 --------------------
> |---------------------| remote lustre setup|
> --------------------
>
> Is this possible?
>
> -k
I did manage to get this to work properly under the following conditions:
remote lustre setup uses tcp(eth0)
local lustre setup uses o2ib(ib0)
on the ib client
lnet o2ib(ib0),tcp(ib0)
With this configuration, all lustre servers are active and reachable.
If the client ordering is reversed, then the OSSs on the local lustre
always reports as temporarily unreachable.
--
Gary Molenkamp SHARCNET
Systems Administrator University of Western Ontario
Compute/Calcul Canada http://www.computecanada.org
gary at sharcnet.ca http://www.sharcnet.ca
(519) 661-2111 x88429 (519) 661-4000
More information about the lustre-discuss
mailing list