[Lustre-discuss] Mount 2 clusters, different networks - LNET tcp1-tcp2-o2ib

[Michael Shuey]

That may be because your gateway doesn't have an interface on tcp (aka
tcp0).  I suspect you want to keep your ethernet clients in tcp0, your
IB clients in o2ib0, and your router in both.  Personally, I find it
easiest to just give different module options on each system (rather
than try ip2nets stuff).

On the ether clients, I'd try:

options lnet networks=tcp0(eth0) routes="o2ib0
LNET-router-eth_IP at tcp0" dead_router_check_interval=300

On IB clients:

options lnet networks=o2ib0(ib0) routes="tcp0 LNET-router-IB_IP at ib0"
dead_router_check_interval=300

then on the router:

options lnet networks=tcp0(eth0),o2ib0(ib0) forwarding=enabled accept_timeout=15

Obviously, your file servers will need to have lnet options similar to
the clients:

options lnet networks=tcp0(eth0) routes="o2ib0
LNET-router-eth_IP at tcp0" dead_router_check_interval=300
options lnet networks=o2ib0(ib0) routes="tcp0 LNET-router-IB_IP at o2ib0"
dead_router_check_interval=300

That's just a guess, your mileage may vary, etc., but I think it's
close to what you want.  Note that you really want the
dead_router_check_interval if you're using lnet routers.  Without that
parameter, the lustre client will automatically mark a router as
failed when it's unavailable but will not check to see if it ever
comes back.  With this param, it checks every 300 seconds (and
re-enables it if found).

Hope this helps.

--
Mike Shuey



On Tue, Jun 14, 2011 at 1:26 PM, Thomas Roth <t.roth at gsi.de> wrote:
> Hm, the ethernet FS is in tcp0 - MGS says its nids are MGS-IP at tcp.
> So not surprising it refuses that connection.
> On the other hand,
>> options lnet networks=tcp1(eth0),tcp(eth0:0) routes="o2ib
>> LNET-Router-IP at tcp1; tcp Default-Gateway-IP at tcp"
>
> results in
>> Can't create route to tcp via Gateway-IP at tcp
>
> Cheers,
> Thomas
>
>
> On 06/14/2011 07:00 PM, Michael Shuey wrote:
>>
>> Is your ethernet FS in tcp1, or tcp0? Your config bits indicate the
>> client is in tcp1 - do the servers agree?
>>
>> --
>> Mike Shuey
>>
>>
>>
>> On Tue, Jun 14, 2011 at 12:23 PM, Thomas Roth <t.roth at gsi.de> wrote:
>>  > Hi all,
>>  >
>>  > I'd like to mount two Lustre filesystems on one client. Issues with
>> more than one MGS set aside,
>>  > the point here is that one of them is an Infiniband-cluster, the other
>> is ethernet-based.
>>  > And my client is on the ethernet.
>>  > I have managed to mount the o2ib-fs by setting up an LNET router, but
>> now this client's LNET doesn;t
>>  > known how to reach the ethernet-fs.
>>  >
>>  > So the basic modprobe.conf reads
>>  > > options lnet networks=tcp1(eth0) routes="o2ib LNET-Router-IP at tcp1"
>>  > This mounts the MGS on the o2ib network.
>>  >
>>  > What do I have to add to get to the MGS on the tpc network?
>>  >
>>  > Meanwhile I have studied more posts here and came up with
>>  > > options lnet networks=tcp1(eth0),tcp2(eth0:0) routes="o2ib
>> LNET-Router-IP at tcp1; tcp
>>  > Default-Gateway-IP at tcp2"
>>  >
>>  > Doesn't work either, but I see in the log of the (tcp-)MGS:
>>  > > LustreError: 120-3: Refusing connection from Client-IP for
>> MGS-IP at tcp2: No matching NI
>>  >
>>  > Somethings getting through ...
>>  >
>>  > Any ideas?
>>  >
>>  > Regards,
>>  > Thomas
>>  > _______________________________________________
>>  > Lustre-discuss mailing list
>>  > Lustre-discuss at lists.lustre.org
>>  > http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>  >
>>
>
>
>