[lustre-discuss] Mounting Lustre over IB-to-Ethernet gateway

[Martin Hecht]

Hi Kevin,

I think your proposed lnet config line is correct and it would add tcp0.
If you add a new lnet on the servers you have to reload the lnet module,
which implies that you have to restart lustre (you don't have to reboot
if unloading the modules works smoothly, i.e. unmounting all targets,
followed by lustre_rmmod, and then mounting targets again, you don't
have to restart the ib clients though).

If you have clients with an interface on both networks (which could act
as lnet routers) you can do without restarting the servers. You don't
have to add the lnet on the servers in that case, but you just have to
add the routes to the new lnet on all servers which works in production
with lctl --net tcp0 add_route client-ip at o2ib0. On the routers you need
forwarding="enabled" and they need both lnets, each of them assigned to
the appropriate interface (in order to configure this you have to reload
the lnet module on the clients which will act as routers). On the tcp
clients you would need the route across the routers in the opposite
direction. However, in that scenario you wouldn't use the ib2eth gateway.

Greetings,
Martin

On 08/01/2016 01:05 PM, Kevin M. Hildebrand wrote:
> Our Lustre filesystem is currently set up to use the o2ib interface only-
> all of the servers have
> options lnet networks=o2ib0(ib0)
>
> We've just added a Mellanox IB-to-Ethernet gateway and would like to be
> able to have clients on the Ethernet side also mount Lustre.  The gateway
> extends the same layer-2 IP range that's being used for IPoIB out to the
> Ethernet clients
>
> How should I go about doing this?  Since the clients don't have IB, it
> doesn't appear that I can use o2ib0 to mount.  Do I need to add another
> lnet network on the servers?  Something like
> options lnet networks=o2ib0(ib0),tcp0(ib0)?  Can I have both protocols on
> the same interface?
> And if I do have to add another lnet network, is there any way to do so
> without restarting the servers?
>
> Thanks,
> Kevin
>
> --
> Kevin Hildebrand
> University of Maryland, College Park
> Division of IT
>
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20160802/826f49b6/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2252 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20160802/826f49b6/attachment.bin>