[lustre-discuss] Does Lustre support RoCE?

[Indivar Nair]

Thanks a lot, Michael, Andreas, Simon, Doug,
I have already installed MLNX OFED 4:-(
I will now have to undo it and install the earlier version.

Roughly, by when would the support for MLNX OFED 4 be available?

Regards,


Indivar Nair

On Thu, May 11, 2017 at 9:35 PM, Oucharek, Doug S <doug.s.oucharek at intel.com
> wrote:

> The note regarding MOFED 4 not supported by Lustre: I’m working on it.
> MOFED 4 did not drop support of Lustre, but did make API/behaviour changes
> which Lustre has not fully adapted to yet.  The ball is in the Lustre
> community’s court on this one now.
>
> Doug
>
> On May 11, 2017, at 8:47 AM, Simon Guilbault <
> simon.guilbault at calculquebec.ca> wrote:
>
> Hi, your lnet.conf look fine, I tested lnet with RoCE V2 a while back with
> a pair of server using Connectx4 with a single 25Gb interface and RDMA was
> working with Centos 7.3, stock RHEL OFED and Lustre 2.9. The only settings
> that I had to use in lustre's config was this one:
>
> options lnet networks=o2ib(ens2)
>
> The performance was about the same (1.9GB/s) without any tuning with the
> lnet self-test but the CPU utilisation was a lot lower with RDMA than TCP
> (3% vs 65% of a core).
>
> From my notes I took back then Lustre needed to be recompiled with MLNX
> OFED 3.4 and MLNX OFED 4 dropped support of Lustre accordings to their
> release notes.
>
> Ref 965588
> https://www.mellanox.com/related-docs/prod_software/
> Mellanox_OFED_Linux_Release_Notes_4_0-2_0_0_1.pdf
> https://www.mellanox.com/related-docs/prod_software/
> Mellanox_OFED_Linux_Release_Notes_4_0-2_0_2_0.pdf
>
>
> On Thu, May 11, 2017 at 11:34 AM, Indivar Nair <indivar.nair at techterra.in>
>  wrote:
>
>> So I should add something like this in lnet.conf -
>>
>> options lnet networks=o2ib0(p4p1)
>>
>> Thats it, right?
>>
>> Regards,
>>
>>
>> Indivar Nair
>>
>> On Thu, May 11, 2017 at 8:39 PM, Dilger, Andreas <andreas.dilger at intel.
>> com> wrote:
>>
>>> If you have RoCE cards and configure them with OFED, and configure
>>> Lustre to use o2iblnd then it should use RDMA for those interfaces. The
>>> fact that they are RoCE cards is hidden below OFED.
>>>
>>> Cheers, Andreas
>>>
>>> > On May 11, 2017, at 08:36, Indivar Nair <indivar.nair at techterra.in>
>>> wrote:
>>> >
>>> > Hi ...,
>>> >
>>> > I have read in different forums and blogs that Lustre supports RoCE.
>>> > But I cant find any documentation on it.
>>> >
>>> > I have a Lustre setup with 6 OSS and 2 SMB/NFS Gateways.
>>> > They are all interconnected using Mellanox SN2700 100G Switch and
>>> Mellanox Connect-X4 100G NICs.
>>> > I have installed the Mellanox OFED Drivers, but I cant find a way to
>>> tell Lustre / LNET to use RoCE.
>>> >
>>> > How do I go about?
>>> >
>>> > Regards,
>>> >
>>> >
>>> > Indivar Nair
>>> >
>>> >
>>> > _______________________________________________
>>> > lustre-discuss mailing list
>>> > lustre-discuss at lists.lustre.org
>>> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>
>>
>>
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170511/22969439/attachment.htm>