[lustre-discuss] Does Lustre support RoCE?

Simon Guilbault simon.guilbault at calculquebec.ca
Thu May 11 08:47:54 PDT 2017


Hi, your lnet.conf look fine, I tested lnet with RoCE V2 a while back with
a pair of server using Connectx4 with a single 25Gb interface and RDMA was
working with Centos 7.3, stock RHEL OFED and Lustre 2.9. The only settings
that I had to use in lustre's config was this one:

options lnet networks=o2ib(ens2)

The performance was about the same (1.9GB/s) without any tuning with the
lnet self-test but the CPU utilisation was a lot lower with RDMA than TCP
(3% vs 65% of a core).

>From my notes I took back then Lustre needed to be recompiled with MLNX
OFED 3.4 and MLNX OFED 4 dropped support of Lustre accordings to their
release notes.

Ref 965588
https://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_0-2_0_0_1.pdf
https://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_0-2_0_2_0.pdf


On Thu, May 11, 2017 at 11:34 AM, Indivar Nair <indivar.nair at techterra.in>
wrote:

> So I should add something like this in lnet.conf -
>
> options lnet networks=o2ib0(p4p1)
>
> Thats it, right?
>
> Regards,
>
>
> Indivar Nair
>
> On Thu, May 11, 2017 at 8:39 PM, Dilger, Andreas <andreas.dilger at intel.com
> > wrote:
>
>> If you have RoCE cards and configure them with OFED, and configure Lustre
>> to use o2iblnd then it should use RDMA for those interfaces. The fact that
>> they are RoCE cards is hidden below OFED.
>>
>> Cheers, Andreas
>>
>> > On May 11, 2017, at 08:36, Indivar Nair <indivar.nair at techterra.in>
>> wrote:
>> >
>> > Hi ...,
>> >
>> > I have read in different forums and blogs that Lustre supports RoCE.
>> > But I cant find any documentation on it.
>> >
>> > I have a Lustre setup with 6 OSS and 2 SMB/NFS Gateways.
>> > They are all interconnected using Mellanox SN2700 100G Switch and
>> Mellanox Connect-X4 100G NICs.
>> > I have installed the Mellanox OFED Drivers, but I cant find a way to
>> tell Lustre / LNET to use RoCE.
>> >
>> > How do I go about?
>> >
>> > Regards,
>> >
>> >
>> > Indivar Nair
>> >
>> >
>> > _______________________________________________
>> > lustre-discuss mailing list
>> > lustre-discuss at lists.lustre.org
>> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170511/b6d8bc6d/attachment.htm>


More information about the lustre-discuss mailing list