[lustre-discuss] Lustre and OFED

[Ben Evans]

From: "E.S. Rosenberg" <esr+lustre at mail.hebrew.edu<mailto:esr+lustre at mail.hebrew.edu>>
Date: Sunday, July 30, 2017 at 7:53 AM
To: Ben Evans <bevans at cray.com<mailto:bevans at cray.com>>
Cc: Harald van Pee <pee at hiskp.uni-bonn.de<mailto:pee at hiskp.uni-bonn.de>>, "lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>" <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>>
Subject: Re: [lustre-discuss] Lustre and OFED



On Fri, Jul 28, 2017 at 6:22 PM, Ben Evans <bevans at cray.com<mailto:bevans at cray.com>> wrote:


On 7/28/17, 11:12 AM, "Harald van Pee" <pee at hiskp.uni-bonn.de<mailto:pee at hiskp.uni-bonn.de>> wrote:

>Hello
>
>On Friday 28 July 2017 15:48:12 Ben Evans wrote:
>> Eli, just to clarify are you talking about using the in-kernel OFED vs.
>>a
>> vendor (Mellanox) OFED, or
>
>In our case we are using the OFED of the debian distribution used.
I am using the IB support that ships with Debian/CentOS/mainline kernel and did not install any OFED/Mellanox OFED package.
As far as I can tell RDMA does work. (using the various test tools suggested here https://community.mellanox.com/docs/DOC-2086)
>
>> are you talking about using the ConnectX-3
>> hardware in IPoIB mode and just using it as a faster Ethernet?
>
>is  possible? How one have to do this?

You'd configure the lustre LNET to use it like any other ethernet device.
The downside of this is that it's slower due to a lack of RDMA and other
features that IB has.  I'm not sure if there's a real upside to it.

Now you've made me unsure of what whether or not my Lustre install is using RDMA, how should I be able to tell (we are definitely using IPoIB/o2ib)?

If you are mounting Lustre with a string that looks like 192.168.1.10 at o2ib,192.168.0.11 at o2ib:/lustre then you're using OFED and RDMA.

Thanks,
Eli

>
>Harald
>
>
>>
>> -Ben Evans
>>
>> From: lustre-discuss
>>
>><lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org><mailto:lustre-discuss-bounces at li<mailto:lustre-discuss-bounces at li>
>>s
>> ts.lustre.org<http://ts.lustre.org>>> on behalf of "E.S. Rosenberg"
>> <esr+lustre at mail.hebrew.edu<mailto:esr%2Blustre at mail.hebrew.edu><mailto:esr+lustre at mail.hebrew.edu<mailto:esr%2Blustre at mail.hebrew.edu>>> Date:
>> Thursday, July 27, 2017 at 4:55 PM
>> To:
>>
>>"lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org><mailto:lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>>"
>>
>><lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org><mailto:lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>>>
>> Subject: [lustre-discuss] Lustre and OFED
>>
>> Hi all,
>>
>> How 'needed' is OFED for Lustre? In the LUG talks it is mentioned every
>> once in a while and that got me thinking a bit.
>>
>> What things are gained by installing OFED? Performance? Accurate traffic
>> reports?
>>
>> Currently I am using a lustre system without OFED but our IB hardware is
>> from the FDR generation so not bleeding edge and probably doesn't need
>> OFED because of that....
>>
>> Thanks,
>> Eli
>>
>> Tech specs:
>> Servers: CentOS 6.8 + Lustre 2.8 (kernel from Lustre RPMs)
>> Clients: Debian + kernel 4.2 + Lustre 2.8
>> IB: ConnectX-3 FDR
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170731/7829aa9a/attachment.htm>