[lustre-discuss] Lustre and OFED

[Arman Khalatyan]

To find the RDMA devices you can use:
ibv_devices

You can bench your RDMA connection using qperf:
yum install qperf -y

on the client machine: qperf
on the server or ost  machine: qperf clienthostname  ud_lat ud_bw


On Mon, Jul 31, 2017 at 4:08 PM, Ben Evans <bevans at cray.com> wrote:

>
>
> From: "E.S. Rosenberg" <esr+lustre at mail.hebrew.edu>
> Date: Sunday, July 30, 2017 at 7:53 AM
> To: Ben Evans <bevans at cray.com>
> Cc: Harald van Pee <pee at hiskp.uni-bonn.de>, "lustre-discuss at lists.lustre.
> org" <lustre-discuss at lists.lustre.org>
> Subject: Re: [lustre-discuss] Lustre and OFED
>
>
>
> On Fri, Jul 28, 2017 at 6:22 PM, Ben Evans <bevans at cray.com> wrote:
>
>>
>>
>> On 7/28/17, 11:12 AM, "Harald van Pee" <pee at hiskp.uni-bonn.de> wrote:
>>
>> >Hello
>> >
>> >On Friday 28 July 2017 15:48:12 Ben Evans wrote:
>> >> Eli, just to clarify are you talking about using the in-kernel OFED vs.
>> >>a
>> >> vendor (Mellanox) OFED, or
>> >
>> >In our case we are using the OFED of the debian distribution used.
>>
> I am using the IB support that ships with Debian/CentOS/mainline kernel
> and did not install any OFED/Mellanox OFED package.
> As far as I can tell RDMA does work. (using the various test tools
> suggested here https://community.mellanox.com/docs/DOC-2086)
>
>> >
>> >> are you talking about using the ConnectX-3
>> >> hardware in IPoIB mode and just using it as a faster Ethernet?
>> >
>> >is  possible? How one have to do this?
>>
>> You'd configure the lustre LNET to use it like any other ethernet device.
>> The downside of this is that it's slower due to a lack of RDMA and other
>> features that IB has.  I'm not sure if there's a real upside to it.
>>
>
> Now you've made me unsure of what whether or not my Lustre install is
> using RDMA, how should I be able to tell (we are definitely using
> IPoIB/o2ib)?
>
> If you are mounting Lustre with a string that looks like 192.168.1.10 at o2ib
> ,192.168.0.11 at o2ib:/lustre then you're using OFED and RDMA.
>
> Thanks,
> Eli
>
>>
>> >
>> >Harald
>> >
>> >
>> >>
>> >> -Ben Evans
>> >>
>> >> From: lustre-discuss
>> >>
>> >><lustre-discuss-bounces at lists.lustre.org<mailto:lustre-
>> discuss-bounces at li
>> >>s
>> >> ts.lustre.org>> on behalf of "E.S. Rosenberg"
>> >> <esr+lustre at mail.hebrew.edu<mailto:esr+lustre at mail.hebrew.edu>> Date:
>> >> Thursday, July 27, 2017 at 4:55 PM
>> >> To:
>> >>
>> >>"lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org
>> >"
>> >>
>> >><lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org
>> >>
>> >> Subject: [lustre-discuss] Lustre and OFED
>> >>
>> >> Hi all,
>> >>
>> >> How 'needed' is OFED for Lustre? In the LUG talks it is mentioned every
>> >> once in a while and that got me thinking a bit.
>> >>
>> >> What things are gained by installing OFED? Performance? Accurate
>> traffic
>> >> reports?
>> >>
>> >> Currently I am using a lustre system without OFED but our IB hardware
>> is
>> >> from the FDR generation so not bleeding edge and probably doesn't need
>> >> OFED because of that....
>> >>
>> >> Thanks,
>> >> Eli
>> >>
>> >> Tech specs:
>> >> Servers: CentOS 6.8 + Lustre 2.8 (kernel from Lustre RPMs)
>> >> Clients: Debian + kernel 4.2 + Lustre 2.8
>> >> IB: ConnectX-3 FDR
>> >
>>
>>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170731/b084cd6c/attachment-0001.htm>