<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 28, 2017 at 6:22 PM, Ben Evans <span dir="ltr"><<a href="mailto:bevans@cray.com" target="_blank">bevans@cray.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-"><br>
<br>
On 7/28/17, 11:12 AM, "Harald van Pee" <<a href="mailto:pee@hiskp.uni-bonn.de">pee@hiskp.uni-bonn.de</a>> wrote:<br>
<br>
>Hello<br>
><br>
>On Friday 28 July 2017 15:48:12 Ben Evans wrote:<br>
>> Eli, just to clarify are you talking about using the in-kernel OFED vs.<br>
>>a<br>
>> vendor (Mellanox) OFED, or<br>
><br>
>In our case we are using the OFED of the debian distribution used.<br></span></blockquote><div>I am using the IB support that ships with Debian/CentOS/mainline kernel and did not install any OFED/Mellanox OFED package.<br></div><div>As far as I can tell RDMA does work. (using the various test tools suggested here <a href="https://community.mellanox.com/docs/DOC-2086">https://community.mellanox.com/docs/DOC-2086</a>) <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">
><br>
>> are you talking about using the ConnectX-3<br>
>> hardware in IPoIB mode and just using it as a faster Ethernet?<br>
><br>
>is possible? How one have to do this?<br>
<br>
</span>You'd configure the lustre LNET to use it like any other ethernet device.<br>
The downside of this is that it's slower due to a lack of RDMA and other<br>
features that IB has. I'm not sure if there's a real upside to it.<br></blockquote><div><br></div><div>Now you've made me unsure of what whether or not my Lustre install is using RDMA, how should I be able to tell (we are definitely using IPoIB/o2ib)? <br></div><div>Thanks,<br></div><div>Eli<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="gmail-HOEnZb"><div class="gmail-h5"><br>
><br>
>Harald<br>
><br>
><br>
>><br>
>> -Ben Evans<br>
>><br>
>> From: lustre-discuss<br>
>><br>
>><<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@<wbr>lists.lustre.org</a><mailto:<a href="mailto:lustre-discuss-bounces@li">lustre<wbr>-discuss-bounces@li</a><br>
>>s<br>
>> <a href="http://ts.lustre.org" rel="noreferrer" target="_blank">ts.lustre.org</a>>> on behalf of "E.S. Rosenberg"<br>
>> <<a href="mailto:esr%2Blustre@mail.hebrew.edu">esr+lustre@mail.hebrew.edu</a><<wbr>mailto:<a href="mailto:esr%2Blustre@mail.hebrew.edu">esr+lustre@mail.hebrew.<wbr>edu</a>>> Date:<br>
>> Thursday, July 27, 2017 at 4:55 PM<br>
>> To:<br>
>><br>
>>"<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.<wbr>lustre.org</a><mailto:<a href="mailto:lustre-discuss@lists.lustre.org">lustre-<wbr>discuss@lists.lustre.org</a>>"<br>
>><br>
>><<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.<wbr>lustre.org</a><mailto:<a href="mailto:lustre-discuss@lists.lustre.org">lustre-<wbr>discuss@lists.lustre.org</a>>><br>
>> Subject: [lustre-discuss] Lustre and OFED<br>
>><br>
>> Hi all,<br>
>><br>
>> How 'needed' is OFED for Lustre? In the LUG talks it is mentioned every<br>
>> once in a while and that got me thinking a bit.<br>
>><br>
>> What things are gained by installing OFED? Performance? Accurate traffic<br>
>> reports?<br>
>><br>
>> Currently I am using a lustre system without OFED but our IB hardware is<br>
>> from the FDR generation so not bleeding edge and probably doesn't need<br>
>> OFED because of that....<br>
>><br>
>> Thanks,<br>
>> Eli<br>
>><br>
>> Tech specs:<br>
>> Servers: CentOS 6.8 + Lustre 2.8 (kernel from Lustre RPMs)<br>
>> Clients: Debian + kernel 4.2 + Lustre 2.8<br>
>> IB: ConnectX-3 FDR<br>
><br>
<br>
</div></div></blockquote></div><br></div></div>