<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>"E.S. Rosenberg" <<a href="mailto:esr+lustre@mail.hebrew.edu">esr+lustre@mail.hebrew.edu</a>><br>
<span style="font-weight:bold">Date: </span>Sunday, July 30, 2017 at 7:53 AM<br>
<span style="font-weight:bold">To: </span>Ben Evans <<a href="mailto:bevans@cray.com">bevans@cray.com</a>><br>
<span style="font-weight:bold">Cc: </span>Harald van Pee <<a href="mailto:pee@hiskp.uni-bonn.de">pee@hiskp.uni-bonn.de</a>>, "<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [lustre-discuss] Lustre and OFED<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jul 28, 2017 at 6:22 PM, Ben Evans <span dir="ltr">
<<a href="mailto:bevans@cray.com" target="_blank">bevans@cray.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-"><br>
<br>
On 7/28/17, 11:12 AM, "Harald van Pee" <<a href="mailto:pee@hiskp.uni-bonn.de">pee@hiskp.uni-bonn.de</a>> wrote:<br>
<br>
>Hello<br>
><br>
>On Friday 28 July 2017 15:48:12 Ben Evans wrote:<br>
>> Eli, just to clarify are you talking about using the in-kernel OFED vs.<br>
>>a<br>
>> vendor (Mellanox) OFED, or<br>
><br>
>In our case we are using the OFED of the debian distribution used.<br>
</span></blockquote>
<div>I am using the IB support that ships with Debian/CentOS/mainline kernel and did not install any OFED/Mellanox OFED package.<br>
</div>
<div>As far as I can tell RDMA does work. (using the various test tools suggested here
<a href="https://community.mellanox.com/docs/DOC-2086">https://community.mellanox.com/docs/DOC-2086</a>)
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-">><br>
>> are you talking about using the ConnectX-3<br>
>> hardware in IPoIB mode and just using it as a faster Ethernet?<br>
><br>
>is possible? How one have to do this?<br>
<br>
</span>You'd configure the lustre LNET to use it like any other ethernet device.<br>
The downside of this is that it's slower due to a lack of RDMA and other<br>
features that IB has. I'm not sure if there's a real upside to it.<br>
</blockquote>
<div><br>
</div>
<div>Now you've made me unsure of what whether or not my Lustre install is using RDMA, how should I be able to tell (we are definitely using IPoIB/o2ib)?
</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>If you are mounting Lustre with a string that looks like 192.168.1.10@o2ib,192.168.0.11@o2ib:/lustre then you're using OFED and RDMA.</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Eli<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="gmail-HOEnZb">
<div class="gmail-h5"><br>
><br>
>Harald<br>
><br>
><br>
>><br>
>> -Ben Evans<br>
>><br>
>> From: lustre-discuss<br>
>><br>
>><<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@<wbr>lists.lustre.org</a><mailto:<a href="mailto:lustre-discuss-bounces@li">lustre<wbr>-discuss-bounces@li</a><br>
>>s<br>
>> <a href="http://ts.lustre.org" rel="noreferrer" target="_blank">ts.lustre.org</a>>> on behalf of "E.S. Rosenberg"<br>
>> <<a href="mailto:esr%2Blustre@mail.hebrew.edu">esr+lustre@mail.hebrew.edu</a><<wbr>mailto:<a href="mailto:esr%2Blustre@mail.hebrew.edu">esr+lustre@mail.hebrew.<wbr>edu</a>>> Date:<br>
>> Thursday, July 27, 2017 at 4:55 PM<br>
>> To:<br>
>><br>
>>"<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.<wbr>lustre.org</a><mailto:<a href="mailto:lustre-discuss@lists.lustre.org">lustre-<wbr>discuss@lists.lustre.org</a>>"<br>
>><br>
>><<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.<wbr>lustre.org</a><mailto:<a href="mailto:lustre-discuss@lists.lustre.org">lustre-<wbr>discuss@lists.lustre.org</a>>><br>
>> Subject: [lustre-discuss] Lustre and OFED<br>
>><br>
>> Hi all,<br>
>><br>
>> How 'needed' is OFED for Lustre? In the LUG talks it is mentioned every<br>
>> once in a while and that got me thinking a bit.<br>
>><br>
>> What things are gained by installing OFED? Performance? Accurate traffic<br>
>> reports?<br>
>><br>
>> Currently I am using a lustre system without OFED but our IB hardware is<br>
>> from the FDR generation so not bleeding edge and probably doesn't need<br>
>> OFED because of that....<br>
>><br>
>> Thanks,<br>
>> Eli<br>
>><br>
>> Tech specs:<br>
>> Servers: CentOS 6.8 + Lustre 2.8 (kernel from Lustre RPMs)<br>
>> Clients: Debian + kernel 4.2 + Lustre 2.8<br>
>> IB: ConnectX-3 FDR<br>
><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</span>
</body>
</html>