<div dir="ltr"><div><div><div>To find the RDMA devices you can use:<br>ibv_devices<br><br>You can bench your RDMA connection using qperf:<br></div>yum install qperf -y <br><br></div>on the client machine: qperf<br></div>on the server or ost machine: qperf clienthostname ud_lat ud_bw <br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 31, 2017 at 4:08 PM, Ben Evans <span dir="ltr"><<a href="mailto:bevans@cray.com" target="_blank">bevans@cray.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div><br>
</div>
<div><br>
</div>
<span id="m_5797113540752796843OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>"E.S. Rosenberg" <<a href="mailto:esr+lustre@mail.hebrew.edu" target="_blank">esr+lustre@mail.hebrew.edu</a>><br>
<span style="font-weight:bold">Date: </span>Sunday, July 30, 2017 at 7:53 AM<br>
<span style="font-weight:bold">To: </span>Ben Evans <<a href="mailto:bevans@cray.com" target="_blank">bevans@cray.com</a>><br>
<span style="font-weight:bold">Cc: </span>Harald van Pee <<a href="mailto:pee@hiskp.uni-bonn.de" target="_blank">pee@hiskp.uni-bonn.de</a>>, "<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<wbr>org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<wbr>org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [lustre-discuss] Lustre and OFED<br>
</div><span class="">
<div><br>
</div>
<div>
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jul 28, 2017 at 6:22 PM, Ben Evans <span dir="ltr">
<<a href="mailto:bevans@cray.com" target="_blank">bevans@cray.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="m_5797113540752796843gmail-"><br>
<br>
On 7/28/17, 11:12 AM, "Harald van Pee" <<a href="mailto:pee@hiskp.uni-bonn.de" target="_blank">pee@hiskp.uni-bonn.de</a>> wrote:<br>
<br>
>Hello<br>
><br>
>On Friday 28 July 2017 15:48:12 Ben Evans wrote:<br>
>> Eli, just to clarify are you talking about using the in-kernel OFED vs.<br>
>>a<br>
>> vendor (Mellanox) OFED, or<br>
><br>
>In our case we are using the OFED of the debian distribution used.<br>
</span></blockquote>
<div>I am using the IB support that ships with Debian/CentOS/mainline kernel and did not install any OFED/Mellanox OFED package.<br>
</div>
<div>As far as I can tell RDMA does work. (using the various test tools suggested here
<a href="https://community.mellanox.com/docs/DOC-2086" target="_blank">https://community.mellanox.<wbr>com/docs/DOC-2086</a>)
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="m_5797113540752796843gmail-">><br>
>> are you talking about using the ConnectX-3<br>
>> hardware in IPoIB mode and just using it as a faster Ethernet?<br>
><br>
>is possible? How one have to do this?<br>
<br>
</span>You'd configure the lustre LNET to use it like any other ethernet device.<br>
The downside of this is that it's slower due to a lack of RDMA and other<br>
features that IB has. I'm not sure if there's a real upside to it.<br>
</blockquote>
<div><br>
</div>
<div>Now you've made me unsure of what whether or not my Lustre install is using RDMA, how should I be able to tell (we are definitely using IPoIB/o2ib)?
</div>
</div>
</div>
</div>
</div>
</div>
</span></span>
<div><br>
</div>
<div>If you are mounting Lustre with a string that looks like 192.168.1.10@o2ib,192.168.0.<wbr>11@o2ib:/lustre then you're using OFED and RDMA.</div><div><div class="h5">
<span id="m_5797113540752796843OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Eli<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="m_5797113540752796843gmail-HOEnZb">
<div class="m_5797113540752796843gmail-h5"><br>
><br>
>Harald<br>
><br>
><br>
>><br>
>> -Ben Evans<br>
>><br>
>> From: lustre-discuss<br>
>><br>
>><<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@list<wbr>s.lustre.org</a><mailto:<a href="mailto:lustre-discuss-bounces@li" target="_blank">lustre-<wbr>discuss-bounces@li</a><br>
>>s<br>
>> <a href="http://ts.lustre.org" rel="noreferrer" target="_blank">ts.lustre.org</a>>> on behalf of "E.S. Rosenberg"<br>
>> <<a href="mailto:esr%2Blustre@mail.hebrew.edu" target="_blank">esr+lustre@mail.hebrew.edu</a><ma<wbr>ilto:<a href="mailto:esr%2Blustre@mail.hebrew.edu" target="_blank">esr+lustre@mail.hebrew.ed<wbr>u</a>>> Date:<br>
>> Thursday, July 27, 2017 at 4:55 PM<br>
>> To:<br>
>><br>
>>"<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre<wbr>.org</a><mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@<wbr>lists.lustre.org</a>>"<br>
>><br>
>><<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre<wbr>.org</a><mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@<wbr>lists.lustre.org</a>>><br>
>> Subject: [lustre-discuss] Lustre and OFED<br>
>><br>
>> Hi all,<br>
>><br>
>> How 'needed' is OFED for Lustre? In the LUG talks it is mentioned every<br>
>> once in a while and that got me thinking a bit.<br>
>><br>
>> What things are gained by installing OFED? Performance? Accurate traffic<br>
>> reports?<br>
>><br>
>> Currently I am using a lustre system without OFED but our IB hardware is<br>
>> from the FDR generation so not bleeding edge and probably doesn't need<br>
>> OFED because of that....<br>
>><br>
>> Thanks,<br>
>> Eli<br>
>><br>
>> Tech specs:<br>
>> Servers: CentOS 6.8 + Lustre 2.8 (kernel from Lustre RPMs)<br>
>> Clients: Debian + kernel 4.2 + Lustre 2.8<br>
>> IB: ConnectX-3 FDR<br>
><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</span>
</div></div></div>
<br>______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br></blockquote></div><br></div>