<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
The note regarding MOFED 4 not supported by Lustre: I’m working on it. MOFED 4 did not drop support of Lustre, but did make API/behaviour changes which Lustre has not fully adapted to yet. The ball is in the Lustre community’s court on this one now.
<div class=""><br class="">
</div>
<div class="">Doug</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 11, 2017, at 8:47 AM, Simon Guilbault <<a href="mailto:simon.guilbault@calculquebec.ca" class="">simon.guilbault@calculquebec.ca</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<div class="">
<div class="">Hi, your lnet.conf look fine, I tested lnet with RoCE V2 a while back with a pair of server using Connectx4 with a single 25Gb interface and RDMA was working with Centos 7.3, stock RHEL OFED and Lustre 2.9. The only settings that I had to use
in lustre's config was this one:</div>
<div class=""><br class="">
</div>
<div class="">options lnet networks=o2ib(ens2)</div>
</div>
<div class=""><br class="">
</div>
<div class="">The performance was about the same (1.9GB/s) without any tuning with the lnet self-test but the CPU utilisation was a lot lower with RDMA than TCP (3% vs 65% of a core). </div>
<div class=""><br class="">
</div>
<div class="">From my notes I took back then Lustre needed to be recompiled with MLNX OFED 3.4 and MLNX OFED 4 dropped support of Lustre accordings to their release notes.</div>
<div class=""><br class="">
</div>
<div class="">Ref 965588</div>
<div class=""><a href="https://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_0-2_0_0_1.pdf" class="">https://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_0-2_0_0_1.pdf</a><br class="">
</div>
<div class=""><a href="https://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_0-2_0_2_0.pdf" class="">https://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_0-2_0_2_0.pdf</a><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<div class="gmail_extra" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<br class="">
<div class="gmail_quote">On Thu, May 11, 2017 at 11:34 AM, Indivar Nair<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:indivar.nair@techterra.in" target="_blank" class="">indivar.nair@techterra.in</a>></span><span class="Apple-converted-space"> </span>wrote:<br class="">
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div dir="ltr" class="">
<div class="">
<div class="">
<div class="">
<div class="">So I should add something like this in lnet.conf -<br class="">
<br class="">
</div>
options lnet networks=o2ib0(p4p1)<br class="">
<br class="">
</div>
Thats it, right?<br class="">
<br class="">
</div>
Regards,<br class="">
<br class="">
<br class="">
</div>
Indivar Nair<br class="">
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Thu, May 11, 2017 at 8:39 PM, Dilger, Andreas<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:andreas.dilger@intel.com" target="_blank" class="">andreas.dilger@intel.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class="">
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
If you have RoCE cards and configure them with OFED, and configure Lustre to use o2iblnd then it should use RDMA for those interfaces. The fact that they are RoCE cards is hidden below OFED.<br class="">
<br class="">
Cheers, Andreas<br class="">
<div class="">
<div class="m_-2019842202792000363h5"><br class="">
> On May 11, 2017, at 08:36, Indivar Nair <<a href="mailto:indivar.nair@techterra.in" target="_blank" class="">indivar.nair@techterra.in</a>> wrote:<br class="">
><br class="">
> Hi ...,<br class="">
><br class="">
> I have read in different forums and blogs that Lustre supports RoCE.<br class="">
> But I cant find any documentation on it.<br class="">
><br class="">
> I have a Lustre setup with 6 OSS and 2 SMB/NFS Gateways.<br class="">
> They are all interconnected using Mellanox SN2700 100G Switch and Mellanox Connect-X4 100G NICs.<br class="">
> I have installed the Mellanox OFED Drivers, but I cant find a way to tell Lustre / LNET to use RoCE.<br class="">
><br class="">
> How do I go about?<br class="">
><br class="">
> Regards,<br class="">
><br class="">
><br class="">
> Indivar Nair<br class="">
><br class="">
><br class="">
</div>
</div>
> ______________________________<wbr class="">_________________<br class="">
> lustre-discuss mailing list<br class="">
><span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.or<wbr class="">g</a><br class="">
><span class="Apple-converted-space"> </span><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank" class="">http://lists.lustre.org/listin<wbr class="">fo.cgi/lustre-discuss-lustre.<wbr class="">org</a><br class="">
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
<br class="">
______________________________<wbr class="">_________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.<wbr class="">org</a><br class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank" class="">http://lists.lustre.org/<wbr class="">listinfo.cgi/lustre-discuss-<wbr class="">lustre.org</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">lustre-discuss
mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="mailto:lustre-discuss@lists.lustre.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">lustre-discuss@lists.lustre.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>