<div dir="ltr"><div dir="ltr"><div>If your routers have multiple OPA/MLX interfaces we found that linux routing can return the wrong HW address, which causes address resolution error. <br></div><div><br></div><div>You can try the following linux routing config to see if it helps:<br></div><div><a href="https://wiki.whamcloud.com/display/LNet/MR+Cluster+Setup">https://wiki.whamcloud.com/display/LNet/MR+Cluster+Setup</a><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 11 Feb 2019 at 12:04, Michael Di Domenico <<a href="mailto:mdidomenico4@gmail.com">mdidomenico4@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">i've narrowed down that my issue seems to stem from running over ipoib<br>
on an opa network<br>
<br>
i managed to pull all the routing and other things around so the only<br>
difference was whether i road the ipoib or not<br>
<br>
when i mount via ethernet, it works fine<br>
<br>
when i try the same mount via ipoib running ontop of opa it gets<br>
"input/output error". i can however lctl ping the storage and i see<br>
connections from the client to the MGS. so some of the connectivity<br>
is working, but it's breaking down somewhere else<br>
<br>
is anyone else running over ipoib on an opa network? if so, do you<br>
have lnet routing?<br>
<br>
some particulars<br>
<br>
rhel 7.6 clients<br>
2.10.5 clients<br>
2.5.x lustre servers (cray)<br>
lnet routing between storage and other networks<br>
currently running tcp ethernet, qdr infinipath, and fdr10 mellanox to<br>
the storage through routers<br>
no other machines are having mount issues<br>
<br>
<br>
On Fri, Feb 8, 2019 at 9:33 AM Michael Di Domenico<br>
<<a href="mailto:mdidomenico4@gmail.com" target="_blank">mdidomenico4@gmail.com</a>> wrote:<br>
> poking at this further, it doesn't look like it's ECMP issue.<br>
><br>
> Are there any known reports of issues when running Lustre over ipoib<br>
> over an opa fabric? seems a stretch, but it's the only difference in<br>
> the network at this point.<br>
><br>
> can anyone suggest somewhere to look for more debug info?<br>
> /var/log/messages and dmesg, don't reveal much info<br>
><br>
> On Mon, Feb 4, 2019 at 9:19 AM Michael Di Domenico<br>
> <<a href="mailto:mdidomenico4@gmail.com" target="_blank">mdidomenico4@gmail.com</a>> wrote:<br>
> ><br>
> > Has anyone heard of lustre having trouble mounting when ECMP is used<br>
> > on the compute nodes default gateway?<br>
> ><br>
> > I'm trying to mount an existing lustre filesystem on a new cluster,<br>
> > where the connections ride over OPA IPoIB, which is then converted to<br>
> > 10ge via four routers. I'm using ECMP to distribute the packets over<br>
> > the four routers.<br>
> ><br>
> > I can mount lustre on other ethernet clients, but not the ones behind<br>
> > my ECMP gateways. Changing the compute node gateway from ECMP to a<br>
> > single device doesn't change anything. I'm not easily able to revert<br>
> > the network side from ECMP to a single route, so i haven't tried that.<br>
> ><br>
> > The output i get from mount is, "failed: Input/output error retries left: 0"<br>
> ><br>
> > syslog on the client and the MGS seem to show that the connection is<br>
> > being broken between the MGS and client during the mount with a "timed<br>
> > oout for slow reply" message.<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div>