<div dir="ltr"><div><div><div><div>Follow-up to Subject: Lustre client mount fails: Request sent has timed out for slow reply<br><br></div>Thank you for the suggestions. I was able to work past this error. I am not certain of the exact solution. I did stop and restart my CentOS 7.2 opensm service. While that did not seem to change anything immediately, upon my return to the office after Thanksgiving the next compute nodes were successfully connected on the InfiniBand network fabric and the Lustre (2.8.0) file system mounted quickly as I issued the command.<br><br></div>So guessing here: I had to restart the opensm service and just be patient.<br><br></div>Cheers,<br></div>megan<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 25, 2016 at 4:06 PM, <span dir="ltr"><<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send lustre-discuss mailing list submissions to<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-discuss-request@lists.lustre.org">lustre-discuss-request@lists.<wbr>lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-discuss-owner@lists.lustre.org">lustre-discuss-owner@lists.<wbr>lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of lustre-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Lustre client mount fails: Request sent has timed out for<br>
slow reply (Dilger, Andreas)<br>
2. Re: Distributing locally.... (Dilger, Andreas)<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Fri, 25 Nov 2016 20:25:54 +0000<br>
From: "Dilger, Andreas" <<a href="mailto:andreas.dilger@intel.com">andreas.dilger@intel.com</a>><br>
To: "Ms. Megan Larko" <<a href="mailto:dobsonunit@gmail.com">dobsonunit@gmail.com</a>><br>
Cc: Lustre User Discussion Mailing List<br>
<<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a>><br>
Subject: Re: [lustre-discuss] Lustre client mount fails: Request sent<br>
has timed out for slow reply<br>
Message-ID: <<a href="mailto:0E44CDE8-D3E4-41C2-84A0-683B398FF846@intel.com">0E44CDE8-D3E4-41C2-84A0-<wbr>683B398FF846@intel.com</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Possible causes in cases like this:<br>
- duplicate client IP addresses (used only at connect time for o2iblnd)<br>
- firewall rules (though unlikely to be the case for IB)<br>
- SELinux (this is supported in Lustre 2.7+ but can still have rules that prevent mounting)<br>
<br>
Sorry, I don't know anything about opensm. Presumably you've restarted these clients, and<br>
other IB-level communications are working?<br>
<br>
Cheers, Andreas<br>
<br>
On Nov 25, 2016, at 12:05, Ms. Megan Larko <<a href="mailto:dobsonunit@gmail.com">dobsonunit@gmail.com</a>> wrote:<br>
><br>
> Greetings List!<br>
><br>
> I have a very small HPC cluster running CentOS 7.2. The lustre servers are running lustre kernel-3.10.0-327.3.1.el7_<wbr>lustre.x86_64. The clients are running kernel-3.10.0-327.3.1.el7.x86_<wbr>64.<br>
><br>
> I have two compute node clients successfully mounting the Lustre file system from the servers. The next two compute clients will not mount lustre. I have the lustre-client-3.8.0-3.10.0_<wbr>327.3.1.el7.x86_64 and lustre-client-modules-2.8.0-e.<wbr>10.0_327.3.1.el7.x86_64 rpm installed on all compute clients, including the next two. My InfiniBand network is up and successfully pings the other systems. I can cleanly "modprobe lustre" using /etc/modprobe.d/lustre.conf containing one line: options lnet networks="o2ib0(ib0)". This information is the same on both Lustre client and server systems, all of which use ib0.<br>
><br>
> On the next two compute clients I can successfully "lctl ping mds-ib@o2ib0" and successfully ping the oss similarly. I try to mount the Lustre file system on the next two compute clients via the command "mount -t lustre A.B.C.D@o2ib0:/myLustre /myLustre where the A.B.C.D address exists and works as described above and the Lustre FS is "myLustre" and successfully mounts on the two earlier compute clients.<br>
><br>
> This mount fails on both of my next two compute clients with the STDERR:<br>
><br>
> mount.lustre: mount A.B.C.D@o2ib0:/myLustre /myLustre failed: Input/output error<br>
><br>
> The compute client /var/log/messages file shows:<br>
> [date] [hostname] kernel: Lustre: 51814:0:(client.c:2063:ptlrpc_<wbr>expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1480097968/real 1480097992] req@ffff8800aa14000 x1551992831868952/t0(0) o250->MCGA.B.C.D@o2ib@A.B.C.D@<wbr>o2ib:26:25 lens 520/544 e 0 to 1 dl 1480997973 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br>
><br>
> The above appears 2X in a row followed by:<br>
> [date] [hostname] kernel: LustreError: 15c-8: MGCA.B.C.D@o2ib: The configuration from log 'myLustre-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br>
> [date] [hostname] kernel: Lustre: Unmounted myLustre-client<br>
> [date] [hostname] kernel: LustreError: 53873:0:(obd_mount.c:1426:<wbr>lustre_fill_super()) unable to mount (-5)<br>
><br>
> As all four compute nodes are built from a single kickstart file, I do not understand why two compute clients can mount the /myLustre file system and two cannot. The IB fabric on the in-kernel opensm-3.3.10-1.el7.x86_64 looks clean with no entries in the /var/log/opensm-unhealthy-<wbr>ports-dump. If I go all the way back to the last opensm start I do see a single line in /var/log/opensm.log on the opensm server for the next compute client stating:<br>
> subn_validate_neighbor: ERR 7518: neighbor does not point back at us (guid: [GUID of my next compute client])<br>
><br>
> Is this last opensm error completely stopping my Lustre mount when all other IP pings are completely successful?<br>
><br>
> TIA,<br>
> megan<br>
> ______________________________<wbr>_________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Fri, 25 Nov 2016 20:50:03 +0000<br>
From: "Dilger, Andreas" <<a href="mailto:andreas.dilger@intel.com">andreas.dilger@intel.com</a>><br>
To: Thomas Stibor <<a href="mailto:t.stibor@gsi.de">t.stibor@gsi.de</a>><br>
Cc: "<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a>><br>
Subject: Re: [lustre-discuss] Distributing locally....<br>
Message-ID: <<a href="mailto:731807CD-AA6E-469C-9593-C27426DF0139@intel.com">731807CD-AA6E-469C-9593-<wbr>C27426DF0139@intel.com</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
On Nov 25, 2016, at 04:27, Thomas Stibor <<a href="mailto:t.stibor@gsi.de">t.stibor@gsi.de</a>> wrote:<br>
><br>
> Remove in debian/lustre-dev.install the line<br>
> -debian/tmp/usr/lib/*.so.* usr/lib<br>
> and it will work.<br>
><br>
> @@ -1,6 +1,5 @@<br>
> lustre/contrib/README usr/share/doc/lustre-dev/<wbr>contrib<br>
> lustre/contrib/mpich-1.2.6-<wbr>lustre.patch usr/share/doc/lustre-dev/<wbr>contrib<br>
> debian/tmp/usr/include/lustre/<wbr>* usr/include/lustre<br>
> -debian/tmp/usr/lib/*.so.* usr/lib<br>
> debian/tmp/usr/lib/*.so usr/lib<br>
> debian/tmp/usr/lib/*.a usr/lib<br>
<br>
Thomas or Phill,<br>
could you please submit a patch to Gerrit with this change.<br>
<br>
> Note, also make sure to update<br>
> debian/changelog<br>
> e.g. with cmd<br>
><br>
> export DEBFULLNAME="My Name"<br>
> export EMAIL="<a href="mailto:myname@mydomain.cz">myname@mydomain.cz</a>"<br>
><br>
> # Extract lustre version, replace "_" by "." and remove leading letter "v".<br>
> LUSTRE_VERSION=$(echo `git describe` | sed -e "s/_/\./g" | cut -c2-)<br>
> LUSTRE_DEBIAN_REV='1'<br>
><br>
> # Add entry into debian/changelog such that packages have proper version names.<br>
> dch --newversion ${LUSTRE_VERSION}-${LUSTRE_<wbr>DEBIAN_REV} --distribution unstable --nomultimaint -t "Build from official master upstream."<br>
><br>
> otherwise you get package version names according to top entry in debian/changelog<br>
> which does not usually match with the GIT version you are compiling.<br>
<br>
It would be nice to add this as part of the "make debs" target so that the build is<br>
done with the right version. Bonus points if it checks the top changelog entry to<br>
see there is already an entry for the current version and doesn't add a new entry.<br>
<br>
Cheers, Andreas<br>
<br>
> Cheers<br>
> Thomas<br>
><br>
> On Fri, Nov 25, 2016 at 10:04:06AM +0000, Phill Harvey-Smith wrote:<br>
>> On 02/11/2016 17:54, Dilger, Andreas wrote:<br>
>>> There is a "make debs" target, but I don't know how often this is<br>
>>> tested. That would be the best thing to use for Ubuntu, and if it isn't<br>
>>> working then please feel free to report to the list and/or Jira.<br>
>><br>
>> Just got back to this,<br>
>><br>
>> make debs gets further but still seems to crash out....<br>
>><br>
>> Steps :<br>
>><br>
>> Get source from git.<br>
>> Select 2.8.0 with : git checkout 2.8.0<br>
>> sh ./autogen.sh<br>
>> ./configure --disable-server --with-o2ib=no<br>
>> make<br>
>><br>
>> The make completes correctly, without errors, I have done a make install<br>
>> on this node in the past with this version which is up and running<br>
>> correctly.<br>
>><br>
>> make debs<br>
>><br>
>> bombs out, log below :<br>
>><br>
>> I've uploaded the log to :<br>
>><br>
>> <a href="http://penguin.stats.warwick.ac.uk/~stsxab/Lustre/lustre_make_deb_error.txt" rel="noreferrer" target="_blank">http://penguin.stats.warwick.<wbr>ac.uk/~stsxab/Lustre/lustre_<wbr>make_deb_error.txt</a><br>
>><br>
>> As the list refused to accept it as it was too big :(<br>
>><br>
>> Cheers.<br>
>><br>
>> Phill.<br>
>><br>
>> ______________________________<wbr>_________________<br>
>> lustre-discuss mailing list<br>
>> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
>> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
> ______________________________<wbr>_________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
<br>
------------------------------<br>
<br>
End of lustre-discuss Digest, Vol 128, Issue 11<br>
******************************<wbr>*****************<br>
</blockquote></div><br></div>