<p>Hi Steve, You're welcome for the suggestion. I offered it as you mentioned adding a couple new oss servers and noticing the entries in the logs. Helpful to know would be where you are seeing the errors - new nodes only, or ?? Generally, networks with existing problems seems to work ok at low bandwidths, but problems start to appear as loads increase - hence the suggestion to check the network for problems. A quick check could be made with LNet self test between two different sets of nodes - set 1 nodes indicate the problem, and set 2 do not. Best,</p>
<div class="gmail_quote">On Dec 11, 2016 6:05 PM, "Steve Barnet" <<a href="mailto:barnet@icecube.wisc.edu">barnet@icecube.wisc.edu</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Brett,<br>
<br>
<br>
On 12/11/16 4:46 PM, Brett Lee wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Steve, It might be the network that LNet is running on. Have you run<br>
some bandwidth tests without LNet to check for network problems?<br>
</blockquote>
<br>
<br>
It's running over a 10Gb/s Ethernet network that is carrying<br>
other OSS traffic successfully. No routers or other fancy LNET<br>
features in play. However, it is quite possible that there are<br>
issues with the networking on the host side. Definitely on my<br>
list of things to test out.<br>
<br>
At this point, I'm just trying to narrow the search space.<br>
I didn't find anything particularly revealing when I searched<br>
around, so I'm hoping some expert eyes can shine a bit of<br>
light on the situation.<br>
<br>
Thanks for the tip!<br>
<br>
Best,<br>
<br>
---Steve<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On Dec 11, 2016 3:37 PM, "Steve Barnet" <<a href="mailto:barnet@icecube.wisc.edu" target="_blank">barnet@icecube.wisc.edu</a><br>
<mailto:<a href="mailto:barnet@icecube.wisc.edu" target="_blank">barnet@icecube.wisc.ed<wbr>u</a>>> wrote:<br>
<br>
Hi all,<br>
<br>
Seeing something very strange. I recently added two OSSes<br>
and 10 OSTs to one of our filesystems. Things look OK under<br>
light loads, but when we load them up, we start seeing lots<br>
of LNet errors.<br>
<br>
OS: Scientific Linux 6.7<br>
Lustre - Server: 2.8.0 Community version<br>
Lustre - Client: 2.5.3<br>
<br>
The errors are below. Do these narrow the range of possible<br>
problems?<br>
<br>
<br>
Dec 11 11:17:39 lfs-ex-oss-20 kernel: LNetError:<br>
7732:0:(socklnd_cb.c:2509:ksoc<wbr>knal_check_peer_timeouts()) Total 4<br>
stale ZC_REQs for peer 10.128.10.29@tcp1 detected; the<br>
oldest(ffff880f6a90e000) timed out 7 secs ago, resid: 0, wmem: 0<br>
Dec 11 11:17:39 lfs-ex-oss-20 kernel: LustreError:<br>
7732:0:(events.c:447:server_bu<wbr>lk_callback()) event type 5, status<br>
-5, desc ffff8805379f8000<br>
Dec 11 11:17:39 lfs-ex-oss-20 kernel: LustreError:<br>
7732:0:(events.c:447:server_bu<wbr>lk_callback()) event type 5, status<br>
-5, desc ffff880f375dc000<br>
Dec 11 11:17:39 lfs-ex-oss-20 kernel: LustreError:<br>
8234:0:(ldlm_lib.c:3175:target<wbr>_bulk_io()) @@@ network error on bulk<br>
READ req@ffff880e506263c0 x1551187318090340/t0(0)<br>
o3->092e941d-272a-09e3-502b-93<wbr>38dbf387d3@10.128.10.29@tcp1:<wbr>587/0<br>
lens 488/432 e 3 to 0 dl 1481476687 ref 1 fl Interpret:/0/0 rc 0/0<br>
Dec 11 11:17:39 lfs-ex-oss-20 kernel: LustreError:<br>
8234:0:(ldlm_lib.c:3175:target<wbr>_bulk_io()) Skipped 1 previous similar<br>
message<br>
Dec 11 11:17:39 lfs-ex-oss-20 kernel: Lustre: lfs2-OST0024: Bulk IO<br>
read error with 092e941d-272a-09e3-502b-9338db<wbr>f387d3 (at<br>
10.128.10.29@tcp1), client will retry: rc -110<br>
Dec 11 11:17:39 lfs-ex-oss-20 kernel: LustreError:<br>
7732:0:(events.c:447:server_bu<wbr>lk_callback()) event type 5, status<br>
-5, desc ffff8804db0ce000<br>
Dec 11 11:17:39 lfs-ex-oss-20 kernel: LustreError:<br>
7732:0:(events.c:447:server_bu<wbr>lk_callback()) event type 5, status<br>
-5, desc ffff880aa4374000<br>
<br>
<br>
Thanks much!<br>
<br>
Best,<br>
<br>
---Steve<br>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.or<wbr>g</a> <mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.l<wbr>ustre.org</a>><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listin<wbr>fo.cgi/lustre-discuss-lustre.<wbr>org</a><br>
<<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listi<wbr>nfo.cgi/lustre-discuss-lustre.<wbr>org</a>><br>
<br>
</blockquote>
<br>
</blockquote></div>