<div dir="ltr"><div>With regards to (WRT) Subject "Robinhood exhausting RPC resources against 2.5.5 lustre file systems", what version of robinhood and what version of MySQL database? I mention this because I have been working with robinhood-3.0-0.rc1 and initially MySQL-5.5.32 and Lustre 2.5.42.1 on kernel-2.6.32-573 and had issues in which the robinhood server consumed more than the total amount of 32 CPU cores on the robinhood server (with 128 G RAM) and would functionally hang the robinhood server. The issue was solved for me by changing to MySQL-5.6.35. It was the "sort" command in robinhood that was not working well with the MySQL-5.5.32.</div><div><br></div><div>Cheers,</div><div>megan</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 17, 2017 at 2:04 PM, <span dir="ltr"><<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send lustre-discuss mailing list submissions to<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank" rel="noreferrer">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-discuss-request@lists.lustre.org">lustre-discuss-request@lists.<wbr>lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-discuss-owner@lists.lustre.org">lustre-discuss-owner@lists.<wbr>lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of lustre-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: seclabel (Robin Humble)<br>
2. Re: seclabel (Sebastien Buisson)<br>
3. Robinhood exhausting RPC resources against 2.5.5 lustre file<br>
systems (Jessica Otey)<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Wed, 17 May 2017 10:16:51 -0400<br>
From: Robin Humble <<a href="mailto:rjh%2Blustre@cita.utoronto.ca">rjh+lustre@cita.utoronto.ca</a>><br>
To: <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
Subject: Re: [lustre-discuss] seclabel<br>
Message-ID: <<a href="mailto:20170517141651.GA1064@trinity.cita.utoronto.ca">20170517141651.GA1064@<wbr>trinity.cita.utoronto.ca</a>><br>
Content-Type: text/plain; charset=us-ascii<br>
<br>
I setup a couple of VMs with 2.9 clients and servers (ldiskfs) and<br>
unfortunately setcap/getcap still are unhappy - same as with my<br>
previous 2.9 clients with 2.8 servers (ZFS).<br>
<br>
hmm.<br>
I took a gander at the source and noticed that llite/xattr.c<br>
deliberately filters out 'security.capability' and returns 0/-ENODATA<br>
for setcap/getcap, which is indeed what strace sees. so setcap/getcap<br>
is never even sent to the MDS.<br>
<br>
if I remove that filter (see patch on lustre-devel) then setcap/getcap<br>
works -><br>
<br>
# df .<br>
Filesystem 1K-blocks Used Available Use% Mounted on<br>
10.122.1.5@tcp:/test8 4797904 33992 4491480 1% /mnt/test8<br>
# touch blah<br>
# setcap cap_net_admin,cap_net_raw+p blah<br>
# getcap blah<br>
blah = cap_net_admin,cap_net_raw+p<br>
<br>
and I also tested that the 'ping' binary run as unprivileged user works<br>
from lustre.<br>
success!<br>
<br>
'b15587' is listed as the reason for the filtering.<br>
I don't know what that refers to.<br>
is it still relevant?<br>
<br>
cheers,<br>
robin<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 17 May 2017 14:37:31 +0000<br>
From: Sebastien Buisson <<a href="mailto:sbuisson@ddn.com">sbuisson@ddn.com</a>><br>
To: Robin Humble <<a href="mailto:rjh%2Blustre@cita.utoronto.ca">rjh+lustre@cita.utoronto.ca</a>><br>
Cc: "<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a>><br>
Subject: Re: [lustre-discuss] seclabel<br>
Message-ID: <<a href="mailto:B4673494-40C1-4E5C-AB92-6A146386A86E@ddn.com">B4673494-40C1-4E5C-AB92-<wbr>6A146386A86E@ddn.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi Robin,<br>
<br>
b15587 refers to the old Lustre Bugzilla tracking tool:<br>
<a href="https://projectlava.xyratex.com/show_bug.cgi?id=15587" target="_blank" rel="noreferrer">https://projectlava.xyratex.<wbr>com/show_bug.cgi?id=15587</a><br>
<br>
Reading the discussion in the ticket, supporting xattr at the time of Lustre 1.8 and 2.0 was causing issues on MDS side in some situations. So it was decided to discard security.capability xattr on Lustre client side. I think Andreas might have some insight, as he apparently participated in b15587.<br>
<br>
In any case, it is important to make clear that file capabilities, the feature you want to use, is completely distinct from SELinux.<br>
On the one hand, Capabilities are a Linux mechanism to refine permissions granted to privileged processes, by dividing the privileges traditionally associated with superuser into distinct units (known as capabilities).<br>
On the other hand, SELinux is the Linux implementation of Mandatory Access Control.<br>
Both Capabilities and SELinux rely on values stored into file extended attributes, but this is the only thing they have in common.<br>
<br>
Cheers,<br>
Sebastien.<br>
<br>
> Le 17 mai 2017 ? 16:16, Robin Humble <<a href="mailto:rjh%2Blustre@cita.utoronto.ca">rjh+lustre@cita.utoronto.ca</a>> a ?crit :<br>
><br>
> I setup a couple of VMs with 2.9 clients and servers (ldiskfs) and<br>
> unfortunately setcap/getcap still are unhappy - same as with my<br>
> previous 2.9 clients with 2.8 servers (ZFS).<br>
><br>
> hmm.<br>
> I took a gander at the source and noticed that llite/xattr.c<br>
> deliberately filters out 'security.capability' and returns 0/-ENODATA<br>
> for setcap/getcap, which is indeed what strace sees. so setcap/getcap<br>
> is never even sent to the MDS.<br>
><br>
> if I remove that filter (see patch on lustre-devel) then setcap/getcap<br>
> works -><br>
><br>
> # df .<br>
> Filesystem 1K-blocks Used Available Use% Mounted on<br>
> 10.122.1.5@tcp:/test8 4797904 33992 4491480 1% /mnt/test8<br>
> # touch blah<br>
> # setcap cap_net_admin,cap_net_raw+p blah<br>
> # getcap blah<br>
> blah = cap_net_admin,cap_net_raw+p<br>
><br>
> and I also tested that the 'ping' binary run as unprivileged user works<br>
> from lustre.<br>
> success!<br>
><br>
> 'b15587' is listed as the reason for the filtering.<br>
> I don't know what that refers to.<br>
> is it still relevant?<br>
><br>
> cheers,<br>
> robin<br>
> ______________________________<wbr>_________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank" rel="noreferrer">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Wed, 17 May 2017 14:04:28 -0400<br>
From: Jessica Otey <<a href="mailto:jotey@nrao.edu">jotey@nrao.edu</a>><br>
To: "<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a>>,<br>
<a href="mailto:robinhood-support@lists.sourceforge.net">robinhood-support@lists.<wbr>sourceforge.net</a><br>
Subject: [lustre-discuss] Robinhood exhausting RPC resources against<br>
2.5.5 lustre file systems<br>
Message-ID: <<a href="mailto:32c23887-3ee5-c0be-8df5-8b9d6a6791f1@nrao.edu">32c23887-3ee5-c0be-8df5-<wbr>8b9d6a6791f1@nrao.edu</a>><br>
Content-Type: text/plain; charset="utf-8"; Format="flowed"<br>
<br>
All,<br>
<br>
We have observed an unfortunate interaction between Robinhood and two<br>
Lustre 2.5.5 file systems (both of which originated as 1.8.9 file systems).<br>
<br>
Robinhood was used successfully against these file systems when they<br>
were both 1.8.9, 2.4.3, and then 2.5.3 (a total time span of about 11<br>
months).<br>
<br>
We also have a third Lustre file system that originated as 2.4.3, and<br>
has since been upgraded to 2.5.5, against which Robinhood is currently<br>
operating as expected. This leads me to suppose that the issue may have<br>
to do the interaction between Robinhood and a<br>
legacy-1.8.x-now-lustre-2.5.5 system. But I don't know.<br>
<br>
The problem manifests itself as follows: Either a Robinhood file scan or<br>
the initiation of the consumption of changelogs results in the<br>
consumption all the available RPC resources on the MDT. This in turn<br>
leads to the MDT not being able to satisfy any other requests from<br>
clients, which in turn leads to client disconnections (the MDT thinks<br>
they are dead and evicts them). Meanwhile, Robinhood itself is unable to<br>
traverse the file system to gather the information it seeks, and so its<br>
scans either hang (due to the client disconnect) or run at a rate such<br>
that they would never complete (less than 1 file per second).<br>
<br>
If we don't run robinhood at all, the file system performs (after a<br>
remount of the MDT) as expected.<br>
<br>
Initially, we thought that the difficulty might be that we neglected to<br>
activate the FID-in-direct feature when we upgraded to 2.4.3. We did so<br>
on one of these systems, and ran an lfsck oi_scrub, but that did not<br>
ameliorate the problem.<br>
<br>
Any thoughts on this matter would be appreciated. (We miss using Robinhood!)<br>
<br>
Thanks,<br>
<br>
Jessica<br>
<br>
******************************<wbr>******************************<br>
<br>
More data for those who cannot help themselves:<br>
<br>
April 2016 - Robinhood comes into production use against both our 1.8.9<br>
file systems.<br>
<br>
July 2016 - Upgrade to 2.4.3 (on both production lustre file systems) --<br>
Robinhood rebuilt against 2.4.3 client; changelog consumption now included.<br>
<br>
Lustre "reconnects" (from /var/log/messages on one of the MDTs):<br>
<br>
July 2016: 4<br>
<br>
Aug 2016: 20<br>
<br>
Sept 2016: 8<br>
<br>
Oct 2016: 8<br>
<br>
Nov 4-6, 2016 - Upgrade to 2.5.3 (on both production lustre file<br>
systems) -- Robinhood rebuilt against 2.5.3 client.<br>
<br>
Lustre "reconnects":<br>
<br>
Nov. 2016: 180<br>
<br>
Dec. 2016: 62<br>
<br>
Jan. 2017: 96<br>
<br>
Feb 1-24, 2017: 2<br>
<br>
Feb 24, 2017 - Upgrade to 2.5.5 (on both production lustre file systems)<br>
<br>
**** NAASC-Lustre MDT coming back ****<br>
<br>
Feb 24 20:46:44 10.7.7.8 kernel: Lustre: Lustre: Build Version:<br>
2.5.5-g22a210f-CHANGED-2.6.32-<wbr>642.6.2.el6_lustre.2.5.5.x86_<wbr>64<br>
Feb 24 20:46:44 10.7.7.8 kernel: Lustre: Lustre: Build Version:<br>
2.5.5-g22a210f-CHANGED-2.6.32-<wbr>642.6.2.el6_lustre.2.5.5.x86_<wbr>64<br>
Feb 24 20:46:44 10.7.7.8 kernel: LNet: Added LNI 10.7.17.8@o2ib<br>
[8/256/0/180]<br>
Feb 24 20:46:44 10.7.7.8 kernel: LNet: Added LNI 10.7.17.8@o2ib<br>
[8/256/0/180]<br>
Feb 24 20:46:45 10.7.7.8 kernel: LDISKFS-fs (md127): mounted filesystem<br>
with ordered data mode. quota=off. Opts:<br>
Feb 24 20:46:45 10.7.7.8 kernel: LDISKFS-fs (md127): mounted filesystem<br>
with ordered data mode. quota=off. Opts:<br>
Feb 24 20:46:46 10.7.7.8 kernel: Lustre: MGC10.7.17.8@o2ib: Connection<br>
restored to MGS (at 0@lo)<br>
Feb 24 20:46:46 10.7.7.8 kernel: Lustre: MGC10.7.17.8@o2ib: Connection<br>
restored to MGS (at 0@lo)<br>
Feb 24 20:46:47 10.7.7.8 kernel: Lustre: naaschpc-MDT0000: used disk,<br>
loading<br>
Feb 24 20:46:47 10.7.7.8 kernel: Lustre: naaschpc-MDT0000: used disk,<br>
loading<br>
<br>
The night after this upgrade, a regular rsync to the backup Lustre<br>
system provokes a failure/client disconnect. (Unfortunately, I don't<br>
have the logs to look at Robinhood activity from this time, but I<br>
believe I restarted the service after the system came back.)<br>
<br>
Feb 25 02:14:24 10.7.7.8 kernel: LustreError:<br>
25103:0:(service.c:2020:<wbr>ptlrpc_server_handle_request()<wbr>) @@@ Dropping<br>
timed-out request from 12345-10.7.17.123@o2ib: deadline 6:11s ago<br>
Feb 25 02:14:24 10.7.7.8 kernel: LustreError:<br>
25103:0:(service.c:2020:<wbr>ptlrpc_server_handle_request()<wbr>) @@@ Dropping<br>
timed-out request from 12345-10.7.17.123@o2ib: deadline 6:11s ago<br>
Feb 25 02:14:24 10.7.7.8 kernel: req@ffff88045b3a2050<br>
x1560271381909936/t0(0)<br>
o103->bb228923-4216-cc59-d847-<wbr>38b543af1ae2@10.7.17.123@o2ib:<wbr>0/0 lens<br>
3584/0 e 0 to 0 dl 1488006853 ref 1 fl Interpret:/0/ffffffff rc 0/-1<br>
Feb 25 02:14:24 10.7.7.8 kernel: req@ffff88045b3a2050<br>
x1560271381909936/t0(0)<br>
o103->bb228923-4216-cc59-d847-<wbr>38b543af1ae2@10.7.17.123@o2ib:<wbr>0/0 lens<br>
3584/0 e 0 to 0 dl 1488006853 ref 1 fl Interpret:/0/ffffffff rc 0/-1<br>
Feb 25 02:14:24 10.7.7.8 kernel: Lustre:<br>
25111:0:(service.c:2052:<wbr>ptlrpc_server_handle_request()<wbr>) @@@ Request took<br>
longer than estimated (6:11s); client may timeout. req@ffff88045b3a2850<br>
x1560271381909940/t0(0)<br>
o103->bb228923-4216-cc59-d847-<wbr>38b543af1ae2@10.7.17.123@o2ib:<wbr>0/0 lens<br>
3584/0 e 0 to 0 dl 1488006853 ref 1 fl Interpret:/0/ffffffff rc 0/-1<br>
Feb 25 02:14:24 10.7.7.8 kernel: Lustre:<br>
25111:0:(service.c:2052:<wbr>ptlrpc_server_handle_request()<wbr>) @@@ Request took<br>
longer than estimated (6:11s); client may timeout. req@ffff88045b3a2850<br>
x1560271381909940/t0(0)<br>
o103->bb228923-4216-cc59-d847-<wbr>38b543af1ae2@10.7.17.123@o2ib:<wbr>0/0 lens<br>
3584/0 e 0 to 0 dl 1488006853 ref 1 fl Interpret:/0/ffffffff rc 0/-1<br>
Feb 25 02:14:24 10.7.7.8 kernel: LustreError:<br>
25103:0:(service.c:2020:<wbr>ptlrpc_server_handle_request()<wbr>) Skipped 30<br>
previous similar messages<br>
Feb 25 02:14:24 10.7.7.8 kernel: LustreError:<br>
25103:0:(service.c:2020:<wbr>ptlrpc_server_handle_request()<wbr>) Skipped 30<br>
previous similar messages<br>
Feb 25 02:14:25 10.7.7.8 kernel: LustreError:<br>
7526:0:(service.c:2020:ptlrpc_<wbr>server_handle_request()) @@@ Dropping<br>
timed-out request from 12345-10.7.17.123@o2ib: deadline 6:11s ago<br>
Feb 25 02:14:25 10.7.7.8 kernel: LustreError:<br>
7526:0:(service.c:2020:ptlrpc_<wbr>server_handle_request()) @@@ Dropping<br>
timed-out request from 12345-10.7.17.123@o2ib: deadline 6:11s ago<br>
Feb 25 02:14:25 10.7.7.8 kernel: req@ffff880149b75850<br>
x1560271381926356/t0(0)<br>
o103->bb228923-4216-cc59-d847-<wbr>38b543af1ae2@10.7.17.123@o2ib:<wbr>0/0 lens<br>
3584/0 e 0 to 0 dl 1488006854 ref 1 fl Interpret:/0/ffffffff rc 0/-1<br>
Feb 25 02:14:25 10.7.7.8 kernel: req@ffff880149b75850<br>
x1560271381926356/t0(0)<br>
o103->bb228923-4216-cc59-d847-<wbr>38b543af1ae2@10.7.17.123@o2ib:<wbr>0/0 lens<br>
3584/0 e 0 to 0 dl 1488006854 ref 1 fl Interpret:/0/ffffffff rc 0/-1<br>
Feb 25 02:14:25 10.7.7.8 kernel: Lustre:<br>
7528:0:(service.c:2052:ptlrpc_<wbr>server_handle_request()) @@@ Request took<br>
longer than estimated (6:11s); client may timeout. req@ffff880149bb0050<br>
x1560271381926372/t0(0)<br>
o103->bb228923-4216-cc59-d847-<wbr>38b543af1ae2@10.7.17.123@o2ib:<wbr>0/0 lens<br>
3584/0 e 0 to 0 dl 1488006854 ref 1 fl Interpret:/0/ffffffff rc 0/-1<br>
Feb 25 02:14:25 10.7.7.8 kernel: Lustre:<br>
7528:0:(service.c:2052:ptlrpc_<wbr>server_handle_request()) @@@ Request took<br>
longer than estimated (6:11s); client may timeout. req@ffff880149bb0050<br>
x1560271381926372/t0(0)<br>
o103->bb228923-4216-cc59-d847-<wbr>38b543af1ae2@10.7.17.123@o2ib:<wbr>0/0 lens<br>
3584/0 e 0 to 0 dl 1488006854 ref 1 fl Interpret:/0/ffffffff rc 0/-1<br>
Feb 25 02:14:25 10.7.7.8 kernel: Lustre:<br>
7528:0:(service.c:2052:ptlrpc_<wbr>server_handle_request()) Skipped 1533<br>
previous similar messages<br>
Feb 25 02:14:25 10.7.7.8 kernel: Lustre:<br>
7528:0:(service.c:2052:ptlrpc_<wbr>server_handle_request()) Skipped 1533<br>
previous similar messages<br>
Feb 25 02:14:25 10.7.7.8 kernel: LustreError:<br>
7526:0:(service.c:2020:ptlrpc_<wbr>server_handle_request()) Skipped 1653<br>
previous similar messages<br>
Feb 25 02:14:25 10.7.7.8 kernel: LustreError:<br>
7526:0:(service.c:2020:ptlrpc_<wbr>server_handle_request()) Skipped 1653<br>
previous similar messages<br>
Feb 25 02:14:49 10.7.7.8 kernel: Lustre: naaschpc-MDT0000: Client<br>
bb228923-4216-cc59-d847-<wbr>38b543af1ae2 (at 10.7.17.123@o2ib) reconnecting<br>
Feb 25 02:14:49 10.7.7.8 kernel: Lustre: naaschpc-MDT0000: Client<br>
bb228923-4216-cc59-d847-<wbr>38b543af1ae2 (at 10.7.17.123@o2ib) reconnecting<br>
Feb 25 02:14:49 10.7.7.8 kernel: Lustre: naaschpc-MDT0000: Client<br>
bb228923-4216-cc59-d847-<wbr>38b543af1ae2 (at 10.7.17.123@o2ib) refused<br>
reconnection, still busy with 1 active RPCs<br>
Feb 25 02:14:49 10.7.7.8 kernel: Lustre: naaschpc-MDT0000: Client<br>
bb228923-4216-cc59-d847-<wbr>38b543af1ae2 (at 10.7.17.123@o2ib) refused<br>
reconnection, still busy with 1 active RPCs<br>
Feb 25 02:14:49 10.7.7.8 kernel: Lustre: Skipped 1 previous similar message<br>
Feb 25 02:14:49 10.7.7.8 kernel: Lustre: Skipped 1 previous similar message<br>
Feb 25 02:14:49 10.7.7.8 kernel: LustreError:<br>
9437:0:(ldlm_lib.c:2693:<wbr>target_bulk_io()) @@@ bulk PUT failed: rc -107<br>
req@ffff88050290d050 x1560271382966952/t0(0)<br>
o37->bb228923-4216-cc59-d847-<wbr>38b543af1ae2@10.7.17.123@o2ib:<wbr>0/0 lens<br>
448/440 e 0 to 0 dl 1488006895 ref 1 fl Interpret:/0/0 rc 0/0<br>
Feb 25 02:14:49 10.7.7.8 kernel: LustreError:<br>
9437:0:(ldlm_lib.c:2693:<wbr>target_bulk_io()) @@@ bulk PUT failed: rc -107<br>
req@ffff88050290d050 x1560271382966952/t0(0)<br>
o37->bb228923-4216-cc59-d847-<wbr>38b543af1ae2@10.7.17.123@o2ib:<wbr>0/0 lens<br>
448/440 e 0 to 0 dl 1488006895 ref 1 fl Interpret:/0/0 rc 0/0<br>
Feb 25 02:15:14 10.7.7.8 kernel: Lustre: naaschpc-MDT0000: Client<br>
bb228923-4216-cc59-d847-<wbr>38b543af1ae2 (at 10.7.17.123@o2ib) reconnecting<br>
Feb 25 02:15:14 10.7.7.8 kernel: Lustre: naaschpc-MDT0000: Client<br>
bb228923-4216-cc59-d847-<wbr>38b543af1ae2 (at 10.7.17.123@o2ib) reconnecting<br>
<br>
This problem then escalates... the Lustre "reconnects" from the logs for<br>
March will be 2818; for April<br>
4574! *<br>
<br>
Grepping for "active RPCs", excluding dates of known downtimes (Nov 5,<br>
2016 | Feb 24, 2017 | Apr 21, 2017)". Dates not included are zero.<br>
<br>
**2016*<br>
Nov 22nd: 72<br>
Dec 14th: 32<br>
<br>
*2017*<br>
Jan 8th: 20<br>
Jan 28th: 18<br>
---<br>
Feb 25: 2<br>
---<br>
Mar 13: 56<br>
Mar 14: 236 First user ticket filed about fs non-responsiveness<br>
Mar 15: 60<br>
Mar 16: 90<br>
Mar 17: 160<br>
Mar 18: 280<br>
Mar 19: 178<br>
Mar 20: 280<br>
Mar 21: 276<br>
Mar 22: 140<br>
Mar 23: 102<br>
Mar 24: 216<br>
Mar 25: 278<br>
Mar 26: 142<br>
Mar 27: 82<br>
Mar 28: 52<br>
Mar 29: 78<br>
Mar 30: 52<br>
Mar 31: 52<br>
--<br>
Apr 1: 112<br>
Apr 2: 216<br>
Apr 3: 276<br>
Apr 4: 278<br>
Apr 5: 278<br>
Apr 6: 278<br>
Apr 7: 158<br>
Apr 8: 216<br>
Apr 9: 278<br>
Apr 10: 118<br>
Apr 11: 216<br>
Apr 12: 156<br>
Apr 13: 78<br>
Apr 14: 42<br>
Apr 15: 74<br>
Apr 16: 216<br>
Apr 17: 88<br>
Apr 18: 62<br>
Apr 19: 198<br>
Apr 20: 280<br>
April 21, 2017 - on naasc lustre only: MDT and OSTs unmounted; MDT<br>
remounted in ordered mode; lfsck oi_scrub run<br>
Apr 22: 0<br>
Apr 23: 0<br>
Apr 24: 0<br>
Apr 25: 0<br>
Apr 26: 0<br>
Apr 27: 0<br>
<br>
On Friday, April 28, I start a file system scan from our 'new' robinhood<br>
box, akebono, on both production file systems<br>
<br>
2017/04/28 14:38:35 [130550/1] CheckFS | '/.lustre/naasc' matches mount<br>
point '/.lustre/naasc', type=lustre, fs=10.7.17.8@o2ib0:/naaschpc<br>
<br>
This scan does not successfully complete. And the RPC resource problem<br>
reappears:<br>
Apr 28: 114 The first one is at 3pm...<br>
<br>
Apr 28 15:01:30 10.7.7.8 kernel: Lustre: naaschpc-MDT0000: Client<br>
cab57f86-2124-884d-49a2-<wbr>ec2871f36f78 (at 10.7.17.122@o2ib) refused<br>
reconnection, still busy with 1 active RPCs<br>
<br>
Apr 29: 282<br>
Apr 30: 284<br>
---<br>
May 1: 186<br>
May 2: 248<br>
May 3: 204 (day of emergency MDT restart)<br>
<br>
Since May 4: 0 Robinhood has not been restarted against naasc<br>
lustre... and rsyncs have been running without issues.<br>
<br>
*********<br>
Robinhood scan/changelog consumption also produced an identical<br>
situation on our other lustre fs<br>
<br>
2017/04/28 14:53:11 [130926/1] CheckFS | '/.lustre/cv' matches mount<br>
point '/.lustre/cv', type=lustre, fs=10.7.3.9@tcp2:/cvlustre<br>
<br>
***** FROM CV-lustre's MDT *****<br>
<br>
Apr 28 14:53:24 10.7.7.9 kernel: Lustre: cvlustre-MDT0000: Client<br>
deb3834b-6b53-5e68-123b-<wbr>dfaedd352ac6 (at 10.7.17.122@o2ib) refused<br>
reconnection, still busy with 1 active RPCs<br>
Apr 28 14:53:24 10.7.7.9 kernel: Lustre: cvlustre-MDT0000: Client<br>
deb3834b-6b53-5e68-123b-<wbr>dfaedd352ac6 (at 10.7.17.122@o2ib) refused<br>
reconnection, still busy with 1 active RPCs<br>
Apr 28 14:53:24 10.7.7.9 kernel: Lustre: Skipped 19 previous similar<br>
messages<br>
Apr 28 14:53:24 10.7.7.9 kernel: Lustre: Skipped 19 previous similar<br>
messages<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170517/c0f848ba/attachment.htm" target="_blank" rel="noreferrer">http://lists.lustre.org/<wbr>pipermail/lustre-discuss-<wbr>lustre.org/attachments/<wbr>20170517/c0f848ba/attachment.<wbr>htm</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank" rel="noreferrer">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
<br>
------------------------------<br>
<br>
End of lustre-discuss Digest, Vol 134, Issue 36<br>
******************************<wbr>*****************<br>
</blockquote></div><br></div>