<div dir="ltr"><div class="gmail_default" style="font-size:small">Yep, looks like that's indeed the issue. Reducing peer_credits to 42 makes the problem go away.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Thanks,</div><div class="gmail_default" style="font-size:small">Kevin<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 13, 2020 at 4:25 PM <<a href="mailto:lustre-discuss-request@lists.lustre.org">lustre-discuss-request@lists.lustre.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send lustre-discuss mailing list submissions to<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-discuss-owner@lists.lustre.org" target="_blank">lustre-discuss-owner@lists.lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of lustre-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Lustre 2.12.3 client can't mount filesystem (Weiss, Karsten)<br>
2. Re: Lustre 2.12.3 client can't mount filesystem<br>
(Kevin M. Hildebrand)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Thu, 13 Feb 2020 08:11:08 +0000<br>
From: "Weiss, Karsten" <<a href="mailto:karsten.weiss@atos.net" target="_blank">karsten.weiss@atos.net</a>><br>
To: "<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
Subject: Re: [lustre-discuss] Lustre 2.12.3 client can't mount<br>
filesystem<br>
Message-ID: <<a href="mailto:cd1d4d54bbb4499998867447d1b8b56b@atos.net" target="_blank">cd1d4d54bbb4499998867447d1b8b56b@atos.net</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Hi,<br>
<br>
this is probably <a href="https://jira.whamcloud.com/browse/LU-12901" rel="noreferrer" target="_blank">https://jira.whamcloud.com/browse/LU-12901</a> which is still open and was just postponed to Lustre 2.14.0.<br>
<br>
Reducing peer_credits to 42 is a workaround.<br>
<br>
Best regards,<br>
Karsten<br>
<br>
From: lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> On Behalf Of Andreas Dilger<br>
Sent: Wednesday, February 12, 2020 21:50<br>
To: Kevin M. Hildebrand <<a href="mailto:kevin@umd.edu" target="_blank">kevin@umd.edu</a>><br>
Cc: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
Subject: Re: [lustre-discuss] Lustre 2.12.3 client can't mount filesystem<br>
<br>
Can you please try 2.12.4, it was just released yesterday and has a number of fixes.<br>
<br>
<br>
On Feb 12, 2020, at 13:36, Kevin M. Hildebrand <<a href="mailto:kevin@umd.edu" target="_blank">kevin@umd.edu</a><mailto:<a href="mailto:kevin@umd.edu" target="_blank">kevin@umd.edu</a>>> wrote:<br>
<br>
I just updated some of my clients to RHEL 7.7, Lustre 2.12.3, MOFED 4.7.<br>
Server version is 2.10.8.<br>
<br>
I'm now getting errors mounting the filesystem on the client. In fact, I can't even do an 'lctl ping' to any of the servers without getting an I/O error.<br>
<br>
Debug logs show this message when I attempt an lctl ping:<br>
00000800:00020000:0.0:1581538955.090767:0:20471:0:(o2iblnd.c:941:kiblnd_create_conn()) Can't create QP: -12, send_wr: 32634, recv_wr: 254, send_sge: 2, recv_sge: 1<br>
<br>
# lctl list_nids<br>
10.11.80.65@o2ib3<mailto:<a href="mailto:10.11.80.65@o2ib3" target="_blank">10.11.80.65@o2ib3</a>><br>
# lctl ping 10.11.80.50@o2ib3<mailto:<a href="mailto:10.11.80.50@o2ib3" target="_blank">10.11.80.50@o2ib3</a>><br>
failed to ping 10.11.80.50@o2ib3<mailto:<a href="mailto:10.11.80.50@o2ib3" target="_blank">10.11.80.50@o2ib3</a>>: Input/output error<br>
<br>
Interestingly, if I do an 'lctl ping' to the client _from_ the server, the ping succeeds, and from that point on pings from client _to_ server work fine until the client is rebooted or lnet is reloaded.<br>
<br>
ko2iblnd parameters match on clients and servers, namely:<br>
options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1<br>
<br>
Anyone have any thoughts?<br>
<br>
Thanks,<br>
Kevin<br>
<br>
--<br>
Kevin Hildebrand<br>
University of Maryland<br>
Division of IT<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
Cheers, Andreas<br>
--<br>
Andreas Dilger<br>
Principal Lustre Architect<br>
Whamcloud<br>
<br>
<br>
<br>
<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200213/4ba9d033/attachment-0001.html" rel="noreferrer" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200213/4ba9d033/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Thu, 13 Feb 2020 08:24:30 -0500<br>
From: "Kevin M. Hildebrand" <<a href="mailto:kevin@umd.edu" target="_blank">kevin@umd.edu</a>><br>
To: Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>><br>
Cc: "<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
Subject: Re: [lustre-discuss] Lustre 2.12.3 client can't mount<br>
filesystem<br>
Message-ID:<br>
<<a href="mailto:CAJmU7QmAmoYmb5ZaVYMeFPNi2p2qxOkTczM2bzKDbRzB9TNwtg@mail.gmail.com" target="_blank">CAJmU7QmAmoYmb5ZaVYMeFPNi2p2qxOkTczM2bzKDbRzB9TNwtg@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Ok, I just tried 2.12.4, and the problem still persists. The only<br>
difference I see now is that the error messages are appearing in syslog<br>
instead of needing to pull them from the debug log.<br>
[ 230.413761] LNetError: 1423:0:(o2iblnd.c:941:kiblnd_create_conn()) Can't<br>
create QP: -12, send_wr: 32634, recv_wr: 254, send_sge: 2, recv_sge: 1<br>
<br>
Thanks,<br>
Kevin<br>
<br>
On Wed, Feb 12, 2020 at 3:50 PM Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>><br>
wrote:<br>
<br>
> Can you please try 2.12.4, it was just released yesterday and has a number<br>
> of fixes.<br>
><br>
> On Feb 12, 2020, at 13:36, Kevin M. Hildebrand <<a href="mailto:kevin@umd.edu" target="_blank">kevin@umd.edu</a>> wrote:<br>
><br>
> I just updated some of my clients to RHEL 7.7, Lustre 2.12.3, MOFED 4.7.<br>
> Server version is 2.10.8.<br>
><br>
> I'm now getting errors mounting the filesystem on the client. In fact, I<br>
> can't even do an 'lctl ping' to any of the servers without getting an I/O<br>
> error.<br>
><br>
> Debug logs show this message when I attempt an lctl ping:<br>
> 00000800:00020000:0.0:1581538955.090767:0:20471:0:(o2iblnd.c:941:kiblnd_create_conn())<br>
> Can't create QP: -12, send_wr: 32634, recv_wr: 254, send_sge: 2, recv_sge: 1<br>
><br>
> # lctl list_nids<br>
> 10.11.80.65@o2ib3<br>
> # lctl ping 10.11.80.50@o2ib3<br>
> failed to ping 10.11.80.50@o2ib3: Input/output error<br>
><br>
> Interestingly, if I do an 'lctl ping' to the client _from_ the server, the<br>
> ping succeeds, and from that point on pings from client _to_ server work<br>
> fine until the client is rebooted or lnet is reloaded.<br>
><br>
> ko2iblnd parameters match on clients and servers, namely:<br>
> options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024<br>
> concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048<br>
> fmr_flush_trigger=512 fmr_cache=1<br>
><br>
> Anyone have any thoughts?<br>
><br>
> Thanks,<br>
> Kevin<br>
><br>
> --<br>
> Kevin Hildebrand<br>
> University of Maryland<br>
> Division of IT<br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
><br>
><br>
> Cheers, Andreas<br>
> --<br>
> Andreas Dilger<br>
> Principal Lustre Architect<br>
> Whamcloud<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200213/452b1c88/attachment-0001.html" rel="noreferrer" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200213/452b1c88/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
<br>
------------------------------<br>
<br>
End of lustre-discuss Digest, Vol 167, Issue 14<br>
***********************************************<br>
</blockquote></div>