<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Jesse,
<div class="">what Lustre version are you using? I'm wondering if this is an issue that has already been fixed, or if there is a bug with the peer discovery in some cases that needs to be fixed?<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 10, 2024, at 07:40, Jesse Stroik <<a href="mailto:jesse.stroik@ssec.wisc.edu" class="">jesse.stroik@ssec.wisc.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">[You don't often get email from <a href="mailto:jesse.stroik@ssec.wisc.edu" class="">
jesse.stroik@ssec.wisc.edu</a>. Learn why this is important at <a href="https://aka.ms/LearnAboutSenderIdentification" class="">
https://aka.ms/LearnAboutSenderIdentification</a> ]<br class="">
<br class="">
I spent a good deal of time tracking down and cleaning up infiniband errors (we had some bad cables) but that didn't matter. The solution was to disable peer discovery on all of the ko2iblnd hosts because they were mistakenly showing up as multi-rail in some
cases. I suspect it's related to multi-port IB HCAs.<br class="">
<br class="">
For this to work consistently we had to disable discovery, clear up all peers on the hosts, then let the lustre file systems mount.<br class="">
<br class="">
Thanks for lending me a bit of your experience, Shane.<br class="">
<br class="">
Jesse<br class="">
<br class="">
________________________________________<br class="">
From: Nehring, Shane R [ITS] <<a href="mailto:snehring@iastate.edu" class="">snehring@iastate.edu</a>><br class="">
Sent: Wednesday, December 4, 2024 11:08 AM<br class="">
To: Jesse Stroik; <a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
Subject: Re: Connectivity issues after client crash<br class="">
<br class="">
No, we're using o2iblnd primarily now, we just used socklnd until we were able to migrate the bulk of nodes to infiniband. There are a small number of nodes that are still using socklnd either via ethernet or remaining opa nodes.<br class="">
<br class="">
Our cluster is relatively small and everything is single rail, so discovery isn't necessary in our case.<br class="">
<br class="">
<br class="">
<br class="">
________________________________<br class="">
From: Jesse Stroik <<a href="mailto:jesse.stroik@ssec.wisc.edu" class="">jesse.stroik@ssec.wisc.edu</a>><br class="">
Sent: Tuesday, December 3, 2024 18:18<br class="">
To: Nehring, Shane R [ITS] <<a href="mailto:snehring@iastate.edu" class="">snehring@iastate.edu</a>>;
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a> <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
Subject: Re: Connectivity issues after client crash<br class="">
<br class="">
Thanks for your reply.<br class="">
<br class="">
I did consider disabling peer discovery on the servers for one of the file systems as a test, but did not yet test it. i'll keep that idea in my back pocket in case the issue recurs with our workaround or we have the issue happen on any of our other 2.15 file
systems.<br class="">
<br class="">
Did you use your ib equipment just using ksocklnd and forgo rdma?<br class="">
<br class="">
For us, this would be a fairly major change as our clusters and lustre systems are all on infiniband directly, but we have a lot of ethernet clients as well. And we have seven lustre file systems most of which are routed to many ethernet clients.<br class="">
<br class="">
Jesse<br class="">
<br class="">
________________________________________<br class="">
From: Nehring, Shane R [ITS]<br class="">
Sent: Tuesday, December 3, 2024 2:50 PM<br class="">
To: <a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>; Jesse Stroik<br class="">
Subject: Re: Connectivity issues after client crash<br class="">
<br class="">
<br class="">
Hello Jesse,<br class="">
<br class="">
What I think we may have been hitting in this particular case was<br class="">
<a href="https://jira.whamcloud.com/browse/LU-16349<https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-16349__;!!Mak6IKo!KjIS-mlygGvZXA84UL7NpPd6BbzEHT4hP-NTUTL4RmVJ2GjVu_zPrP0oO1MAkx3CWy4hebKZ4MYH7r4O-XxnkKucCgg$>" class="">https://jira.whamcloud.com/browse/LU-16349<https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-16349__;!!Mak6IKo!KjIS-mlygGvZXA84UL7NpPd6BbzEHT4hP-NTUTL4RmVJ2GjVu_zPrP0oO1MAkx3CWy4hebKZ4MYH7r4O-XxnkKucCgg$></a>
which we in the short term worked<br class="">
around by switching to ksocklnd from o2iblnd, and then ultimately mostly moved<br class="">
away from omnipath to ib.<br class="">
<br class="">
I've seen some weird behavior in our environment around peer discovery (we have<br class="">
tcp and o2ib nids on the oss and mds nodes to provide access over both methods)<br class="">
that've caused problems in the past, such that we just outright disable peer<br class="">
discovery with modprobe (eg options lnet lnet_peer_discovery_disabled=1 in<br class="">
/etc/modprobe.d/lnet.conf). I dunno if you've tried that yet, but if not it<br class="">
might be a place to start (assuming you don't need automatic multi-rail)<br class="">
<br class="">
Shane<br class="">
<br class="">
On Tue, 2024-12-03 at 20:29 +0000, Jesse Stroik wrote:<br class="">
<blockquote type="cite" class="">Hi Shane,<br class="">
<br class="">
I realize this is quite an old post but I think it is worth responding for<br class="">
posterity and because I suspect others who upgrade may run into this issue.<br class="">
<br class="">
I'm observing some similar issues to what you describe. They started this<br class="">
weekend for us on two of our servers which were upgraded to rocky 8 and lustre<br class="">
2.15.5 a couple of months ago. Like you, we run infiniband. We also route to<br class="">
ethernet clients and our lustre routers are now running rocky 8 and either<br class="">
lustre 2.15.5 or 2.15.3.<br class="">
<br class="">
The behavior that we observe is widely similar to yours though we have a few<br class="">
observations that may be relevant. I suspect that if we can limit the<br class="">
infiniband communication to certain segments or LIDs it seems to behave fine.<br class="">
<br class="">
Today i was able to reproduce the problem when routing to an ethernet client<br class="">
that ran our robinhood policy engine. With the file systems behaving normally<br class="">
to our infiniband network clients, I enabled routing on them again and tested<br class="">
with all lrouters enabled or just individual lrouters enabled. In all cases<br class="">
that i tested, mounting one of those file systems resulted in the<br class="">
communication problems on the MDS and OSS servers for that file system. In one<br class="">
case an MDS panicked as soon as that client mounted its file system. I put an<br class="">
IB card into that policy engine client, removed all lnet routing for those<br class="">
file systems and lnet behavior was stable with that client mounting those file<br class="">
systems and ingesting the changelogs.<br class="">
<br class="">
When the servers have failed lnet communication, they can sometimes be fixed<br class="">
by stopping the lustre service / unmounting the OSTs and MDTs and restarting /<br class="">
reconfiguring lnet. If i'm unable to do that, rebooting the affected systems<br class="">
seems to be the only solution and they sometimes hang indefinitely trying to<br class="">
shut down lnet necessitating a power cycle.<br class="">
<br class="">
When the issue first started I did suspect the infiniband fabric and looked<br class="">
for problems with that, though, like you, I didn't find anything conclusive. A<br class="">
recent change we've had is our OpenSM subnet manager definitely restarted and<br class="">
moved when we updated the kernel and lustre-client versions on two of our<br class="">
lrouters. However, even when lnet is unable to communicate, the nodes can all<br class="">
still ping each other with ipv4 ping IPoIB which suggests the subnet manager<br class="">
and infiniband are working.<br class="">
<br class="">
I did also attempt manual lnet peer discovery on individual systems while they<br class="">
were affected to see if i could recover them that way. Discovery failed on<br class="">
those systems, but only when we were already observing the communication<br class="">
problem with lnet.<br class="">
<br class="">
Jesse<br class="">
<br class="">
________________________________________<br class="">
From: lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" class="">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of<br class="">
Nehring, Shane R [LAS] via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
Sent: Wednesday, December 21, 2022 11:49 AM<br class="">
To: <a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
Subject: [lustre-discuss] Connectivity issues after client crash<br class="">
<br class="">
Hello all,<br class="">
<br class="">
I'm hoping that someone might be able to help me with an issue I've been<br class="">
seeing<br class="">
periodically since updating to 2.15.x.<br class="">
<br class="">
Back in July we remade our scratch storage volume with 2.15 after running 2.12<br class="">
for a long time. As part of the upgrade we reinstalled our oss and mds nodes<br class="">
with rhel 8 so we'd finally be able to take advantage of project quotas. For<br class="">
background, our nodes are connected via omnipath and we have a mix of el7<br class="">
clients, el8 clients, and el9 clients. We've been piecemeal updating our<br class="">
clients<br class="">
to el9 over the past few months with the goal of being 99% el9 clients by mid<br class="">
January.<br class="">
<br class="">
Since upgrading we've seen recurring connectivity issues arise in the cluster<br class="">
from time to time which seem to be very strongly correlated with a client<br class="">
crashing. The fabric itself seems fine. There's no evidence of error or any<br class="">
packet loss, so I cannot confidently blame it.<br class="">
<br class="">
On an oss that's having trouble communicating we see the following messages<br class="">
for<br class="">
various osts:<br class="">
<br class="">
kernel: LNetError: 31282:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed<br class="">
out tx: active_txs(WSQ:100), 19 seconds<br class="">
kernel: LNetError: 31282:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked())<br class="">
Skipped<br class="">
6 previous similar messages<br class="">
kernel: LNetError: 31282:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out<br class="">
RDMA with 172.16.100.19@o2ib (4): c: 31, oc: 0, rc: 31<br class="">
kernel: LNetError: 31282:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Skipped 6<br class="">
previous similar messages<br class="">
kernel: LustreError: 109351:0:(ldlm_lib.c:3543:target_bulk_io()) @@@ network<br class="">
error on bulk WRITE req@00000000f848e208 x1751146351599040/t0(0) o4-<br class="">
<blockquote type="cite" class=""><a href="mailto:10ccde33-01ef-47cf-873d-da9a1b6bb1ea@172.16.100.19" class="">10ccde33-01ef-47cf-873d-da9a1b6bb1ea@172.16.100.19</a>@o2ib:19/0 lens 488/448 e<br class="">
0<br class="">
</blockquote>
to 0 dl 1671634949 ref 1 fl Interpret:/2/0 rc 0/0 job:'873032'<br class="">
kernel: Lustre: work-OST0007: Bulk IO write error with 10ccde33-01ef-47cf-<br class="">
873d-<br class="">
da9a1b6bb1ea (at 172.16.100.19@o2ib), client will retry: rc = -110<br class="">
kernel: Lustre: Skipped 5 previous similar messages<br class="">
kernel: LustreError: 109351:0:(ldlm_lib.c:3543:target_bulk_io()) @@@ network<br class="">
error on bulk WRITE req@000000008176189a x1751153771751616/t0(0) o4-<br class="">
<blockquote type="cite" class=""><a href="mailto:63e88958-d0b1-4c7f-8413-da96b181cd92@172.16.100.25" class="">63e88958-d0b1-4c7f-8413-da96b181cd92@172.16.100.25</a>@o2ib:94/0 lens 488/448 e<br class="">
0<br class="">
</blockquote>
to 0 dl 1671635024 ref 1 fl Interpret:/0/0 rc 0/0 job:'873641'<br class="">
Lustre: work-OST000d: Bulk IO write error with 63e88958-d0b1-4c7f-8413-<br class="">
da96b181cd92 (at 172.16.100.25@o2ib), client will retry: rc = -110<br class="">
<br class="">
An already connected client will see messages along the lines of:<br class="">
<br class="">
kernel: Lustre: 3736:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request<br class="">
sent has timed out for slow reply: [sent 1671634931/real 1671634931]<br class="">
req@ffff94a0ed490000 x1751139078233344/t0(0) o9-<br class="">
<blockquote type="cite" class=""><a href="mailto:work-OST0001-osc-ffff95552570a000@172.16.100.253" class="">work-OST0001-osc-ffff95552570a000@172.16.100.253</a>@o2ib:28/4 lens 224/224 e 0<br class="">
to<br class="">
</blockquote>
1<br class="">
<br class="">
A stack trace on one of the clients of an ls against the volume that hangs:<br class="">
<br class="">
[<ffffffffc1576d05>] cl_sync_io_wait+0x1c5/0x480 [obdclass]<br class="">
[<ffffffffc1572943>] cl_lock_request+0x1d3/0x210 [obdclass]<br class="">
[<ffffffffc17db1d9>] cl_glimpse_lock+0x329/0x380 [lustre]<br class="">
[<ffffffffc17db5a5>] cl_glimpse_size0+0x255/0x280 [lustre]<br class="">
[<ffffffffc1793cdc>] ll_getattr_dentry+0x50c/0x9c0 [lustre]<br class="">
[<ffffffffc17941ae>] ll_getattr+0x1e/0x20 [lustre]<br class="">
[<ffffffff99a53d49>] vfs_getattr+0x49/0x80<br class="">
[<ffffffff99a53e55>] vfs_fstatat+0x75/0xc0<br class="">
[<ffffffff99a54261>] SYSC_newlstat+0x31/0x60<br class="">
[<ffffffff99a546ce>] SyS_newlstat+0xe/0x10<br class="">
[<ffffffff99f99f92>] system_call_fastpath+0x25/0x2a<br class="">
[<ffffffffffffffff>] 0xffffffffffffffff<br class="">
<br class="">
If we try to reboot a client to clear the issue it won't be able to mount the<br class="">
filesystem; the mount process hangs in D until it times out. A stack trace<br class="">
from<br class="">
the mount:<br class="">
[<0>] llog_process_or_fork+0x2de/0x570 [obdclass]<br class="">
[<0>] llog_process+0x10/0x20 [obdclass]<br class="">
[<0>] class_config_parse_llog+0x1eb/0x3e0 [obdclass]<br class="">
[<0>] mgc_process_cfg_log+0x659/0xc90 [mgc]<br class="">
[<0>] mgc_process_log+0x667/0x800 [mgc]<br class="">
[<0>] mgc_process_config+0x42b/0x6e0 [mgc]<br class="">
[<0>] obd_process_config.constprop.0+0x76/0x1a0 [obdclass]<br class="">
[<0>] lustre_process_log+0x562/0x8f0 [obdclass]<br class="">
[<0>] ll_fill_super+0x6ec/0x1020 [lustre]<br class="">
[<0>] lustre_fill_super+0xe4/0x470 [lustre]<br class="">
[<0>] mount_nodev+0x41/0x90<br class="">
[<0>] legacy_get_tree+0x24/0x40<br class="">
[<0>] vfs_get_tree+0x22/0xb0<br class="">
[<0>] do_new_mount+0x176/0x310<br class="">
[<0>] __x64_sys_mount+0x103/0x140<br class="">
[<0>] do_syscall_64+0x38/0x90<br class="">
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae<br class="">
<br class="">
<br class="">
When this has happened in the past we've had to shut down the entire lustre<br class="">
backend, and if we're lucky clients would start recovering once everything was<br class="">
back up again. The last time this happened we actually had to shut down the<br class="">
entire cluster.<br class="">
<br class="">
In this particular case rebooting the osses and mds/mgs has stopped the errors<br class="">
on the server side, but el9 and el8 clients are now seeing:<br class="">
<br class="">
kernel: LNetError: 149017:0:(o2iblnd_cb.c:966:kiblnd_post_tx_locked()) Error -<br class="">
22<br class="">
posting transmit to 172.16.100.250@o2ib<br class="">
kernel: LNetError: 149017:0:(o2iblnd_cb.c:966:kiblnd_post_tx_locked()) Skipped<br class="">
31 previous similar messages<br class="">
and<br class="">
LustreError: 3585788:0:(file.c:5096:ll_inode_revalidate_fini()) work:<br class="">
revalidate<br class="">
FID [0x200000007:0x1:0x0] error: rc = -4<br class="">
<br class="">
el7 clients have recovered seemingly without issue.<br class="">
<br class="">
Wondering if anyone might have any suggestions on where to look as to why this<br class="">
is breaking, or pointers as to how to recover from this situation without<br class="">
rebooting if possible.<br class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mak6IKo!KjIS-mlygGvZXA84UL7NpPd6BbzEHT4hP-NTUTL4RmVJ2GjVu_zPrP0oO1MAkx3CWy4hebKZ4MYH7r4O-Xxn04MGbMU$><br class="">
</blockquote>
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>