<div><div dir="auto">Can’t you also use tcpkill to kill the connections? I’ve used it to kill “stuck” NFS connections before due to MTU related issues, etc.</div><div dir="auto"><br></div><div dir="auto">It’s distributed normally with the dsniff package if you are using an rpm based distribution.</div><div dir="auto"><br></div><div dir="auto">-Steve</div></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 20, 2020 at 20:39 NeilBrown <<a href="mailto:neilb@suse.com">neilb@suse.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><br>
I haven't tried this, but you might be able to close a socket by:<br>
<br>
1/ decrease /proc/sys/net/ipv4/tcp_keepalive_time<br>
so that keep-alives get sent sooner. Maybe set to 60,<br>
set ..._intvl to 5, and _probes to 3.<br>
<br>
2/ create a rule with iptables to drop all messages sent<br>
on the particular connection.<br>
iptables -A OUTPUT -m multiport --dports ... -sports .. -j DROP<br>
<br>
<br>
Given the suggested keep alive settings, you should only have to wait 75<br>
seconds after creating the IP tables rule before the connection is<br>
broken.<br>
<br>
NeilBrown<br>
<br>
<br>
On Thu, Feb 20 2020, Degremont, Aurelien wrote:<br>
<br>
> Thanks. It feels like the theory is valid.<br>
> Ideally to confirm I would need a way to manually force close the socklnd socket to force the other peer to re-established it.<br>
> Could not find a way to do it for socket opened by kernel threads.<br>
><br>
> Le 19/02/2020 23:12, « NeilBrown » <<a href="mailto:neilb@suse.com" target="_blank">neilb@suse.com</a>> a écrit :<br>
><br>
> <br>
> When LNet wants to send a message over a SOCKLND interface,<br>
> ksocknal_launch_packet() is called.<br>
> <br>
> This calls ksocknal_launch_all_connections_locked()<br>
> This loops over all "routes" to the "peer" to make sure they all have<br>
> "connections".<br>
> If it finds a route without a connection (returned by<br>
> ksocknal_find_connectable_route_locked()) it calls<br>
> ksocknal_launch_connection_locked() which adds the connection request to<br>
> ksnd_connd_routes, and wakes up the connd. The connd thread will then<br>
> make the connection.<br>
> <br>
> Hope that helps.<br>
> <br>
> NeilBrown<br>
> <br>
> <br>
> <br>
> On Wed, Feb 19 2020, Degremont, Aurelien wrote:<br>
> <br>
> > Thanks! That's really interesting.<br>
> > Do you have a code pointer that could show where the code will establish this connection if missing?<br>
> ><br>
> > Le 18/02/2020 23:34, « NeilBrown » <<a href="mailto:neilb@suse.com" target="_blank">neilb@suse.com</a>> a écrit :<br>
> ><br>
> > <br>
> > It is not true that:<br>
> > LNET will established connections only if asked for by upper layers.<br>
> > <br>
> > or at least, not in the sense that the upper layers ask for a<br>
> > connection.<br>
> > Lustre knows nothing about connections. Even LNet doesn't really know<br>
> > about connections. It is only at the socklnd level that connections mean<br>
> > much.<br>
> > <br>
> > Lustre and LNet are message-passing protocols.<br>
> > Lustre asks LNet to send a message to a given peer, and gives some<br>
> > details of the sort of reply to expect.<br>
> > LNet chooses a route and thus a network interface, and asked the LND to<br>
> > send the message.<br>
> > The socklnd LND will see if it already has a TCP connection. If it<br>
> > does, it will use it. If not, it will create one.<br>
> > <br>
> > So yes : it is exactly:<br>
> > possible that the server in this case opens the connection itself<br>
> > without waiting for the client to reconnect?<br>
> > <br>
> > NeilBrown<br>
> > <br>
> > <br>
> > On Tue, Feb 18 2020, Aurelien Degremont wrote:<br>
> > <br>
> > > Thanks for your reply.<br>
> > > I think I have a good enough understanding of LNET itself. My question was more about how LNET is being used by Lustre itself.<br>
> > ><br>
> > > LNET will established connections only if asked for by upper layers. <br>
> > > When I was talking about client and server, I was talking about how Lustre was using it.<br>
> > ><br>
> > > As far as I understood, Lustre server only contact clients when they need to send LDLM callbacks.<br>
> > > They do so through the socket already opened by the client (reverse import).<br>
> > > What happened if the socket is closed is what I'm not sure. I though the server is rather waiting for the client to reconnect and if not, is more or less evicting it.<br>
> > > Could it be possible that the server in this case opens the connection itself without waiting for the client to reconnect?<br>
> > ><br>
> > ><br>
> > > Aurélien<br>
> > ><br>
> > > Le 18/02/2020 05:42, « NeilBrown » <<a href="mailto:neilb@suse.com" target="_blank">neilb@suse.com</a>> a écrit :<br>
> > ><br>
> > > <br>
> > > LNet is a peer-to-peer protocol, it has no concept of client and server.<br>
> > > If one host needs to send a message to another but doesn't already have<br>
> > > a connection, it creates a new connection.<br>
> > > I don't yet know enough specifics of the lustre protocol to be certain<br>
> > > of the circumstances when a lustre server will need to initiate a message<br>
> > > to a client, but I imagine that recalling a lock might be one.<br>
> > > <br>
> > > I think you should assume that any LNet node might receive a connection<br>
> > > from any other LNet node (for which they share an LNet network), and<br>
> > > that the connection could come from any port between 512 and 1023<br>
> > > (LNET_ACCEPTOR_MIN_PORT to LNET_ACCEPTOR_MAX_PORT).<br>
> > > <br>
> > > NeilBrown<br>
> > > <br>
> > > <br>
> > > <br>
> > > On Mon, Feb 17 2020, Degremont, Aurelien wrote:<br>
> > > <br>
> > > > Hi all,<br>
> > > ><br>
> > > > From what I've understood so far, LNET listens on port 988 by default and peers connect to it using 1021-1023 TCP ports as source ports.<br>
> > > > At Lustre level, servers listen on 988 and clients connect to them using the same source ports 1021-1023.<br>
> > > > So only accepting connections to port 988 on server side sounded pretty safe to me. However, I've seen connections from 1021-1023 to 988, from server hosts to client hosts sometimes.<br>
> > > > I can't understand what mechanism could trigger these connections. Did I miss something?<br>
> > > ><br>
> > > > Thanks<br>
> > > ><br>
> > > > Aurélien<br>
> > > ><br>
> > > > _______________________________________________<br>
> > > > lustre-discuss mailing list<br>
> > > > <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> > > > <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
> > > <br>
> > <br>
> <br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><p style="line-height:1.44;margin-top:0pt;margin-bottom:0pt"><font color="#000000" face="Arial"><span style="font-size:13.3333px;white-space:pre-wrap"><b>Steve Crusan</b></span></font></p><p style="line-height:1.44;margin-top:0pt;margin-bottom:0pt"><font color="#000000" face="Arial"><span style="font-size:13.3333px;white-space:pre-wrap">Storage Specialist</span></font></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"> </p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"> </p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh4.googleusercontent.com/Q4KHsGT4WoIleJUBVa2XGPnpzVoA_Y7tryRXSNSrbOtcgRxd8-JxxBfGWLnil2ZCupCxSUihV8wGLl1rUn_71nT3xLC0CBUysFoJk1coZ9IT0TBSoPMuRiKxxwWyZyAKYEfOYyCH" width="118" height="73" style="border:none"></span></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"> </p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(102,102,102);font-weight:700;vertical-align:baseline;white-space:pre-wrap">DownUnder GeoSolutions</span></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"> </p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap">16200 Park Row Drive, Suite 100</span></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap">Houston TX 77084, USA</span></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(102,102,102);font-weight:700;font-style:italic;vertical-align:baseline;white-space:pre-wrap">tel </span><span style="font-size:10pt;font-family:Arial;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap">+1 832 582 3221</span></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><a href="mailto:stevec@dug.com" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:10pt;font-family:Arial;font-weight:700;vertical-align:baseline;white-space:pre-wrap">stevec@dug.com</span></a></p><p style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"></p><p dir="ltr" style="font-family:arial,helvetica,sans-serif;line-height:1.44;margin-top:0pt;margin-bottom:0pt"><a href="http://www.dug.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:10pt;font-family:Arial;font-weight:700;vertical-align:baseline;white-space:pre-wrap">www.dug.com</span></a></p></div></div>