<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Very nice one! Thanks!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">I was able to reproduce and confirm that with the proper sequence, the LNET connections will be initiated from server to client 988 port.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Thanks all!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span style="font-size:12.0pt;color:black">De :
</span></b><span style="font-size:12.0pt;color:black">Steve Crusan <stevec@dug.com><br>
<b>Date : </b>vendredi 21 février 2020 à 03:53<br>
<b>À : </b>NeilBrown <neilb@suse.com><br>
<b>Cc : </b>"Degremont, Aurelien" <degremoa@amazon.com>, "lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Objet : </b>Re: [lustre-discuss] LNET ports and connections<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Can\u2019t you also use tcpkill to kill the connections? I\u2019ve used it to kill \u201cstuck\u201d NFS connections before due to MTU related issues, etc.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">It\u2019s distributed normally with the dsniff package if you are using an rpm based distribution.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">-Steve<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Thu, Feb 20, 2020 at 20:39 NeilBrown <<a href="mailto:neilb@suse.com">neilb@suse.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><br>
I haven't tried this, but you might be able to close a socket by:<br>
<br>
1/ decrease /proc/sys/net/ipv4/tcp_keepalive_time<br>
so that keep-alives get sent sooner. Maybe set to 60,<br>
set ..._intvl to 5, and _probes to 3.<br>
<br>
2/ create a rule with iptables to drop all messages sent<br>
on the particular connection.<br>
iptables -A OUTPUT -m multiport --dports ... -sports .. -j DROP<br>
<br>
<br>
Given the suggested keep alive settings, you should only have to wait 75<br>
seconds after creating the IP tables rule before the connection is<br>
broken.<br>
<br>
NeilBrown<br>
<br>
<br>
On Thu, Feb 20 2020, Degremont, Aurelien wrote:<br>
<br>
> Thanks. It feels like the theory is valid.<br>
> Ideally to confirm I would need a way to manually force close the socklnd socket to force the other peer to re-established it.<br>
> Could not find a way to do it for socket opened by kernel threads.<br>
><br>
> Le 19/02/2020 23:12, « NeilBrown » <<a href="mailto:neilb@suse.com" target="_blank">neilb@suse.com</a>> a écrit :<br>
><br>
> <br>
> When LNet wants to send a message over a SOCKLND interface,<br>
> ksocknal_launch_packet() is called.<br>
> <br>
> This calls ksocknal_launch_all_connections_locked()<br>
> This loops over all "routes" to the "peer" to make sure they all have<br>
> "connections".<br>
> If it finds a route without a connection (returned by<br>
> ksocknal_find_connectable_route_locked()) it calls<br>
> ksocknal_launch_connection_locked() which adds the connection request to<br>
> ksnd_connd_routes, and wakes up the connd. The connd thread will then<br>
> make the connection.<br>
> <br>
> Hope that helps.<br>
> <br>
> NeilBrown<br>
> <br>
> <br>
> <br>
> On Wed, Feb 19 2020, Degremont, Aurelien wrote:<br>
> <br>
> > Thanks! That's really interesting.<br>
> > Do you have a code pointer that could show where the code will establish this connection if missing?<br>
> ><br>
> > Le 18/02/2020 23:34, « NeilBrown » <<a href="mailto:neilb@suse.com" target="_blank">neilb@suse.com</a>> a écrit :<br>
> ><br>
> > <br>
> > It is not true that:<br>
> > LNET will established connections only if asked for by upper layers.<br>
> > <br>
> > or at least, not in the sense that the upper layers ask for a<br>
> > connection.<br>
> > Lustre knows nothing about connections. Even LNet doesn't really know<br>
> > about connections. It is only at the socklnd level that connections mean<br>
> > much.<br>
> > <br>
> > Lustre and LNet are message-passing protocols.<br>
> > Lustre asks LNet to send a message to a given peer, and gives some<br>
> > details of the sort of reply to expect.<br>
> > LNet chooses a route and thus a network interface, and asked the LND to<br>
> > send the message.<br>
> > The socklnd LND will see if it already has a TCP connection. If it<br>
> > does, it will use it. If not, it will create one.<br>
> > <br>
> > So yes : it is exactly:<br>
> > possible that the server in this case opens the connection itself<br>
> > without waiting for the client to reconnect?<br>
> > <br>
> > NeilBrown<br>
> > <br>
> > <br>
> > On Tue, Feb 18 2020, Aurelien Degremont wrote:<br>
> > <br>
> > > Thanks for your reply.<br>
> > > I think I have a good enough understanding of LNET itself. My question was more about how LNET is being used by Lustre itself.<br>
> > ><br>
> > > LNET will established connections only if asked for by upper layers.
<br>
> > > When I was talking about client and server, I was talking about how Lustre was using it.<br>
> > ><br>
> > > As far as I understood, Lustre server only contact clients when they need to send LDLM callbacks.<br>
> > > They do so through the socket already opened by the client (reverse import).<br>
> > > What happened if the socket is closed is what I'm not sure. I though the server is rather waiting for the client to reconnect and if not, is more or less evicting it.<br>
> > > Could it be possible that the server in this case opens the connection itself without waiting for the client to reconnect?<br>
> > ><br>
> > ><br>
> > > Aurélien<br>
> > ><br>
> > > Le 18/02/2020 05:42, « NeilBrown » <<a href="mailto:neilb@suse.com" target="_blank">neilb@suse.com</a>> a écrit :<br>
> > ><br>
> > > <br>
> > > LNet is a peer-to-peer protocol, it has no concept of client and server.<br>
> > > If one host needs to send a message to another but doesn't already have<br>
> > > a connection, it creates a new connection.<br>
> > > I don't yet know enough specifics of the lustre protocol to be certain<br>
> > > of the circumstances when a lustre server will need to initiate a message<br>
> > > to a client, but I imagine that recalling a lock might be one.<br>
> > > <br>
> > > I think you should assume that any LNet node might receive a connection<br>
> > > from any other LNet node (for which they share an LNet network), and<br>
> > > that the connection could come from any port between 512 and 1023<br>
> > > (LNET_ACCEPTOR_MIN_PORT to LNET_ACCEPTOR_MAX_PORT).<br>
> > > <br>
> > > NeilBrown<br>
> > > <br>
> > > <br>
> > > <br>
> > > On Mon, Feb 17 2020, Degremont, Aurelien wrote:<br>
> > > <br>
> > > > Hi all,<br>
> > > ><br>
> > > > From what I've understood so far, LNET listens on port 988 by default and peers connect to it using 1021-1023 TCP ports as source ports.<br>
> > > > At Lustre level, servers listen on 988 and clients connect to them using the same source ports 1021-1023.<br>
> > > > So only accepting connections to port 988 on server side sounded pretty safe to me. However, I've seen connections from 1021-1023 to 988, from server hosts to client hosts sometimes.<br>
> > > > I can't understand what mechanism could trigger these connections. Did I miss something?<br>
> > > ><br>
> > > > Thanks<br>
> > > ><br>
> > > > Aurélien<br>
> > > ><br>
> > > > _______________________________________________<br>
> > > > lustre-discuss mailing list<br>
> > > > <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">
lustre-discuss@lists.lustre.org</a><br>
> > > > <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
> > > <br>
> > <br>
> <br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</blockquote>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">-- <o:p></o:p></p>
<div>
<div>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Steve Crusan</span></b><o:p></o:p></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Storage Specialist</span><o:p></o:p></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<span style="font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<span style="font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black;border:solid windowtext 1.0pt;padding:0cm"><img border="0" width="118" height="73" style="width:1.2291in;height:.7604in" id="_x0000_i1025" src="cid:~WRD0000.jpg" alt="Image supprimée par l'expéditeur."></span></b><span style="font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<span style="font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#666666">DownUnder GeoSolutions</span></b><span style="font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<span style="font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#666666">16200 Park Row Drive, Suite 100</span><span style="font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#666666">Houston TX 77084, USA</span><span style="font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<b><i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#666666">tel
</span></i></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#666666">+1 832 582 3221</span><span style="font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<span style="font-family:"Arial",sans-serif"><a href="mailto:stevec@dug.com" target="_blank"><b><span style="font-size:10.0pt;color:#1155CC">stevec@dug.com</span></b></a><o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:35.4pt;margin-bottom:.0001pt">
<span style="font-family:"Arial",sans-serif"><a href="http://www.dug.com/" target="_blank"><b><span style="font-size:10.0pt;color:#1155CC">www.dug.com</span></b></a><o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>