[lustre-discuss] LNET ports and connections

[NeilBrown]

When LNet wants to send a message over a SOCKLND interface,
ksocknal_launch_packet() is called.

This calls ksocknal_launch_all_connections_locked()
This loops over all "routes" to the "peer" to make sure they all have
"connections".
If it finds a route without a connection (returned by
ksocknal_find_connectable_route_locked()) it calls
ksocknal_launch_connection_locked() which adds the connection request to
ksnd_connd_routes, and wakes up the connd.  The connd thread will then
make the connection.

Hope that helps.

NeilBrown



On Wed, Feb 19 2020, Degremont, Aurelien wrote:

> Thanks! That's really interesting.
> Do you have a code pointer that could show where the code will establish this connection if missing?
>
> Le 18/02/2020 23:34, « NeilBrown » <neilb at suse.com> a écrit :
>
>     
>     It is not true that:
>        LNET will established connections only if asked for by upper layers.
>     
>     or at least, not in the sense that the upper layers ask for a
>     connection.
>     Lustre knows nothing about connections.  Even LNet doesn't really know
>     about connections. It is only at the socklnd level that connections mean
>     much.
>     
>     Lustre and LNet are message-passing protocols.
>     Lustre asks LNet to send a message to a given peer, and gives some
>     details of the sort of reply to expect.
>     LNet chooses a route and thus a network interface, and asked the LND to
>     send the message.
>     The socklnd LND will see if it already has a TCP connection.  If it
>     does, it will use it.  If not, it will create one.
>     
>     So yes : it is exactly:
>       possible that the server in this case opens the connection itself
>       without waiting for the client to reconnect?
>     
>     NeilBrown
>     
>     
>     On Tue, Feb 18 2020, Aurelien Degremont wrote:
>     
>     > Thanks for your reply.
>     > I think I have a good enough understanding of LNET itself. My question was more about how LNET is being used by Lustre itself.
>     >
>     > LNET will established connections only if asked for by upper layers. 
>     > When I was talking about client and server, I was talking about how Lustre was using it.
>     >
>     > As far as I understood, Lustre server only contact clients when they need to send LDLM callbacks.
>     > They do so through the socket already opened by the client (reverse import).
>     > What happened if the socket is closed is what I'm not sure. I though the server is rather waiting for the client to reconnect and if not, is more or less evicting it.
>     > Could it be possible that the server in this case opens the connection itself without waiting for the client to reconnect?
>     >
>     >
>     > Aurélien
>     >
>     > Le 18/02/2020 05:42, « NeilBrown » <neilb at suse.com> a écrit :
>     >
>     >     
>     >     LNet is a peer-to-peer protocol, it has no concept of client and server.
>     >     If one host needs to send a message to another but doesn't already have
>     >     a connection, it creates a new connection.
>     >     I don't yet know enough specifics of the lustre protocol to be certain
>     >     of the circumstances when a lustre server will need to initiate a message
>     >     to a client, but I imagine that recalling a lock might be one.
>     >     
>     >     I think you should assume that any LNet node might receive a connection
>     >     from any other LNet node (for which they share an LNet network), and
>     >     that the connection could come from any port between 512 and 1023
>     >     (LNET_ACCEPTOR_MIN_PORT to LNET_ACCEPTOR_MAX_PORT).
>     >     
>     >     NeilBrown
>     >     
>     >     
>     >     
>     >     On Mon, Feb 17 2020, Degremont, Aurelien wrote:
>     >     
>     >     > Hi all,
>     >     >
>     >     > From what I've understood so far, LNET listens on port 988 by default and peers connect to it using 1021-1023 TCP ports as source ports.
>     >     > At Lustre level, servers listen on 988 and clients connect to them using the same source ports 1021-1023.
>     >     > So only accepting connections to port 988 on server side sounded pretty safe to me. However, I've seen connections from 1021-1023 to 988, from server hosts to client hosts sometimes.
>     >     > I can't understand what mechanism could trigger these connections. Did I miss something?
>     >     >
>     >     > Thanks
>     >     >
>     >     > Aurélien
>     >     >
>     >     > _______________________________________________
>     >     > lustre-discuss mailing list
>     >     > lustre-discuss at lists.lustre.org
>     >     > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>     >     
>     
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200220/54854233/attachment.sig>