[lustre-discuss] Full List of Required Open Lustre Ports?

[Andreas Dilger]

Ellis, the addition of dynamic conns_per_peer for TCP connections is relatively new.  There would be no performance "regression" against earlier Lustre releases (which always had the equivalent of conns_per_peer=1), just not additional performance gains for high-speed Ethernet interfaces.

The port selection is somewhat dependent on the software running on the server.  There is no guarantee that 1023/1022/... will be available for Lustre to use, if other TCP listeners have already been opened on those ports, unless you use something like "portreserve" (IIRC), but this is tricky to get right with kernel-side listeners (I don't recall if we ever got that right).

Whether you open those additional ports or just set conns_per_peer=1 is up to you.  You would definitely want to limit the accept rule so that the source or target is port 988.

Cheers, Andreas

On Feb 1, 2023, at 15:18, Ellis Wilson via lustre-discuss <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>> wrote:

Hi folks,

We've seen some weird stuff recently with UFW/iptables dropping packets on our OSS and MDS nodes.  We are running 2.15.1.  Example:

[   69.472030] [UFW BLOCK] IN=eth0 OUT= MAC=<snip> SRC=<snip> DST=<snip> LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58224 DF PROTO=TCP SPT=1022 DPT=988 WINDOW=510 RES=0x00 ACK FIN URGP=0

[11777.280724] [UFW BLOCK] IN=eth0 OUT= MAC=<snip> SRC=<snip> DST=<snip> LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=44206 DF PROTO=TCP SPT=988 DPT=1023 WINDOW=509 RES=0x00 ACK URGP=0

Previously, we were only allowing 988 bidirectionally on BOTH clients and servers.  This was based on guidance from the Lustre manual.  From the above messages it appears we may need to expand that range.  This thread discusses it:
https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg17229.html

Based on that thread and some code reading it appears that sans explicit configuration of conns_per_peer the extra ports potentially required are autotuning (ksocklnd_speed2cpp).  E.G., if we have a node with 50Gbps interface, we may need up to 3 ports open to accommodate the extra ports.  These appear to be selected beginning at 1023 and going down as far as 512.

Questions:
1. If we do not open up more than 988, are there known performance issues for machines at or below say, 50Gbps?  It does seem that with these closed we don't have correctness or visible performance problems, so there must be some fallback mechanism at play.
2. Can we just open 1023 to 1021 for a 50GigE machine?  Or are there situations where binding might fail and the algorithm could potentially attempt to create sockets all the way down to 512?
3. Regardless of the answer to #2, do we need to open these ports on all client and server nodes, or can we get away with just server nodes?
4. Do these need to be opened just for egress from the node in question, or bidirectionally?

Thanks in advance!

Best,

ellis
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20230202/edc830f9/attachment.htm>