[lustre-discuss] Full List of Required Open Lustre Ports?

[Ellis Wilson]

Hi folks,

We've seen some weird stuff recently with UFW/iptables dropping packets on our OSS and MDS nodes.  We are running 2.15.1.  Example:

[   69.472030] [UFW BLOCK] IN=eth0 OUT= MAC=<snip> SRC=<snip> DST=<snip> LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58224 DF PROTO=TCP SPT=1022 DPT=988 WINDOW=510 RES=0x00 ACK FIN URGP=0

[11777.280724] [UFW BLOCK] IN=eth0 OUT= MAC=<snip> SRC=<snip> DST=<snip> LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=44206 DF PROTO=TCP SPT=988 DPT=1023 WINDOW=509 RES=0x00 ACK URGP=0

Previously, we were only allowing 988 bidirectionally on BOTH clients and servers.  This was based on guidance from the Lustre manual.  From the above messages it appears we may need to expand that range.  This thread discusses it:
https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg17229.html

Based on that thread and some code reading it appears that sans explicit configuration of conns_per_peer the extra ports potentially required are autotuning (ksocklnd_speed2cpp).  E.G., if we have a node with 50Gbps interface, we may need up to 3 ports open to accommodate the extra ports.  These appear to be selected beginning at 1023 and going down as far as 512.

Questions:
1. If we do not open up more than 988, are there known performance issues for machines at or below say, 50Gbps?  It does seem that with these closed we don't have correctness or visible performance problems, so there must be some fallback mechanism at play.
2. Can we just open 1023 to 1021 for a 50GigE machine?  Or are there situations where binding might fail and the algorithm could potentially attempt to create sockets all the way down to 512?
3. Regardless of the answer to #2, do we need to open these ports on all client and server nodes, or can we get away with just server nodes?
4. Do these need to be opened just for egress from the node in question, or bidirectionally?

Thanks in advance!

Best,

ellis