<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Ellis, the addition of dynamic conns_per_peer for TCP connections is relatively new. There would be no performance "regression" against earlier Lustre releases (which always had the equivalent of conns_per_peer=1), just not additional performance gains for
high-speed Ethernet interfaces.
<div class=""><br class="">
</div>
<div class="">The port selection is somewhat dependent on the software running on the server. There is no guarantee that 1023/1022/... will be available for Lustre to use, if other TCP listeners have already been opened on those ports, unless you use something
like "portreserve" (IIRC), but this is tricky to get right with kernel-side listeners (I don't recall if we ever got that right).</div>
<div class=""><br class="">
</div>
<div class="">Whether you open those additional ports or just set conns_per_peer=1 is up to you. You would definitely want to limit the accept rule so that the source or target is port 988.</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<div class="">On Feb 1, 2023, at 15:18, Ellis Wilson via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">Hi folks,<br class="">
<br class="">
We've seen some weird stuff recently with UFW/iptables dropping packets on our OSS and MDS nodes. We are running 2.15.1. Example:<br class="">
<br class="">
[ 69.472030] [UFW BLOCK] IN=eth0 OUT= MAC=<snip> SRC=<snip> DST=<snip> LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58224 DF PROTO=TCP SPT=1022 DPT=988 WINDOW=510 RES=0x00 ACK FIN URGP=0<br class="">
<br class="">
[11777.280724] [UFW BLOCK] IN=eth0 OUT= MAC=<snip> SRC=<snip> DST=<snip> LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=44206 DF PROTO=TCP SPT=988 DPT=1023 WINDOW=509 RES=0x00 ACK URGP=0<br class="">
<br class="">
Previously, we were only allowing 988 bidirectionally on BOTH clients and servers. This was based on guidance from the Lustre manual. From the above messages it appears we may need to expand that range. This thread discusses it:<br class="">
<a href="https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg17229.html" class="">https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg17229.html</a><br class="">
<br class="">
Based on that thread and some code reading it appears that sans explicit configuration of conns_per_peer the extra ports potentially required are autotuning (ksocklnd_speed2cpp). E.G., if we have a node with 50Gbps interface, we may need up to 3 ports open
to accommodate the extra ports. These appear to be selected beginning at 1023 and going down as far as 512.<br class="">
<br class="">
Questions:<br class="">
1. If we do not open up more than 988, are there known performance issues for machines at or below say, 50Gbps? It does seem that with these closed we don't have correctness or visible performance problems, so there must be some fallback mechanism at play.<br class="">
2. Can we just open 1023 to 1021 for a 50GigE machine? Or are there situations where binding might fail and the algorithm could potentially attempt to create sockets all the way down to 512?<br class="">
3. Regardless of the answer to #2, do we need to open these ports on all client and server nodes, or can we get away with just server nodes?<br class="">
4. Do these need to be opened just for egress from the node in question, or bidirectionally?<br class="">
<br class="">
Thanks in advance!<br class="">
<br class="">
Best,<br class="">
<br class="">
ellis<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
lustre-discuss@lists.lustre.org<br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>