<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Aug 26, 2020, at 16:37, Faaland, Olaf P. <<a href="mailto:faaland1@llnl.gov" class="">faaland1@llnl.gov</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">Does Lustre 2.12 require that routes for every intermediate network are defined, on every node on a path?<br class="">
<br class="">
For example, given this Lustre network, where:<br class="">
A-D are nodes and 1-6 are addresses<br class="">
network tcp2 has only routers, no clients and no servers<br class="">
<br class="">
A(1) -tcp1- (2)B(3) -tcp2- (4)C(5) -tcp3- (6)D<br class="">
<br class="">
And configured routes:<br class="">
<br class="">
A: options lnet routes="tcp3 2@tcp1"<br class="">
B: options lnet routes="tcp3 4@tcp2"<br class="">
C: options lnet routes="tcp1 3@tcp2"<br class="">
D: options lnet routes="tcp1 5@tcp3"<br class="">
<br class="">
With Lustre <= 2.10 we configured only these routes. The only nodes that need to know tcp2 exist are attached to it, and so there are no routes to tcp2 defined anywhere.<br class="">
<br class="">
It looks to me like Lustre 2.12 attempts to send error notifications back to the original sender, and so nodes A and D may end up receiving messages from nids on tcp2. This then requires nodes A and D to have routes to tcp2 defined, so they can reply to the
messages.<br class="">
</div>
</div>
</blockquote>
<br class="">
</div>
<div>Interesting. I'm no LNet expert, but it seems strange to me that nodes other than B and C should care about the state of connections within @tcp2 if they are not endpoints themselves. A and D should never be sending messges directly to those nodes, and
the LNet routers B/C knowing which connections peers in @tcp2 are working should be enough for them to make routing decisions for A and D.</div>
<div><br class="">
</div>
<div>If B/C nodes are themselves unable to communicate with their peers, then _that_ should be sent back to A/D to indicate they cannot route packets to the target NID, but I wouldn't think A/D should get information about @tcp2 themselves?</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>