<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Oct 10, 2019, at 11:20, Michael Di Domenico <<a href="mailto:mdidomenico4@gmail.com" class="">mdidomenico4@gmail.com</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">On Mon, Oct 7, 2019 at 6:33 PM Andreas Dilger <<a href="mailto:adilger@whamcloud.com" class="">adilger@whamcloud.com</a>> wrote:<br class="">
<blockquote type="cite" class=""><br class="">
With socklnd there are 3 TCP connections per client-server pair.<br class="">
For IB there is no such connection limit that I'm aware of.<br class="">
</blockquote>
<br class="">
just out of morbid curiosity, can very briefly explain the<br class="">
connectivity differences between TCP/IB. Does IB use the same 3<br class="">
connections as TCP? If not, is that why the connectivity limit<br class="">
doesn't exist with IB or is there some other overriding design<br class="">
principal in IB that allows lustre to push past TCP? Not that any of<br class="">
this has any relevance to anything i do, i'm just curious.<br class="">
<br class="">
i'd love to have 2000 OSS's and 20k clients, but sadly i do not... :(<br class="">
</div>
</div>
</blockquote>
<br class="">
</div>
<div>This is a fundamental difference between TCP and IB. TCP needs a persistent</div>
<div>connection between peers (socket) to manage state, and the (very ancient) IP</div>
<div>protocol on which TCP is built has a limit of 65536 connections on a single node.</div>
<div>When computers had 1-2MB of RAM that was more than enough...</div>
<div><br class="">
</div>
<div>IB does not have this limitation, though it does consume some memory for each</div>
<div>peer that that it is communicating with. o2iblnd can establish multiple connections</div>
<div>to a single peer to get better bandwidth, and this is important for OPA performance,</div>
<div>but is not critical for IB networks.</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>