<div dir="ltr">Hi All,<br><br>Just wondering if someone can give us some insight into the logic that ksocklnd uses to decide which connections to make.<br><br>There's not so much in the Lustre operations manual about it, but the impression I get from reading around is that if we have:<br>
<br>options lnet networks=tcp0(eth0,eth1)<br><br>on all of our
dual-connected hosts, then they will load-balance by making multiple
connections between clients and servers. Indeed, they do. However, I
would have expected 4 TCP connection bundles (i.e. 12 connections)
(eth[0,1]<->eth[0,1]), but we actually get two (i.e. 6
connections) (eth0<->eth0 and eth1<->eth1). How does
lustre know which combinations to use???<br>
<br>Some important points about our setup:<br>- This is a shared network segment <a href="http://172.16.0.0/16" target="_blank">172.16.0.0/16</a><br>- Three switches (LeftSwitch<->TopSwitch<-><div dir="ltr">RightSwitch)<br>
- all dual connected hosts are connected to both LeftSwitch and RightSwitch<br>
- clients network interfaces are 172.16.4.x/16 (eth0,leftswitch) and 172.16.5.x/16 (eth1,rightswitch)<br>- OSS/MDS network interfaces are 172.16.0.x/16 (eth0,leftswitch) and 172.16.1.x/16 (eth1,rightswitch)<br>- to get good routing, we have static routes configured as such, on all dual-connect machines:<br>
Kernel IP routing table<br>Destination Gateway Genmask Flags MSS Window irtt Iface<br>
<a href="http://172.16.4.0/" target="_blank">172.16.4.0</a> <a href="http://0.0.0.0/" target="_blank">0.0.0.0</a> <a href="http://255.255.255.0/" target="_blank">255.255.255.0</a> U 0 0 0 eth0<br>
<a href="http://172.16.5.0/" target="_blank">172.16.5.0</a> <a href="http://0.0.0.0/" target="_blank">0.0.0.0</a> <a href="http://255.255.255.0/" target="_blank">255.255.255.0</a> U 0 0 0 eth1<br>
<a href="http://172.16.0.0/" target="_blank">172.16.0.0</a> <a href="http://0.0.0.0/" target="_blank">0.0.0.0</a> <a href="http://255.255.255.0/" target="_blank">255.255.255.0</a> U 0 0 0 eth0<br>
<a href="http://172.16.1.0/" target="_blank">172.16.1.0</a> <a href="http://0.0.0.0/" target="_blank">0.0.0.0</a> <a href="http://255.255.255.0/" target="_blank">255.255.255.0</a> U 0 0 0 eth1<br>
(i.e. all traffic between clients and servers shouldn't traverse TopSwitch, notwithstanding occasional ugly arp issues)<br><br>So
lustre has done the right thing in connecting eth0<->eth0 and
eth1<->eth1 in this case. But how does it know? Does the client
connect to both server addresses and throw away any connections
originating from the same address? Is there some check of the return
path?<br>
<br>My motive here is that I also have a set of singly-connected
machines, and want to have their traffic balanced across both server
networks (single connect machines come in via topswitch). Right now,
these clients all connect to the eth0 address (172.16.0.x) on all OSSes
and the MDS. All the traffic goes via leftswitch, and my peak
bandwidth to a single OSS is therefore 1 gigabit, and the disks are
capable of more than that. What if my single connect client was a
10gig NIC? Or lots of 1gig single connect clients? It seems that we
are getting it wrong in these cases.<br>
<br>I understand the issues with routing return traffic from the OSS -
I am happy/planning to configure source-based routing on the server
nodes, but I only want to go to the effort once I understand whatever
black magic ksocklnd is doing to decide which connections it should
make! If I go ahead and configure the source routing, will we end up
with two connections being made from a client with a single IP?<br>
<br>Thanks for your help,<br><br>Tim</div></div>