<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hello,<br>
I have problems with my lnet configuration on lustre 2.11.0<br>
everything starts just fine but after a while lnet auto discovers
peers and it adds the tcp network interface of my OSSes and clients<br>
so that clients start to write on lustre partition using tcp and no
more o2ib.<br>
I use and need tcp just to contact the MDS, and o2ib for contacting
OSSes. this configuration has always been working with Lustre 2.10.*
<br>
<br>
I tried to switch off auto peer dicovery but it did not work.<br>
I Als otried not tto use lnet.conf at all and just to use
/etc/modprobe/lustre.conf with <br>
<br>
opions lnet networks=o2ib(ib0),tcp(eth0)<br>
<br>
but seems like lustre 2.11.0 does not like it anymore.<br>
<br>
so I went back to lnet.conf but I can't make it stop to auto
discover tcp interfaces.<br>
<br>
after a while the tcp interfaces starts to appear while I did not
configure it to do so. And they overcome the usage of o2ib.<br>
<br>
how I can prevent the usage of tcp interfaces on my OSS and clients
side giving priority to the o2ib interface ?<br>
<br>
<br>
lnetctl export | grep tcp<br>
tcp bonding: 0<br>
- net type: tcp<br>
- nid: 172.21.42.211@tcp<br>
tcp bonding: 0<br>
tcp bonding: 0<br>
<b> - primary nid: 172.21.42.202@tcp</b><b><br>
</b><b> - nid: 172.21.42.202@tcp</b><br>
- primary nid: 172.21.42.213@tcp<br>
- nid: 172.21.42.213@tcp<br>
<br>
so 172.21.42.202@tcp is used instead of the infiniband interface,
and this is discovered automatically.<br>
<br>
This is the configuration on my OSS where 172.21.42.213 is the MDS.<br>
<br>
net:<br>
- net type: tcp<br>
local NI(s):<br>
- nid: 172.21.42.211@tcp<br>
status: up<br>
interfaces:<br>
0: enp1s0f0<br>
- net type: o2ib<br>
local NI(s):<br>
- nid: 172.21.52.86@o2ib<br>
status: up<br>
interfaces:<br>
0: ib0<br>
peer:<br>
- primary nid: 172.21.42.213@tcp<br>
Multi-Rail: False<br>
peer ni:<br>
- nid: 172.21.42.213@tcp<br>
state: NA<br>
- primary nid: 172.21.52.126@o2ib<br>
Multi-Rail: False<br>
peer ni:<br>
- nid: 172.21.52.126@o2ib<br>
state: NA<br>
- primary nid: 172.21.52.127@o2ib<br>
Multi-Rail: False<br>
peer ni:<br>
- nid: 172.21.52.127@o2ib<br>
state: NA<br>
- primary nid: 172.21.52.128@o2ib<br>
Multi-Rail: False<br>
peer ni:<br>
- nid: 172.21.52.128@o2ib<br>
state: NA<br>
- primary nid: 172.21.52.129@o2ib<br>
Multi-Rail: False<br>
peer ni:<br>
- nid: 172.21.52.129@o2ib<br>
state: NA<br>
- primary nid: 172.21.52.130@o2ib<br>
Multi-Rail: False<br>
peer ni:<br>
- nid: 172.21.52.130@o2ib<br>
state: NA<br>
- primary nid: 172.21.52.131@o2ib<br>
Multi-Rail: False<br>
peer ni:<br>
- nid: 172.21.52.131@o2ib<br>
state: NA<br>
global:<br>
numa_range: 0<br>
discovery: 0<br>
<br>
<br>
thanks a lot<br>
<br>
<br>
Rick<br>
<br>
<br>
</body>
</html>