<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
<html><body>
<p>Hi White,</p>
<p>tcp0(eth0) and tcp1(eth1) are connected to different segment. (connected to two virtual bridges in KVM)</p>
<p> </p>
<p>Hi all,</p>
<p>In old Lustre manual (version 1.8), I found that the order of LNET in /etc/modprobe/lustre.conf does matter:</p>
<p>(Quoted in https://wiki.lustre.org/manual/LustreManual18_HTML/MoreComplicatedConfigurations.html)</p>
<p>"The order of LNET lines in <kbd class="Command">modprobe.conf</kbd> is important when configuring multi-homed servers. If a server node can be reached using more than one network, the first network specified in <kbd class="Command">modprobe.conf</kbd> will be used."</p>
<p>That makes me more confused. Someone told me the order doesn't matter, the file just list all the available LNET devices to use.</p>
<p>Does the order does matter ONLY in old version of Lustre?</p>
<p> </p>
<p>Regards,</p>
<p>Patrick</p>
<p> </p>
<p> </p>
<p> </p>
<p>On Fri, 28 Feb 2014 21:20:58 +0000, White, Cliff wrote:</p>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<pre>On 2/28/14, 1:17 AM, "Chan Ching Yu Patrick" <<a href="mailto:cychan@clustertech.com">cychan@clustertech.com</a>>
wrote:
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">Hi Mohr,
The reason why I made this setup is I'm not sure how Lustre selects the
interface in mult-rail environment.
Especially when all node have Infiniband and Ethernet, how can I ensure
Infiniband is used between client and OSS?
</blockquote>
The LNET Œnetworks¹ option is used to specify by interface. For example,
where your Infiniband interface is Œib0¹ you would
add this to your modprobe.conf or equivalent:
‹‹‹‹‹‹‹
options lnet networks="o2ib0(ib0)²
‹‹‹‹‹‹
That will define IB (the interface denoted by ib0 to be specific). Client
mounts using @o2ib0 NIDS will only use IB,regardless of other interfaces
present.
See the Lustre manual for details on the LNET Œnetworks¹ option.
In your case, I would suspect that the two TCP/IP interfaces are
equivalent in TCP/IP routing terms, perhaps on the same segment.
When that happens TCP/IP routing is taking over. Basically, you can
control which interface you send from, but if the receiver sees two equal
TCP/IP paths back, you can¹t control which path it chooses to take. Has
nothing to do with LNET or Lustre.
In the case where the network hardware is dissimilar, you don¹t have this
problem. Connections starting on IB stay on IB.
If you only have one IB network, using the IB NID will ensure all clients
use only IB.
cliffw
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">
Regards,
Patrick
On 02/27/2014 12:28 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">On Feb 26, 2014, at 7:14 PM, "Chan Ching Yu,
Patrick"<<a href="mailto:cychan@clustertech.com">cychan@clustertech.com</a>>
wrote:
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">[root@mds1 ~]# lctl list_nids
192.168.122.240@tcp
192.168.100.100@tcp1
[root@oss1 ~]# lctl list_nids
192.168.122.194@tcp
192.168.100.101@tcp1
[root@client ~]# lctl list_nids
192.168.122.70@tcp
192.168.100.102@tcp1
</blockquote>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">On Lustre client, I intentionally mount it with tcp1
[root@client ~]# mount | grep lustre
192.168.100.100@tcp1:/data on /lustre type lustre (rw)
Now I dd a file on Lustre filesystem, you can see that tcp0 is used
when writing on OST.
Why?
</blockquote>I am not an expert on the inner workings of lustre, but as far as I
understand it, when oss1 connects to the mgs, it will report the nids it
has available. When the client connects to mgs to get info about the
oss1 server, it will receive a list of all the oss1 nids. The client
then steps through that list and compares the oss1 nids with its local
nids to find a match (i.e. - nids that are on the same lnet network).
If it matches tcp0 first, then that is the connection it uses. The lnet
network used to connect to the mgs is irrelevant at that point.
However, I do not know if there are any guarantees about the ordering of
the nids that the mgs will report (ie - will tcp0 always be the first
nid?).
If there is an error in my description, hopefully a lustre developer
will point out the flaw.
It is not clear what you are trying to accomplish with this multi rail
setup. Are you trying to force mds traffic over one client link and oss
traffic over the other? Or are you trying to utilize both links
simultaneously for all traffic?
</blockquote>
_______________________________________________
Lustre-discuss mailing list
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a>
</blockquote>
</pre>
</blockquote>
<p> </p>
</body></html>