<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
The reason why I setup two Ethernet is to experiment the order in
modprobe file takes effect or not.<br>
<br>
Now, I think the order Does take effect, if two network type of LNET
are the same.<br>
<br>
Thanks Keith and Cliff<br>
<br>
Regards,<br>
Patrick<br>
<br>
<br>
On 03/04/2014 01:34 AM, Keith Mannthey wrote:
<blockquote cite="mid:1393868079.14394.289.camel@keith-G75VW"
type="cite">
<pre wrap="">Patrick,
The current manual also has the same language it is I way the system has
been designed.
"
The order of LNET entries is important when configuring servers. If a
server node can be reached using more than one network, the first
network specified in lustre.conf will be used.
"
(Link to Lustre 2.x manual)
<a class="moz-txt-link-freetext" href="https://wiki.hpdd.intel.com/display/PUB/Documentation">https://wiki.hpdd.intel.com/display/PUB/Documentation</a>
Have you considered using Ethernet bonding?
What are you trying to accomplish with the dual Links between all the
systems?
Thanks,
Keith Mannthey
Intel HPDD
On Sun, 2014-03-02 at 08:26 +0800, Chan Ching Yu, Patrick wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi White,
tcp0(eth0) and tcp1(eth1) are connected to different segment.
(connected to two virtual bridges in KVM)
Hi all,
In old Lustre manual (version 1.8), I found that the order of LNET
in /etc/modprobe/lustre.conf does matter:
(Quoted in
<a class="moz-txt-link-freetext" href="https://wiki.lustre.org/manual/LustreManual18_HTML/MoreComplicatedConfigurations.html">https://wiki.lustre.org/manual/LustreManual18_HTML/MoreComplicatedConfigurations.html</a>)
"The order of LNET lines in modprobe.conf is important when
configuring multi-homed servers. If a server node can be reached using
more than one network, the first network specified in modprobe.conf
will be used."
That makes me more confused. Someone told me the order doesn't matter,
the file just list all the available LNET devices to use.
Does the order does matter ONLY in old version of Lustre?
Regards,
Patrick
On Fri, 28 Feb 2014 21:20:58 +0000, White, Cliff wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On 2/28/14, 1:17 AM, "Chan Ching Yu Patrick" <a class="moz-txt-link-rfc2396E" href="mailto:cychan@clustertech.com"><cychan@clustertech.com></a>
wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Mohr,
The reason why I made this setup is I'm not sure how Lustre selects the
interface in mult-rail environment.
Especially when all node have Infiniband and Ethernet, how can I ensure
Infiniband is used between client and OSS?
</pre>
</blockquote>
<pre wrap="">
The LNET Œnetworks¹ option is used to specify by interface. For example,
where your Infiniband interface is Œib0¹ you would
add this to your modprobe.conf or equivalent:
‹‹‹‹‹‹‹
options lnet networks="o2ib0(ib0)²
‹‹‹‹‹‹
That will define IB (the interface denoted by ib0 to be specific). Client
mounts using @o2ib0 NIDS will only use IB,regardless of other interfaces
present.
See the Lustre manual for details on the LNET Œnetworks¹ option.
In your case, I would suspect that the two TCP/IP interfaces are
equivalent in TCP/IP routing terms, perhaps on the same segment.
When that happens TCP/IP routing is taking over. Basically, you can
control which interface you send from, but if the receiver sees two equal
TCP/IP paths back, you can¹t control which path it chooses to take. Has
nothing to do with LNET or Lustre.
In the case where the network hardware is dissimilar, you don¹t have this
problem. Connections starting on IB stay on IB.
If you only have one IB network, using the IB NID will ensure all clients
use only IB.
cliffw
</pre>
<blockquote type="cite">
<pre wrap="">
Regards,
Patrick
On 02/27/2014 12:28 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Feb 26, 2014, at 7:14 PM, "Chan Ching Yu,
Patrick"<a class="moz-txt-link-rfc2396E" href="mailto:cychan@clustertech.com"><cychan@clustertech.com></a>
wrote:
</pre>
<blockquote type="cite">
<pre wrap="">[root@mds1 ~]# lctl list_nids
192.168.122.240@tcp
192.168.100.100@tcp1
[root@oss1 ~]# lctl list_nids
192.168.122.194@tcp
192.168.100.101@tcp1
[root@client ~]# lctl list_nids
192.168.122.70@tcp
192.168.100.102@tcp1
</pre>
</blockquote>
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">On Lustre client, I intentionally mount it with tcp1
[root@client ~]# mount | grep lustre
192.168.100.100@tcp1:/data on /lustre type lustre (rw)
Now I dd a file on Lustre filesystem, you can see that tcp0 is used
when writing on OST.
Why?
</pre>
</blockquote>
<pre wrap="">I am not an expert on the inner workings of lustre, but as far as I
understand it, when oss1 connects to the mgs, it will report the nids it
has available. When the client connects to mgs to get info about the
oss1 server, it will receive a list of all the oss1 nids. The client
then steps through that list and compares the oss1 nids with its local
nids to find a match (i.e. - nids that are on the same lnet network).
If it matches tcp0 first, then that is the connection it uses. The lnet
network used to connect to the mgs is irrelevant at that point.
However, I do not know if there are any guarantees about the ordering of
the nids that the mgs will report (ie - will tcp0 always be the first
nid?).
If there is an error in my description, hopefully a lustre developer
will point out the flaw.
It is not clear what you are trying to accomplish with this multi rail
setup. Are you trying to force mds traffic over one client link and oss
traffic over the other? Or are you trying to utilize both links
simultaneously for all traffic?
</pre>
</blockquote>
<pre wrap="">
_______________________________________________
Lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a>
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
_______________________________________________
Lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a>
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
<br>
<div class="moz-signature">-- <br>
<span style="color:#000; font-size:14px;">
Chan Ching Yu, Patrick<br>
Senior System Engineer<br>
</span>
<span style="font-weight:bold; color:#333333">Cluster Technology
Limited</span><br>
<br>
<span style="font-weight:bold; color:#999;font-size:13px;
font-style:italic;">Modernize Your Business with Advanced
Computing Technologies</span><br>
<span style="font-weight:bold; color:#999;font-size:13px;">Cloud
Computing | Cluster | Financial Engineering | Business
Intelligence</span><br>
<br>
<div style="font-size:13px;color:#666666;">
<span style="color: #00CCFF">Email:</span> <a
href="mailto:cychan@clustertech.com"> cychan @clustertech.com</a><br>
<span style="color: #00CCFF">Direct Line:</span> +852 2655 6113
<span style="color: #00CCFF">Tel:</span> +852 2655 6100
<span style="color: #00CCFF">Fax:</span> +852 2994 2101
<br>
<span style="color: #00CCFF">Website:</span> <a
href="http://www.clustertech.com">www.clustertech.com</a><br>
<span style="color: #00CCFF">Address:</span> Units 211 - 213,
Lakeside 1, No. 8 Science Park West Avenue, Hong Kong Science
Park, N.T. Hong Kong<br>
<div style="line-height:14px; font-style:italic;">Hong Kong
Beijing Shanghai Guangzhou Shenzhen Wuhan
Sydney</div>
<br>
</div>
<span style="font-size:12px;">
**************************************************************************<br>
The information contained in this e-mail and its attachments is
confidential and intended solely for the specified addressees.
If you have received this email in error, please do not read,
copy, distribute, disclose or use any information of this email
in any way and please immediately notify the sender and delete
this email. Thank you for your cooperation.<br>
**************************************************************************
</span></div>
</body>
</html>