[Lustre-discuss] Understanging LNET routing

[Vsevolod Nikonorov]

I have discovered that MDS log contains errors during communication with NIDs that did not exist. Most likely that was a consequence of several interconfused configurations, so I just recreated the whole test stand, and now I've got LNET routing working. Thanks everybody for helping!


On Tue, 20 Aug 2013 06:46:54 +0000
"Lee, Brett" <brett.lee at intel.com> wrote:

> Please check that the Lustre client can ping the MGS, over LNet.
> 
> [root at compute1 ~]# ping lustre-18-mgs
> PING lustre-18-mgs.nitrox.net (192.168.10.120) 56(84) bytes of data.
> 64 bytes from lustre-18-mgs.nitrox.net (192.168.10.120): icmp_seq=1 ttl=64 time=0.377 ms
> 
> --- lustre-18-mgs.nitrox.net ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 0.377/0.377/0.377/0.000 ms
> 
> 
> [root at compute1 ~]# lctl ping 192.168.10.120 at tcp1
> 12345-0 at lo
> 12345-192.168.10.120 at tcp1
> [root at compute1 ~]#
> 
> Once you can ping, you should be able to mount.  If mounting fails, please send the mount command error message and the pertinent log entries (client and MGS).
> 
> --
> Brett Lee
> Sr. Systems Engineer
> Intel High Performance Data Division
> 
> 
> > -----Original Message-----
> > From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-
> > bounces at lists.lustre.org] On Behalf Of Vsevolod Nikonorov
> > Sent: Tuesday, August 20, 2013 12:10 AM
> > To: lustre-discuss at lists.lustre.org
> > Subject: Re: [Lustre-discuss] Understanging LNET routing
> > 
> > Thanks, seems like my router is a router indeed. Nevertheless I am unable to
> > mount Lustre filesystem: "mount.lustre 10.3.0.102 at tcp:/SANDBOX
> > /mnt/lustre" just hangs, though meanwhile Lustre client communicates with
> > Lustre router (which may be seen using rules like "iptables -I INPUT -p tcp ! --
> > dport 22 -j LOG"), and Lustre router communicates with my MDS.
> > /var/log/messages on MDS does not contain a single line about Lustre
> > activity, though mdt volume is formatted and mounted properly. I thing I
> > need to increase debugging level at least on MDS, how can it be done?
> > 
> > On Mon, 19 Aug 2013 14:37:11 -0400
> > David Dillow <dillowda at ornl.gov> wrote:
> > 
> > > On Mon, 2013-08-19 at 16:50 +0400, Vsevolod Nikonorov wrote:
> > > > Is it possible to route Lustre traffic between two TCP networks?
> > >
> > > Yes, and you have multiple ways to achieve that goal. If you are using
> > > TCP, then it routes just like any other IP traffic -- you may not even
> > > need Lustre routers.
> > >
> > > If you are trying to change LNET types -- say from RDMA over IB to TCP
> > > -- you would run an LNET router to handle the protocol translation.
> > >
> > > There are other reasons to run LNET routers, such as maintaining
> > > separation between networks or fabrics, but in many cases those are
> > > more administrative or performance based than absolute technical
> > > requirements.
> > >
> > > >  Manual states that
> > > >
> > > > "the routers would enable LNet forwarding since their NIDs are
> > > > specified in the 'routes' parameters as being routers."
> > > >
> > > > How can I query the router if it consideres itself a router?
> > >
> > > Look in /proc/sys/lnet/routes; if you see "Routing enabled" then the
> > > router knows it is a router.
> > >
> > > Hope this helps,
> > > --
> > > Dave Dillow
> > > National Center for Computational Science Oak Ridge National
> > > Laboratory
> > > (865) 241-6602 office
> > >
> > >
> > 
> > 
> > --
> > Всеволод Никоноров,
> > ОИТТиС, НИКИЭТ
> > 
> > <v.nikonorov at nikiet.ru>
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss at lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 


-- 
Всеволод Никоноров,
ОИТТиС, НИКИЭТ

<v.nikonorov at nikiet.ru>