[Lustre-discuss] LNET routing question

David Noriega tsk133 at my.utsa.edu
Tue Apr 5 11:59:35 PDT 2011


Well I would call our setup a barrier case. The internal 192.168.x.x
network is completely internal to the cluster, inaccessible from the
outside. So following this I can setup a router machine to allow
access from the external network to lustre, correct?

On part two, i would simply be adding tcp1 to the oss/mds, tcp0 which
everything already connects to would still be there. So it was my
guess and looks like you agree that so long as the clients continue to
use tcp0, which they will, they will still be able to connect just
fine. tcp1 would be just for the backup client.

Thanks
David

On Tue, Apr 5, 2011 at 1:48 PM, Cliff White <cliffw at whamcloud.com> wrote:
> That's the 'barrier' case i was talking about - using routers to separate
> public/private networks - basically using a Lustre router as a hole through
> a firewall.
> Second question - depends on your world. Obviously, machines with
> mis-matched network configs may not be able to comunicate, so depends on
> whether you can tolerate some clients not reaching some OST while the
> changes are rolling through.
> I would think adding the second net for the backup would be transparent to
> existing
> clients, modulo the OST restart needed.
> cliffw
>
> On Tue, Apr 5, 2011 at 11:36 AM, David Noriega <tsk133 at my.utsa.edu> wrote:
>>
>> What about this example?
>> http://comments.gmane.org/gmane.comp.file-systems.lustre.user/6687
>>
>> Also to my second question, would these changes have to be done all at
>> once? or could I edit one modprobe.conf at a time and fail over then
>> back as I make changes to each oss/mds?
>>
>> Thanks
>> David
>>
>> On Tue, Apr 5, 2011 at 11:52 AM, Cliff White <cliffw at whamcloud.com> wrote:
>> > Lustre routing is to connect different types of network.
>> > If all your networks are TCP, you should be able to use standard
>> > TCP routing/addressing without needing Lustre routers.
>> > Again, if the Linux workstations in your lab are TCP, you should
>> > be able to create a TCP route to the Lustre servers without needing
>> > a Lustre router in the middle, unless you have some barrier, and
>> > you need a lustre router to cross that barrier.
>> > Generally, people do not use routers as clients, there is nothing
>> > stopping your from doing this, but a) the router will take resources
>> > away from the clients, impacting performance of both.
>> > b) again,clients are typically endpoints, and routers sit in the middle,
>> > so from a network design perspective it's usually silly.
>> > Also, lustre routers function as a pool, failed routers are bypassed,
>> > so a pool of dedicated routers can tolerate individual machine outages.
>> > That's not good for clients.
>> > But the main reason people do dedicated boxes for Lustre routing, is
>> > that
>> > Lustre routing is designed to bridge different network hardware.
>> > If all your nets are TCP, I think using standard networking methods will
>> > be
>> > better for you, simpler and easier to maintain.
>> > cliffw
>> >
>> > On Mon, Apr 4, 2011 at 6:50 PM, David Noriega <tsk133 at my.utsa.edu>
>> > wrote:
>> >>
>> >> The file server does sit on both networks, internal and external. I
>> >> would just like to have a thrid option beyond nfs/samba, such as
>> >> making the linux workstations up in our lab, lustre clients. But you
>> >> are saying either 1) I do some sort of regular tcp routing? or 2) an
>> >> existing client cannot also work as a router?
>> >>
>> >> On Mon, Apr 4, 2011 at 3:43 PM, Cliff White <cliffw at whamcloud.com>
>> >> wrote:
>> >> >
>> >> >
>> >> > On Mon, Apr 4, 2011 at 1:32 PM, David Noriega <tsk133 at my.utsa.edu>
>> >> > wrote:
>> >> >>
>> >> >> Reading up on LNET routing and have a question. Currently have
>> >> >> nothing
>> >> >> special going on, simply specified tcp0(bond0) on the OSSs and MDS.
>> >> >> Same for all the clients as well, we have an internal network for
>> >> >> our
>> >> >> cluster, 192.168.x.x.  How would I go about doing the following?
>> >> >>
>> >> >> Data1,Data2 = OSS, Meta1,Meta2 = MDS.
>> >> >>
>> >> >> Internally its 192.168.1.x for cluster nodes, 192.168.5.x for lustre
>> >> >> nodes.
>> >> >>
>> >> >> But I would like a 1) a 'forwarding' sever, which would be our file
>> >> >> server which exports lustre via samba/nfs to also be the outside
>> >> >> world's access point to lustre(outside world being the rest of the
>> >> >> campus). 2) a second internal network simply connecting the OSSs and
>> >> >> MDS to the backup client to do backups outside of the cluster
>> >> >> network.
>> >> >
>> >> > Slightly confused am I.
>> >> >  1) is just a samba/nfs exporter, while you might
>> >> > have two networks in the one box, you wouldn't be doing any routing,
>> >> > the Lustre client is re-exporting the FS.
>> >> > The Lustre client has to find the Lustre servers, the samba/NFS
>> >> > clients
>> >> > only
>> >> > have to find the Lustre client.
>> >> > 2) if the second internal net connects backup clients directly to
>> >> > OSS/MDS
>> >> > you  again need no routing.
>> >> > Lustre Routing is really to connect disparte network hardware for
>> >> > Lustre traffic, for example Infiniband routed to TCP/IP, or
>> >> > Quadratics
>> >> > to
>> >> > IB.
>> >> > Also, file servers are never routers, since they have direct
>> >> > connections
>> >> > to
>> >> > all clients. Routers are dedicated nodes that have both hardware
>> >> > interfaces
>> >> > and
>> >> > sit between a client and server.
>> >> > Typical setup are things like a cluster with server and clients on
>> >> > IB,
>> >> > you
>> >> > wish to add a second client pool on TCP/IP, you have to build nodes
>> >> > that
>> >> > have both TCP/IP and IB interfaces, and those are Lustre Routers.
>> >> > Since all your traffic is TCP/IP, sounds like normal TCP/IP network
>> >> > manipulation
>> >> > is all you are needing. You would need the 'lnet networks' stuff to
>> >> > align nets with interfaces, and that part looks correct.
>> >> > cliffw
>> >> >
>> >> >>
>> >> >> So would I do the following?
>> >> >>
>> >> >> OSS/MDS
>> >> >> options lnet networks=tcp0(bond0),tcp1(eth3) routes="tcp2
>> >> >> 192.168.2.1"
>> >> >>
>> >> >> Backup client
>> >> >> options lnet networks=tcp1(eth1)
>> >> >>
>> >> >> Cluster clients
>> >> >> options lnet networks=tcp0(eth0)
>> >> >>
>> >> >> File Server
>> >> >> options lnet networks=tcp0(eth1),tcp2(eth2) forwarding="enabled"
>> >> >>
>> >> >> And for any outside clients I would do the following?
>> >> >> options lnet networks=tcp2(eth0)
>> >> >>
>> >> >> And when mounting from the outside I would use in /etc/fstab the
>> >> >> external
>> >> >> ip?
>> >> >> x.x.x.x at tcp2:/lustre /lustre lustre defaults,_netdev 0 0
>> >> >>
>> >> >> Is this how it would work? Also can I do this piece-meal or does it
>> >> >> have to be done all at once?
>> >> >>
>> >> >> Thanks
>> >> >> David
>> >> >>
>> >> >> --
>> >> >> Personally, I liked the university. They gave us money and
>> >> >> facilities,
>> >> >> we didn't have to produce anything! You've never been out of
>> >> >> college!
>> >> >> You don't know what it's like out there! I've worked in the private
>> >> >> sector. They expect results. -Ray Ghostbusters
>> >> >> _______________________________________________
>> >> >> Lustre-discuss mailing list
>> >> >> Lustre-discuss at lists.lustre.org
>> >> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > cliffw
>> >> > Support Guy
>> >> > WhamCloud, Inc.
>> >> > www.whamcloud.com
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Personally, I liked the university. They gave us money and facilities,
>> >> we didn't have to produce anything! You've never been out of college!
>> >> You don't know what it's like out there! I've worked in the private
>> >> sector. They expect results. -Ray Ghostbusters
>> >> _______________________________________________
>> >> Lustre-discuss mailing list
>> >> Lustre-discuss at lists.lustre.org
>> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> >
>> >
>> >
>> > --
>> > cliffw
>> > Support Guy
>> > WhamCloud, Inc.
>> > www.whamcloud.com
>> >
>> >
>>
>>
>>
>> --
>> Personally, I liked the university. They gave us money and facilities,
>> we didn't have to produce anything! You've never been out of college!
>> You don't know what it's like out there! I've worked in the private
>> sector. They expect results. -Ray Ghostbusters
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
> --
> cliffw
> Support Guy
> WhamCloud, Inc.
> www.whamcloud.com
>
>



-- 
Personally, I liked the university. They gave us money and facilities,
we didn't have to produce anything! You've never been out of college!
You don't know what it's like out there! I've worked in the private
sector. They expect results. -Ray Ghostbusters



More information about the lustre-discuss mailing list