[Lustre-discuss] LNET routing question

David Noriega tsk133 at my.utsa.edu
Tue Apr 5 11:36:32 PDT 2011


What about this example?
http://comments.gmane.org/gmane.comp.file-systems.lustre.user/6687

Also to my second question, would these changes have to be done all at
once? or could I edit one modprobe.conf at a time and fail over then
back as I make changes to each oss/mds?

Thanks
David

On Tue, Apr 5, 2011 at 11:52 AM, Cliff White <cliffw at whamcloud.com> wrote:
> Lustre routing is to connect different types of network.
> If all your networks are TCP, you should be able to use standard
> TCP routing/addressing without needing Lustre routers.
> Again, if the Linux workstations in your lab are TCP, you should
> be able to create a TCP route to the Lustre servers without needing
> a Lustre router in the middle, unless you have some barrier, and
> you need a lustre router to cross that barrier.
> Generally, people do not use routers as clients, there is nothing
> stopping your from doing this, but a) the router will take resources
> away from the clients, impacting performance of both.
> b) again,clients are typically endpoints, and routers sit in the middle,
> so from a network design perspective it's usually silly.
> Also, lustre routers function as a pool, failed routers are bypassed,
> so a pool of dedicated routers can tolerate individual machine outages.
> That's not good for clients.
> But the main reason people do dedicated boxes for Lustre routing, is that
> Lustre routing is designed to bridge different network hardware.
> If all your nets are TCP, I think using standard networking methods will be
> better for you, simpler and easier to maintain.
> cliffw
>
> On Mon, Apr 4, 2011 at 6:50 PM, David Noriega <tsk133 at my.utsa.edu> wrote:
>>
>> The file server does sit on both networks, internal and external. I
>> would just like to have a thrid option beyond nfs/samba, such as
>> making the linux workstations up in our lab, lustre clients. But you
>> are saying either 1) I do some sort of regular tcp routing? or 2) an
>> existing client cannot also work as a router?
>>
>> On Mon, Apr 4, 2011 at 3:43 PM, Cliff White <cliffw at whamcloud.com> wrote:
>> >
>> >
>> > On Mon, Apr 4, 2011 at 1:32 PM, David Noriega <tsk133 at my.utsa.edu>
>> > wrote:
>> >>
>> >> Reading up on LNET routing and have a question. Currently have nothing
>> >> special going on, simply specified tcp0(bond0) on the OSSs and MDS.
>> >> Same for all the clients as well, we have an internal network for our
>> >> cluster, 192.168.x.x.  How would I go about doing the following?
>> >>
>> >> Data1,Data2 = OSS, Meta1,Meta2 = MDS.
>> >>
>> >> Internally its 192.168.1.x for cluster nodes, 192.168.5.x for lustre
>> >> nodes.
>> >>
>> >> But I would like a 1) a 'forwarding' sever, which would be our file
>> >> server which exports lustre via samba/nfs to also be the outside
>> >> world's access point to lustre(outside world being the rest of the
>> >> campus). 2) a second internal network simply connecting the OSSs and
>> >> MDS to the backup client to do backups outside of the cluster network.
>> >
>> > Slightly confused am I.
>> >  1) is just a samba/nfs exporter, while you might
>> > have two networks in the one box, you wouldn't be doing any routing,
>> > the Lustre client is re-exporting the FS.
>> > The Lustre client has to find the Lustre servers, the samba/NFS clients
>> > only
>> > have to find the Lustre client.
>> > 2) if the second internal net connects backup clients directly to
>> > OSS/MDS
>> > you  again need no routing.
>> > Lustre Routing is really to connect disparte network hardware for
>> > Lustre traffic, for example Infiniband routed to TCP/IP, or Quadratics
>> > to
>> > IB.
>> > Also, file servers are never routers, since they have direct connections
>> > to
>> > all clients. Routers are dedicated nodes that have both hardware
>> > interfaces
>> > and
>> > sit between a client and server.
>> > Typical setup are things like a cluster with server and clients on IB,
>> > you
>> > wish to add a second client pool on TCP/IP, you have to build nodes that
>> > have both TCP/IP and IB interfaces, and those are Lustre Routers.
>> > Since all your traffic is TCP/IP, sounds like normal TCP/IP network
>> > manipulation
>> > is all you are needing. You would need the 'lnet networks' stuff to
>> > align nets with interfaces, and that part looks correct.
>> > cliffw
>> >
>> >>
>> >> So would I do the following?
>> >>
>> >> OSS/MDS
>> >> options lnet networks=tcp0(bond0),tcp1(eth3) routes="tcp2 192.168.2.1"
>> >>
>> >> Backup client
>> >> options lnet networks=tcp1(eth1)
>> >>
>> >> Cluster clients
>> >> options lnet networks=tcp0(eth0)
>> >>
>> >> File Server
>> >> options lnet networks=tcp0(eth1),tcp2(eth2) forwarding="enabled"
>> >>
>> >> And for any outside clients I would do the following?
>> >> options lnet networks=tcp2(eth0)
>> >>
>> >> And when mounting from the outside I would use in /etc/fstab the
>> >> external
>> >> ip?
>> >> x.x.x.x at tcp2:/lustre /lustre lustre defaults,_netdev 0 0
>> >>
>> >> Is this how it would work? Also can I do this piece-meal or does it
>> >> have to be done all at once?
>> >>
>> >> Thanks
>> >> David
>> >>
>> >> --
>> >> Personally, I liked the university. They gave us money and facilities,
>> >> we didn't have to produce anything! You've never been out of college!
>> >> You don't know what it's like out there! I've worked in the private
>> >> sector. They expect results. -Ray Ghostbusters
>> >> _______________________________________________
>> >> Lustre-discuss mailing list
>> >> Lustre-discuss at lists.lustre.org
>> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> >
>> >
>> >
>> > --
>> > cliffw
>> > Support Guy
>> > WhamCloud, Inc.
>> > www.whamcloud.com
>> >
>> >
>>
>>
>>
>> --
>> Personally, I liked the university. They gave us money and facilities,
>> we didn't have to produce anything! You've never been out of college!
>> You don't know what it's like out there! I've worked in the private
>> sector. They expect results. -Ray Ghostbusters
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
> --
> cliffw
> Support Guy
> WhamCloud, Inc.
> www.whamcloud.com
>
>



-- 
Personally, I liked the university. They gave us money and facilities,
we didn't have to produce anything! You've never been out of college!
You don't know what it's like out there! I've worked in the private
sector. They expect results. -Ray Ghostbusters



More information about the lustre-discuss mailing list