[Lustre-discuss] Lustre routers capabilities

Canon, Richard Shane canonrs at ornl.gov
Thu Apr 10 05:31:54 PDT 2008


Sebastien,

ORNL is assembling a fairly large configuration that will rely heavily on routers.  We have a small configuration running similar to yours (IB to IB).  Being a Cray shop, we also have IB to Portals running.  

We are trying to reach 1.25 GB/s per router.  We haven't achieved that yet.  The best so far is about 1.1 GB/s sustained.  We do see burst up to 1.25-1.3, but not sustained.  This was measured on an IB to portals setup with the XT having PCI-e riser (not GA yet).

I don't have good numbers on the IB-to-IB routers.  If I get some, I will post to the list.

Depending on the scale of the cluster, memory can be a driver.  You will need to allocate enough buffers on the routers to sustain traffic flow.  The main driver for memory consumption is the number of 1M buffers that the routers will run with.  

For IB we have not seen a huge CPU load, but I don't have good numbers to back that up.  So CPU may still be a factor.  We hope to repeat the above test but on a Dual core box (versus the single core used thus far).

I will update the list as things evolve.

--Shane


-----Original Message-----
From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Sébastien Buisson
Sent: Thursday, April 10, 2008 4:07 AM
To: lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] Lustre routers capabilities

Let's consider that the internal bus of the machine is bigger enough so 
that it will not be saturated. In that case, what will be the limiting 
factor? memory? CPU?
I know that it depends on how many I/B cards are plugged in the machine, 
but generally speaking, is the routing activity CPU or memory hungry?

By the way, are there people on that list that have feedback about 
Lustre routers sizing? For instance, I know that Lustre routers have 
been set up at the LLNL. What is the throughput obtained via the 
routers, compared to the raw bandwidth of the interconnect?

Thanks,
Sebastien.


Brian J. Murrell a écrit :
> On Wed, 2008-04-09 at 19:07 +0200, Sébastien Buisson wrote:
>> I mean, if I 
>> have an available bandwith of 100 on each side of a router, what will be 
>> the max reachable bandwith from clients on one side of the router to 
>> servers on the other side of the router? Is it 50? 80? 99? Is the 
>> routing process CPU or memory hungry?
> 
> While I can't answer these things specifically another important
> consideration is the bus architecture involved.  How many I/B cards can
> you put on a bus before you saturate the bus?
> 
> b.
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss



More information about the lustre-discuss mailing list