[Lustre-discuss] Lustre routers capabilities

[Sébastien Buisson]

Hello,

And thanks for your feedback.
I am very interested in the figures you got. I understand that you 
reached 1.1 GB/s sustained on an IB to Portals router. But on that 
router, what is the raw IB bandwidth available, and the raw Portals 
bandwidth available? In your specific case, what do you think is the 
bottleneck on the Lustre router?

Sebastien.


Canon, Richard Shane a écrit :
> Sebastien,
> 
> ORNL is assembling a fairly large configuration that will rely heavily on routers.  We have a small configuration running similar to yours (IB to IB).  Being a Cray shop, we also have IB to Portals running.  
> 
> We are trying to reach 1.25 GB/s per router.  We haven't achieved that yet.  The best so far is about 1.1 GB/s sustained.  We do see burst up to 1.25-1.3, but not sustained.  This was measured on an IB to portals setup with the XT having PCI-e riser (not GA yet).
> 
> I don't have good numbers on the IB-to-IB routers.  If I get some, I will post to the list.
> 
> Depending on the scale of the cluster, memory can be a driver.  You will need to allocate enough buffers on the routers to sustain traffic flow.  The main driver for memory consumption is the number of 1M buffers that the routers will run with.  
> 
> For IB we have not seen a huge CPU load, but I don't have good numbers to back that up.  So CPU may still be a factor.  We hope to repeat the above test but on a Dual core box (versus the single core used thus far).
> 
> I will update the list as things evolve.
> 
> --Shane
> 
> 
> -----Original Message-----
> From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Sébastien Buisson
> Sent: Thursday, April 10, 2008 4:07 AM
> To: lustre-discuss at lists.lustre.org
> Subject: Re: [Lustre-discuss] Lustre routers capabilities
> 
> Let's consider that the internal bus of the machine is bigger enough so 
> that it will not be saturated. In that case, what will be the limiting 
> factor? memory? CPU?
> I know that it depends on how many I/B cards are plugged in the machine, 
> but generally speaking, is the routing activity CPU or memory hungry?
> 
> By the way, are there people on that list that have feedback about 
> Lustre routers sizing? For instance, I know that Lustre routers have 
> been set up at the LLNL. What is the throughput obtained via the 
> routers, compared to the raw bandwidth of the interconnect?
> 
> Thanks,
> Sebastien.
> 
> 
> Brian J. Murrell a écrit :
>> On Wed, 2008-04-09 at 19:07 +0200, Sébastien Buisson wrote:
>>> I mean, if I 
>>> have an available bandwith of 100 on each side of a router, what will be 
>>> the max reachable bandwith from clients on one side of the router to 
>>> servers on the other side of the router? Is it 50? 80? 99? Is the 
>>> routing process CPU or memory hungry?
>> While I can't answer these things specifically another important
>> consideration is the bus architecture involved.  How many I/B cards can
>> you put on a bus before you saturate the bus?
>>
>> b.
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
>