[Lustre-discuss] How to achieve 20GB/s file system throughput?

[Joe Landman]

On 07/23/2010 10:25 PM, Henry_Xu at Dell.com wrote:
> Hello,
>
> One of my customer want to set up HPC with thousands of compute nodes.
> The parallel file system should have 20GB/s throughput. I am not sure
> whether lustre can make it. How many IO nodes needed to achieve this target?

I hate to say "it depends" but, it does in fact depend upon many things. 
  What type of IO is the customer doing; large block sequential spread 
out over many nodes (parallel IO), or small block random, or a mixture?

It is possible to achieve 20GB/s, and quite a bit more, using Lustre. 
As to whether or not that 20GB/s is meaningful to their code(s), thats a 
different question.  It would be 20GB/s in aggregate, over possibly many 
compute nodes doing IO.

> My assumption is 100 or more IO nodes(rack servers) are needed.

Hmmm ... If you can achieve 500+ MB/s per OST, then you would need about 
40 OSTs.  You can have each OSS handle several OSTs.  There are 
efficiency losses you should be aware of, but 20GB/s using some 
mechanism to measure this, should be possible with a realistic number of 
units.  Don't forget to count efficiency losses in the design.

100 IO nodes ... I presume you mean OSSes?

If your units are slower, then yes, you will need more of them to 
achieve this performance.

You would need to make sure you have a well designed and correctly 
functional Infiniband infrastructure in addition to the other issues. 
We've found that Lustre is ... very sensitive ... to the Infiniband 
implementation.

Regards,

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615