[lustre-discuss] [HPDD-discuss] Lustre Server Sizing

Indivar Nair indivar.nair at techterra.in
Tue Jul 21 10:26:25 PDT 2015


Hi Scott,

The 3 - SAN Storages with 240 disks each has its own 3 NAS Headers (NAS
Appliances).
However, even with 240 10K RPM disk and RAID50, it is only providing around
1.2 - 1.4GB/s per NAS Header.

There is no clustered file system, and each NAS Header has its own
file-system.
It uses some custom mechanism to present the 3 file systems as single name
space.
But the directories have to be manually spread across for load-balancing.
As you can guess, this doesn't work most of the time.
Many a times, most of the compute nodes access a single NAS Header,
overloading it.

The customer wants *at least* 9GB/s throughput from a single file-system.

But I think, if we architect the Lustre Storage correctly, with these many
disks, we should get at least 18GB/s throughput, if not more.

Regards,


Indivar Nair


On Tue, Jul 21, 2015 at 10:15 PM, Scott Nolin <scott.nolin at ssec.wisc.edu>
wrote:

> An important question is what performance do they have now, and what do
> they expect if converting it to Lustre. Our more basically, what are they
> looking for in general in changing?
>
> The performance requirements may help drive your OSS numbers for example,
> or interconnect, and all kinds of stuff.
>
> Also I don't have a lot of experience with NFS/CIFS gateways, but that is
> perhaps it's own topic and may need some close attention.
>
> Scott
>
> On 7/21/2015 10:57 AM, Indivar Nair wrote:
>
>> Hi ...,
>>
>> One of our customers has a 3 x 240 Disk SAN Storage Array and would like
>> to convert it to Lustre.
>>
>> They have around 150 Workstations and around 200 Compute (Render) nodes.
>> The File Sizes they generally work with are -
>> 1 to 1.5 million files (images) of 10-20MB in size.
>> And a few thousand files of 500-1000MB in size.
>>
>> Almost 50% of the infra is on MS Windows or Apple MACs
>>
>> I was thinking of the following configuration -
>> 1 MDS
>> 1 Failover MDS
>> 3 OSS (failover to each other)
>> 3 NFS+CIFS Gateway Servers
>> FDR Infiniband backend network (to connect the Gateways to Lustre)
>> Each Gateway Server will have 8 x 10GbE Frontend Network (connecting the
>> clients)
>>
>> *Option A*
>>      10+10 Disk RAID60 Array with 64KB Chunk Size i.e. 1MB Stripe Width
>>      720 Disks / (10+10) = 36 Arrays.
>>      12 OSTs per OSS
>>      18 OSTs per OSS in case of Failover
>>
>> *Option B*
>>      10+10+10+10 Disk RAID60 Array with 128KB Chunk Size i.e. 4MB Stripe
>> Width
>>      720 Disks / (10+10+10+10) = 18 Arrays
>>      6 OSTs per OSS
>>      9 OSTs per OSS in case of Failover
>>      4MB RPC and I/O
>>
>> *Questions*
>> 1. Would it be better to let Lustre do most of the striping / file
>> distribution (as in Option A) OR would it be better to let the RAID
>> Controllers do it (as in Option B)
>>
>> 2. Will Option B allow us to have lesser CPU/RAM than Option A?
>>
>> Regards,
>>
>>
>> Indivar Nair
>>
>>
>>
>> _______________________________________________
>> HPDD-discuss mailing list
>> HPDD-discuss at lists.01.org
>> https://lists.01.org/mailman/listinfo/hpdd-discuss
>>
>>
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20150721/a08eeb56/attachment.htm>


More information about the lustre-discuss mailing list