[Lustre-discuss] Off-topic: largest existing Lustre file system?
mlbarna
mlbarna at sandia.gov
Wed Jan 16 09:43:05 PST 2008
Could you elaborate on the benchmarking application(s) run that provided
these bandwidth numbers. I have a particular interest in MPI coded programs
that perform collective I/O. In discussions, I find this topic sometimes
confused; my meaning is streamed, appending with all the data from all the
processors for a single, atomic write operation filling disjoint sections of
the same file. In MPI-IO, the MPI_File_write_all* family seems to define my
focus area, run with or without two-phase aggregation. Imitating the
operation with simple, Posix I/O is acceptable, as far as I am concerned.
In tests on redstorm from last year, I appended to a single, open file at a
rate of 26 GB/s. I had to use exceptional parameters to achieve this
however: the file had an LFS stripe-count of 160, and I sent a 20 MB buffer,
respectively, from each of a 160, total processor job, for an aggregate of
3.2 GB per write_all operation. I consider this configuration out of the
range of any normal usage.
I believe that a faster rate could be achieved by a similar program that
wrote independently--that is, one-file-per-processor--such as via NetCDF.
For this case, I would set the LFS stripe-count down to one.
Marty Barnaby
On 1/14/08 4:11 PM, "Canon, Richard Shane" <canonrs at ornl.gov> wrote:
>
> Jeff,
>
> I'm not aware of any. For parallel file systems it is usually bandwidth
> centric.
>
> --Shane
>
> -----Original Message-----
> From: Kennedy, Jeffrey [mailto:jkennedy at qualcomm.com]
> Sent: Monday, January 14, 2008 4:56 PM
> To: Canon, Richard Shane; lustre-discuss at clusterfs.com
> Subject: RE: [Lustre-discuss] Off-topic: largest existing Lustre file
> system?
>
> Any spec's on IOPS rather than throughput?
>
> Thanks.
>
> Jeff Kennedy
> QCT Engineering Compute
> 858-651-6592
>
>> -----Original Message-----
>> From: lustre-discuss-bounces at clusterfs.com [mailto:lustre-discuss-
>> bounces at clusterfs.com] On Behalf Of Canon, Richard Shane
>> Sent: Monday, January 14, 2008 1:49 PM
>> To: lustre-discuss at clusterfs.com
>> Subject: Re: [Lustre-discuss] Off-topic: largest existing Lustre file
>> system?
>>
>>
>> Klaus,
>>
>> Here are some that I know are pretty large.
>>
>> * RedStorm - I think it has two roughly 50 GB/s file systems. The
>> capacity may not be quite as large though. I think they used FC
> drives.
>> It was DDN 8500 although that may have changed.
>> * CEA - I think they have a file system approaching 100 GB/s. I think
>> it is DDN 9550. Not sure about the capacities.
>> * TACC has a large Thumper based system. Not sure of the specs.
>> * ORNL - We have a 44 GB/s file system with around 800 TB of total
>> capacity. That is DDN 9550. We also have two new file system (20
> GB/s
>> and 10 GB/s currently LSI XBB2 and DDN 9550 respectively). Those have
>> around 800 TB each (after RAID6).
>> * We are planning a 200 GB/s, around 10 PB file system now.
>>
>> --Shane
>>
>> -----Original Message-----
>> From: lustre-discuss-bounces at clusterfs.com
>> [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of D. Marc
>> Stearman
>> Sent: Monday, January 14, 2008 4:37 PM
>> To: lustre-discuss at clusterfs.com
>> Subject: Re: [Lustre-discuss] Off-topic: largest existing Lustre file
>> system?
>>
>> Klaus,
>>
>> We currently have a 1.2PB lustre filesystem that we will be expanding
>> to 2.4PB in the near future. I not sure about the highest sustained
>> IOPS, but we did have a user peak 19GB/s to one of our 500TB
>> filesystems recently. The backend for that was 16 DDN 8500 couplets
>> with write-cache turned OFF.
>>
>> -Marc
>>
>> ----
>> D. Marc Stearman
>> LC Lustre Systems Administrator
>> marc at llnl.gov
>> 925.423.9670
>> Pager: 1.888.203.0641
>>
>>
>> On Jan 14, 2008, at 12:41 PM, Klaus Steden wrote:
>>
>>>
>>> Hi there,
>>>
>>> I was asked by a friend of a business contact of mine the other day
>>> to share
>>> some information about Lustre; seems he's planning to build what
> will
>>> eventually be about a 3 PB file system.
>>>
>>> The CFS website doesn't appear to have any information on field
>>> deployments
>>> worth bragging about, so I figured I'd ask, just for fun; does
>>> anyone know:
>>>
>>> - the size of the largest working Lustre file system currently in
>>> the field
>>> - the highest sustained number of IOPS seen with Lustre, and what
> the
>>> backend was?
>>>
>>> cheers,
>>> Klaus
>>>
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at clusterfs.com
>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at clusterfs.com
>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at clusterfs.com
>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>
More information about the lustre-discuss
mailing list