[Lustre-discuss] Off-topic: largest existing Lustre file system?

mlbarna mlbarna at sandia.gov
Wed Jan 16 09:43:05 PST 2008


Could you elaborate on the benchmarking application(s) run that provided
these bandwidth numbers. I have a particular interest in MPI coded programs
that perform collective I/O. In discussions, I find this topic sometimes
confused; my meaning is streamed, appending with all the data from all the
processors for a single, atomic write operation filling disjoint sections of
the same file. In MPI-IO, the MPI_File_write_all* family seems to define my
focus area, run with or without two-phase aggregation. Imitating the
operation with simple, Posix I/O is acceptable, as far as I am concerned.

In tests on redstorm from last year, I appended to a single, open file at a
rate of 26 GB/s. I had to use exceptional parameters to achieve this
however: the file had an LFS stripe-count of 160, and I sent a 20 MB buffer,
respectively, from each of a 160, total processor job, for an aggregate of
3.2 GB per write_all operation. I consider this configuration out of the
range of any normal usage.

I believe that a faster rate could be achieved by a similar program that
wrote independently--that is, one-file-per-processor--such as via NetCDF.
For this case, I would set the LFS stripe-count down to one.


Marty Barnaby



On 1/14/08 4:11 PM, "Canon, Richard Shane" <canonrs at ornl.gov> wrote:

> 
> Jeff,
> 
> I'm not aware of any.  For parallel file systems it is usually bandwidth
> centric.
> 
> --Shane
> 
> -----Original Message-----
> From: Kennedy, Jeffrey [mailto:jkennedy at qualcomm.com]
> Sent: Monday, January 14, 2008 4:56 PM
> To: Canon, Richard Shane; lustre-discuss at clusterfs.com
> Subject: RE: [Lustre-discuss] Off-topic: largest existing Lustre file
> system?
> 
> Any spec's on IOPS rather than throughput?
> 
> Thanks.
> 
> Jeff Kennedy
> QCT Engineering Compute
> 858-651-6592
>  
>> -----Original Message-----
>> From: lustre-discuss-bounces at clusterfs.com [mailto:lustre-discuss-
>> bounces at clusterfs.com] On Behalf Of Canon, Richard Shane
>> Sent: Monday, January 14, 2008 1:49 PM
>> To: lustre-discuss at clusterfs.com
>> Subject: Re: [Lustre-discuss] Off-topic: largest existing Lustre file
>> system?
>> 
>> 
>> Klaus,
>> 
>> Here are some that I know are pretty large.
>> 
>> * RedStorm - I think it has two roughly 50 GB/s file systems.  The
>> capacity may not be quite as large though.  I think they used FC
> drives.
>> It was DDN 8500 although that may have changed.
>> * CEA - I think they have a file system approaching 100 GB/s.  I think
>> it is DDN 9550.  Not sure about the capacities.
>> * TACC has a large Thumper based system.  Not sure of the specs.
>> * ORNL - We have a 44 GB/s file system with around 800 TB of total
>> capacity.  That is DDN 9550.  We also have two new file system (20
> GB/s
>> and 10 GB/s currently LSI XBB2 and DDN 9550 respectively).  Those have
>> around 800 TB each (after RAID6).
>> * We are planning a 200 GB/s, around 10 PB file system now.
>> 
>> --Shane
>> 
>> -----Original Message-----
>> From: lustre-discuss-bounces at clusterfs.com
>> [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of D. Marc
>> Stearman
>> Sent: Monday, January 14, 2008 4:37 PM
>> To: lustre-discuss at clusterfs.com
>> Subject: Re: [Lustre-discuss] Off-topic: largest existing Lustre file
>> system?
>> 
>> Klaus,
>> 
>> We currently have a 1.2PB lustre filesystem that we will be expanding
>> to 2.4PB in the near future.  I not sure about the highest sustained
>> IOPS, but we did have a user peak 19GB/s to one of our 500TB
>> filesystems recently. The backend for that was 16 DDN 8500 couplets
>> with write-cache turned OFF.
>> 
>> -Marc
>> 
>> ----
>> D. Marc Stearman
>> LC Lustre Systems Administrator
>> marc at llnl.gov
>> 925.423.9670
>> Pager: 1.888.203.0641
>> 
>> 
>> On Jan 14, 2008, at 12:41 PM, Klaus Steden wrote:
>> 
>>> 
>>> Hi there,
>>> 
>>> I was asked by a friend of a business contact of mine the other day
>>> to share
>>> some information about Lustre; seems he's planning to build what
> will
>>> eventually be about a 3 PB file system.
>>> 
>>> The CFS website doesn't appear to have any information on field
>>> deployments
>>> worth bragging about, so I figured I'd ask, just for fun; does
>>> anyone know:
>>> 
>>> - the size of the largest working Lustre file system currently in
>>> the field
>>> - the highest sustained number of IOPS seen with Lustre, and what
> the
>>> backend was?
>>> 
>>> cheers,
>>> Klaus
>>> 
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at clusterfs.com
>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>> 
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at clusterfs.com
>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>> 
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at clusterfs.com
>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> 





More information about the lustre-discuss mailing list