[lustre-discuss] Do I need Lustre?

[Colin Faber]

The type of storage you decide on should be a factor of the workloads that
you plan on executing against, as well as the back-end storage systems.
Having all SSD based file systems can be useful in the case of small /
random IO however in many cases the controllers will not be able to take
advantage of the performance provided by the SSD array

 As a result you end up paying a whole lot more for a whole lot less space
and ending up with the same overall performance.

On Mon, Apr 30, 2018, 7:11 AM Thackeray, Neil L <neilt at illinois.edu> wrote:

> Sorry, I left out file size. We don't foresee growing tremendously. The
> plan is for researchers to upload their data, get the results, and copy it
> down to a mounted file system. This is going to be used by multiple
> researchers, and we will be charging for compute time. We really don't want
> this cluster to be used for storing data outside of the time needed for
> their computations. We may just start with 100TB of SSD storage.
>
> -----Original Message-----
> From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> On Behalf
> Of Philippe Weill
> Sent: Saturday, April 28, 2018 1:14 AM
> To: lustre-discuss at lists.lustre.org
> Subject: Re: [lustre-discuss] Do I need Lustre?
>
>
>
> Le 27/04/2018 à 19:07, Thackeray, Neil L a écrit :
> > I’m new to the cluster realm, so I’m hoping for some good advice. We
> > are starting up a new cluster, and I’ve noticed that lustre seems to be
> used widely in datacenters. The thing is I’m not sure the scale of our
> cluster will need it.
> >
> > We are planning a small cluster, starting with 6 -8 nodes with 2 GPUs
> > per node. They will be used for Deep Learning, MRI data processing,
> > and Matlab among other things. With the size of the cluster we figure
> > that 10Gb networking will be sufficient. We aren’t going to allow
> persistent storage on the cluster. Users will just upload and download
> data. I’m mostly concerned about I/O speeds. I don’t know if NFS would be
> fast enough to handle the data.
> >
> > We are hoping that the cluster will grow over time. We are already
> talking about buying more nodes next fiscal year.
> >
> > Thanks.
> >
>
> hello
>
> you didn't say anything about filesystem size needed and if you are
> thinking to grow fast we also run a small cluster ( 20 nodes ) but for
> climate data modeling results and satellite atmospheric data analysis we
> are growing at least 300TB per year (2PB now) and it's easier for us to
> grow with lustre
>
>
> --
> Weill Philippe -  Administrateur Systeme et Reseaux
> CNRS/UPMC/IPSL   LATMOS (UMR 8190)
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20180430/ea51b14a/attachment.html>