[lustre-discuss] Do I need Lustre?

[Thackeray, Neil L]

Sorry, I left out file size. We don't foresee growing tremendously. The plan is for researchers to upload their data, get the results, and copy it down to a mounted file system. This is going to be used by multiple researchers, and we will be charging for compute time. We really don't want this cluster to be used for storing data outside of the time needed for their computations. We may just start with 100TB of SSD storage.

-----Original Message-----
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> On Behalf Of Philippe Weill
Sent: Saturday, April 28, 2018 1:14 AM
To: lustre-discuss at lists.lustre.org
Subject: Re: [lustre-discuss] Do I need Lustre?



Le 27/04/2018 à 19:07, Thackeray, Neil L a écrit :
> I’m new to the cluster realm, so I’m hoping for some good advice. We 
> are starting up a new cluster, and I’ve noticed that lustre seems to be used widely in datacenters. The thing is I’m not sure the scale of our cluster will need it.
> 
> We are planning a small cluster, starting with 6 -8 nodes with 2 GPUs 
> per node. They will be used for Deep Learning, MRI data processing, 
> and Matlab among other things. With the size of the cluster we figure 
> that 10Gb networking will be sufficient. We aren’t going to allow persistent storage on the cluster. Users will just upload and download data. I’m mostly concerned about I/O speeds. I don’t know if NFS would be fast enough to handle the data.
> 
> We are hoping that the cluster will grow over time. We are already talking about buying more nodes next fiscal year.
> 
> Thanks.
> 

hello

you didn't say anything about filesystem size needed and if you are thinking to grow fast we also run a small cluster ( 20 nodes ) but for climate data modeling results and satellite atmospheric data analysis we are growing at least 300TB per year (2PB now) and it's easier for us to grow with lustre


--
Weill Philippe -  Administrateur Systeme et Reseaux
CNRS/UPMC/IPSL   LATMOS (UMR 8190)
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org