[lustre-discuss] Do I need Lustre?

[Philippe Weill]

Le 27/04/2018 à 19:07, Thackeray, Neil L a écrit :
> I’m new to the cluster realm, so I’m hoping for some good advice. We are starting up a new cluster, and I’ve noticed that lustre 
> seems to be used widely in datacenters. The thing is I’m not sure the scale of our cluster will need it.
> 
> We are planning a small cluster, starting with 6 -8 nodes with 2 GPUs per node. They will be used for Deep Learning, MRI data 
> processing, and Matlab among other things. With the size of the cluster we figure that 10Gb networking will be sufficient. We aren’t 
> going to allow persistent storage on the cluster. Users will just upload and download data. I’m mostly concerned about I/O speeds. I 
> don’t know if NFS would be fast enough to handle the data.
> 
> We are hoping that the cluster will grow over time. We are already talking about buying more nodes next fiscal year.
> 
> Thanks.
> 

hello

you didn't say anything about filesystem size needed and if you are thinking to grow fast
we also run a small cluster ( 20 nodes )
but for climate data modeling results and satellite atmospheric data analysis
we are growing at least 300TB per year (2PB now)
and it's easier for us to grow with lustre


-- 
Weill Philippe -  Administrateur Systeme et Reseaux
CNRS/UPMC/IPSL   LATMOS (UMR 8190)