<span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 13px; border-collapse: collapse; "><div class="im" style="color: rgb(80, 0, 80); ">Howdy all,</div><div class="im" style="color: rgb(80, 0, 80); ">
I had a question from a colleague and did not have a ready answer.</div><div class="im" style="color: rgb(80, 0, 80); ">What is the community's experience with putting together a small and inexpensive cluster</div>
<div class="im" style="color: rgb(80, 0, 80); ">that serves Lustre from (some of) the compute nodes's local disks?</div><div class="im" style="color: rgb(80, 0, 80); "><br></div><div class="im" style="color: rgb(80, 0, 80); ">
They have run some simple tests with using a) just local disc, b) simple NFS service</div><div class="im" style="color: rgb(80, 0, 80); ">mounted to compute nodes, and c) Lustre with OSS and MDS on the same node.</div><div class="im" style="color: rgb(80, 0, 80); ">
A typical workload for them is to compile the "Visit" visualization package. On a local disk</div><div class="im" style="color: rgb(80, 0, 80); ">this takes 2 to 3 hours. On NFS it was closer to 24 hours, and on the small Lustre example</div>
<div class="im" style="color: rgb(80, 0, 80); ">it was about 5 hours. Now they'd like to go a little further and try to find a Lustre solution that</div><div class="im" style="color: rgb(80, 0, 80); ">would improve performance as compared to local disk. Their workload will be mostly metadata intensive</div>
<div class="im" style="color: rgb(80, 0, 80); ">rather than bulk I/O intensive. Is there any experience like that out there?</div><div class="im" style="color: rgb(80, 0, 80); ">Cheers,</div><div class="im" style="color: rgb(80, 0, 80); ">
Andrew</div><div class="im" style="color: rgb(80, 0, 80); "><br></div><div class="im" style="color: rgb(80, 0, 80); ">Notes from the one asking the question:</div><div class="im" style="color: rgb(80, 0, 80); ">----------------------------------------------------------<br>
What I would like to do now is to develop the cheapest<br>small cluster possible that still has good I/O<br></div>performance. NetAps raise the cost significantly. Also, I<div class="im" style="color: rgb(80, 0, 80); ">
<br>think the whole system must come out of the box with the<br></div>application and all dependencies built and good I/O.<div class="im" style="color: rgb(80, 0, 80); "><br><br>So one possible way would be a system with a head node and<br>
N compute nodes, each with multiple CPUs and cores, of<br>course. I can then imagine a Lustre file system with the<br>MDS on the head node and perhaps M OSSs on the compute<br>nodes, which then serve up their local disks. Of course,<br>
now the compute nodes are running both the computational<br>application (on all cores likely) and 0 or 1 OSS.<br><br>It sounds like from what you are saying that at a minimum<br>I would need two interfaces per node: one over which the<br>
MPI communication goes for the apps, and one for serving<br>the Lustre file system on those nodes which are serving<br>that. Is this right?<br><br></div>Is this a reasonable direction to go? (Having both OSS and<br>computation on some nodes.)<br>
<br>Are there examples of good systems designs out there?<br><br></span>