<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Andreas Dilger a écrit :
<blockquote cite="mid:20080310045315.GB5851@webber.adilger.int"
type="cite">
<pre wrap="">On Mar 07, 2008 12:49 +0100, Joe Barjo wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I made some more tests, and have setup a micro lustre cluster on lvm
volumes.
node a: MDS
node b and c: OST
node a,b,c,d,e,f: clients
Gigabit ethernet network.
Made the optimizations: lnet.debug=0, lru_size to 10000, max_dirty_mb to
1024
</pre>
</blockquote>
<pre wrap=""><!---->
For high RPC-rate operations using an interconnect like Infiniband is
better than ethernet.
</pre>
</blockquote>
infiniband is not in our budget...<br>
<blockquote cite="mid:20080310045315.GB5851@webber.adilger.int"
type="cite">
<pre wrap=""></pre>
<blockquote type="cite">
<pre wrap="">The svn checkout takes 50s ( 15s on a localdisk, 25s on a local lustre
demo (with debug=0))
Launching gkrellm, a single svn checkout consumes about 20% of the MDS
system cpu with about 2.4mbyte/sec ethernet communication.
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
<blockquote type="cite">
<pre wrap="">About 6MByte/s disk bandwidth on OST1, up to 12-16MB/s on OST2 disk
bandwidth, network bandwidth on OST is about 10 to 20 times under disk
bandwidth.
Why so much disk bandwidth on OSTs, is it a readahead problem?
</pre>
</blockquote>
<pre wrap=""><!---->
That does seem strange, I can't really say why. There is some metadata
overhead, and that is higher with small files, but I don't think it
would be 10-20x overhead.
</pre>
</blockquote>
The checkouted source is only 65 megabytes. So much OST disk bandwidth
is probably not normal.<br>
Maybe you should verify this point.<br>
Are you sure there isn't an optimazation for this? This looks like
readahead or something like that.<br>
<blockquote cite="mid:20080310045315.GB5851@webber.adilger.int"
type="cite">
<pre wrap=""></pre>
<blockquote type="cite">
<pre wrap="">I launched a compilation distributed on the 6 clients:
MDS system cpu goes up to 60% system ressource (athlon 64 3500+)
12MByte/s on the ethernet, OST goes up to the same level as previous test.
How come is there so much network communications on the MDT?
</pre>
</blockquote>
<pre wrap=""><!---->
Because every metadata operation has to be done on the MDS currently.
We are working toward having metadata writeback cache operations on
the client, but it doesn't happen currently. For operations like
compilation it is basically entirely metadata overhead.
</pre>
<blockquote type="cite">
<pre wrap="">As I understood that the MDS can not be load balanced, I don't see how
lustre is scalable to thousands of clients...
</pre>
</blockquote>
<pre wrap=""><!---->
Because in many HPC environments there are very few metadata operations
in comparison to the amount of data being read/written. Average file
sizes are 20-30MB instead of 20-30kB.
</pre>
<blockquote type="cite">
<pre wrap="">It looks like lustre is not made for this kind of application
</pre>
</blockquote>
<pre wrap=""><!---->
No, it definitely isn't tuned for small files.
</pre>
</blockquote>
Could it be tuned one day for small files?<br>
Which filesystem would you suggest for me?<br>
I already tried nfs, afs<br>
I will now try glusterfs<br>
<br>
Thanks for your support<br>
<br>
</body>
</html>