<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Thomas,<br>
<br>
LEIBOVICI Thomas wrote:
<blockquote cite="mid:478C7B06.5020805@cea.fr" type="cite">You will
find in attachment to this mail some benchmarks we made on Linux with
pios over ZFS-DMU.
<br>
There are some interesting things about ZFS tuning and some ideas for
breaking a bottleneck we identified in DMU.
<br>
</blockquote>
<br>
Those are some very, very interesting benchmarks.<br>
<br>
Regarding the "ZFS striping performance", you noticed that increasing
the number of threads beyond a certain point didn't improve
performance, in fact it actually decreased.<br>
I think that was something expected due to the fact that the DMU
already has a good streamlined I/O pipeline which already parallelizes
I/O, and increasing the number of threads greatly beyond the number of
cpus causes contention to increase which causes I/O throughput to
decrease.<br>
<br>
However, it is in fact unfortunate that more luns didn't improve
performance. I wonder if you were hitting a CPU wall?<br>
In a previous benchmark I ran, I noticed that PIOS was not getting
improved throughput with more disks, even though there was still a
significant percentage of available CPU time.<br>
So I guess we still have opportunities to do good optimizations.<br>
<br>
The "ZIO threads" is also something that I highly suspected had an
impact in throughput, which is why when I benchmarked the DMU on the
Thumper I increased them from 8 to 24. It is good to have hard data
that confirms this.<br>
<br>
The section about parallelizing checksums is something that the ZFS
team appears to have solved already.<br>
You can see this code section:
<a class="moz-txt-link-freetext" href="http://www.wizy.org/mercurial/zfs-lustre/file/49c2aaa6a859/src/lib/libzfscommon/include/sys/zio_impl.h#101">http://www.wizy.org/mercurial/zfs-lustre/file/49c2aaa6a859/src/lib/libzfscommon/include/sys/zio_impl.h#101</a><br>
<br>
If you take a look at the "ZIO_WRITE_COMMON_STAGES", you will notice
that just before the "checksum generate" stage there is an "issue
async" stage. This "issue async" stage basically consists in
dispatching the I/O (ZIO) to the ZIO thread pool, which effectively
causes them to be parallelized. The I/O dependencies are automatically
tracked by the ZIO pipeline.<br>
<br>
All in all, this was a very good report.<br>
<br>
So far we have only done very limited benchmarking and optimization,
but we are already starting to work on performance improvements. One of
the tasks of our next development cycle will be doing this kind of
analysis but, of course, the sooner we see this, the better :)<br>
<br>
Great work and thanks for sharing this with us!<br>
<br>
Best regards,<br>
Ricardo<br>
<br>
<div class="moz-signature">-- <br>
<table border="0" cellpadding="0" cellspacing="0" width="450">
<tbody>
<tr valign="top">
<td height="121" width="121"><a href="http://www.sun.com"><img
src="cid:part1.09080809.09070401@Sun.COM" border="0" height="116"
width="121"></a></td>
<td style="font-family: Arial; font-size: 10px;" height="121"
width="329"><br>
<br>
<b>Ricardo Manuel Correia</b><br>
<br>
Lustre Engineering<br>
<b>Sun Microsystems, Inc.</b><br>
Portugal<br>
<br>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>