<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
I am running dd 10 times consecutively to read a 64GB file (
stripeCount=4 stripeSize=4M ) on a Lustre client(version 2.10.3)
that has 64GB of memory.<br>
The client node was dedicated.<br>
<br>
<b><font face="Courier New, Courier, monospace">for pass in 1 2 3 4
5 6 7 8 9 10<br>
do<br>
of=/dev/null if=${file} count=128000 bs=512K<br>
done<br>
</font></b><br>
Instrumentation of the I/O from dd reveals varying performance. In
the plot below, the bottom frame has wall time<br>
on the X axis, and file position of the dd reads on the Y axis, with
a dot plotted at the wall time and starting file position of every
read. <br>
The slopes of the lines indicate the data transfer rate, which vary
from 475MB/s to 1.5GB/s. The last 2 passes have sharp breaks<br>
in the performance, one with increasing performance, and one with
decreasing performance.<br>
<br>
The top frame indicates the amount of memory used by each of the
file's 4 OSCs over the course of the 10 dd runs. Nothing terribly
odd here except that<br>
one of the OSC's eventually has its entire stripe ( 16GB ) cached
and then never gives any up.<br>
<br>
I should mention that the file system has 320 OSTs. I found LU-6370
which eventually started discussing LRU management issues on systems
with high<br>
numbers of OST's leading to reduced RPC sizes.<br>
<br>
Any explanations for the varying performance?<br>
Thanks, <br>
John<br>
<br>
<img src="cid:part1.FE7755F6.C36ADB75@iodoctors.com" alt="">
<pre class="moz-signature" cols="72">--
I/O Doctors, LLC
507-766-0378
<a class="moz-txt-link-abbreviated" href="mailto:bauerj@iodoctors.com">bauerj@iodoctors.com</a></pre>
</body>
</html>