[lustre-discuss] varying sequential read performance.

[Colin Faber]

Are you flushing cache between test runs?

On Mon, Apr 2, 2018, 6:06 PM John Bauer <bauerj at iodoctors.com> wrote:

> I am running dd 10 times consecutively to  read a 64GB file (
> stripeCount=4 stripeSize=4M ) on a Lustre client(version 2.10.3) that has
> 64GB of memory.
> The client node was dedicated.
>
>
>
>
>
> *for pass in 1 2 3 4 5 6 7 8 9 10 do    of=/dev/null if=${file}
> count=128000 bs=512K done *
> Instrumentation of the I/O from dd reveals varying performance.  In the
> plot below, the bottom frame has wall time
> on the X axis, and file position of the dd reads on the Y axis, with a dot
> plotted at the wall time and starting file position of every read.
> The slopes of the lines indicate the data transfer rate, which vary from
> 475MB/s to 1.5GB/s.  The last 2 passes have sharp breaks
> in the performance, one with increasing performance, and one with
> decreasing performance.
>
> The top frame indicates the amount of memory used by each of the file's 4
> OSCs over the course of the 10 dd runs.  Nothing terribly odd here except
> that
> one of the OSC's eventually has its entire stripe ( 16GB ) cached and then
> never gives any up.
>
> I should mention that the file system has 320 OSTs.  I found LU-6370 which
> eventually started discussing LRU management issues on systems with high
> numbers of OST's leading to reduced RPC sizes.
>
> Any explanations for the varying performance?
> Thanks,
> John
>
> --
> I/O Doctors, LLC
> 507-766-0378bauerj at iodoctors.com
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20180403/804744f7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: johbmffmkkegkbkh.png
Type: image/png
Size: 34134 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20180403/804744f7/attachment-0001.png>