[Lustre-discuss] Slow performance on the first IOR / iozone (Lustre)

[Anand Bisen]

Hi,

I have been trying to understand why successive execution of iozone  
(also with dd and IOR) produces increasing performance from first run  
to third run. After the third run the performance is constant.

So when a new OSS is put into service or if an OSS server is rebooted  
and it joins the Lustre file system this characteristics is visible  
for each OST that it serves. Once the OST's reach a peak performance  
that performance is consistent, but once the OST's are unmounted and  
mounted back the same escalating performance is seen. Once the OST has  
been primed again mounting from a new client will also get's the same  
peak performance. And this characteristics is only visible for writes  
and not reads.

We have tried multiple scenario's, using one OST to multiple OST's. We  
are using "noop" as the IO scheduler on our OSS servers.

---Example with DD's---

mds2:/mnt/lustre/test # lfs setstripe foo 0 2 1         (We have tried  
each OST here same effect)
mds2:/mnt/lustre/test # dd if=/dev/zero of=foo bs=1048576 count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 43.7827 seconds, 196 MB/s
mds2:/mnt/lustre/test # dd if=/dev/zero of=foo bs=1048576 count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 33.153 seconds, 259 MB/s
mds2:/mnt/lustre/test # dd if=/dev/zero of=foo bs=1048576 count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 32.9747 seconds, 261 MB/s
mds2:/mnt/lustre/test # dd if=/dev/zero of=foo bs=1048576 count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 29.3566 seconds, 293 MB/s
mds2:/mnt/lustre/test # dd if=/dev/zero of=foo bs=1048576 count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 22.23 seconds, 386 MB/s
mds2:/mnt/lustre/test # dd if=/dev/zero of=foo bs=1048576 count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 22.236 seconds, 386 MB/s
mds2:/mnt/lustre/test # dd if=/dev/zero of=foo bs=1048576 count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 22.2838 seconds, 385 MB/s


---Example with IOZONE---
client-mnt # iozone -i 0 -s 10g -r 1m -t 2

         File size set to 10485760 KB
         Record Size 1024 KB
         Command line used: iozone -i 0 -s 10g -r 1m -t 2
         Output is in Kbytes/sec
         Time Resolution = 0.000001 seconds.
         Processor cache size set to 1024 Kbytes.
         Processor cache line size set to 32 bytes.
         File stride size set to 17 * record size.
         Throughput test with 2 processes
         Each process writes a 10485760 Kbyte file in 1024 Kbyte records

         Children see throughput for  2 initial writers  =  151909.33  
KB/sec
         Parent sees throughput for  2 initial writers   =  150657.64  
KB/sec
         Min throughput per process                      =   75476.66  
KB/sec
         Max throughput per process                      =   76432.67  
KB/sec
         Avg throughput per process                      =   75954.66  
KB/sec
         Min xfer                                        = 10354688.00  
KB

         Children see throughput for  2 rewriters        =  249051.45  
KB/sec
         Parent sees throughput for  2 rewriters         =  247016.15  
KB/sec
         Min throughput per process                      =  124407.29  
KB/sec
         Max throughput per process                      =  124644.16  
KB/sec
         Avg throughput per process                      =  124525.72  
KB/sec
         Min xfer                                        = 10465280.00  
KB


#######Second RUN

         Children see throughput for  2 initial writers  =  244629.49  
KB/sec
         Parent sees throughput for  2 initial writers   =  242007.64  
KB/sec
         Min throughput per process                      =  121223.02  
KB/sec
         Max throughput per process                      =  123406.48  
KB/sec
         Avg throughput per process                      =  122314.75  
KB/sec
         Min xfer                                        = 10300416.00  
KB

         Children see throughput for  2 rewriters        =  239316.48  
KB/sec
         Parent sees throughput for  2 rewriters         =  238836.01  
KB/sec
         Min throughput per process                      =  119017.80  
KB/sec
         Max throughput per process                      =  120298.68  
KB/sec
         Avg throughput per process                      =  119658.24  
KB/sec
         Min xfer                                        = 10375168.00  
KB

#######Third RUN

         Children see throughput for  2 initial writers  =  245567.60  
KB/sec
         Parent sees throughput for  2 initial writers   =  241539.16  
KB/sec
         Min throughput per process                      =  121565.91  
KB/sec
         Max throughput per process                      =  124001.69  
KB/sec
         Avg throughput per process                      =  122783.80  
KB/sec
         Min xfer                                        = 10279936.00  
KB

         Children see throughput for  2 rewriters        =  240782.11  
KB/sec
         Parent sees throughput for  2 rewriters         =  240390.69  
KB/sec
         Min throughput per process                      =  119717.36  
KB/sec
         Max throughput per process                      =  121064.75  
KB/sec
         Avg throughput per process                      =  120391.05  
KB/sec
         Min xfer                                        = 10370048.00  
KB

Any insight on explaining this behavior would be really appreciated.

Thanks

Anand