[Lustre-discuss] iozone slow read for 64k record size 2.4 vs. 1.8.9

[JS Landry]

Hi,I'm testing lustre 2.4 with iozone and I can't find why the read of 
64k record size (1GB file) is so slow compare to 1.8.9client.

1.8.9 client
               KB  reclen   write rewrite    read    reread
          1048576      64  677521  794456  6130161  6204552
          1048576    1024  709112  862278  7165733  7152088

2.4.0 client
               KB  reclen   write rewrite    read    reread
          1048576      64  682344  897808  2334044  2331080
          1048576    1024  868466 1217273  4599784  4610098

(read/write in KB/s)

the only difference I can find is the offset in the rpc_stats file for 
each ost.
(but for the write...)


1.8.9 client

# cat /proc/fs/lustre/osc/lustreX-OST0000-osc-ffff880379be4800/rpc_stats
snapshot_time:            1377813075.916427 (secs.usecs)
read RPCs in flight:      0
write RPCs in flight:     0
dio read RPCs in flight:  0
dio write RPCs in flight: 0
pending write pages:      0
pending read pages:       0

                         read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                       0   0   0   |          0   0   0
2:                       0   0   0   |          0   0   0
4:                       0   0   0   |          0   0   0
8:                       0   0   0   |          0   0   0
16:                      0   0   0   |          0   0   0
32:                      0   0   0   |          0   0   0
64:                      0   0   0   |          0   0   0
128:                     0   0   0   |          0   0   0
256:                     0   0   0   |       1536 100 100

                         read                    write
rpcs in flight        rpcs   % cum % |       rpcs   % cum %
0:                       0   0   0   |       1536 100 100

                         read                    write
offset                rpcs   % cum % |       rpcs   % cum %
0:                       0   0   0   |       1536 100 100


on the 2.4.0client

# cat /proc/fs/lustre/osc/lustreX-OST0000-osc-ffff880375010800/rpc_stats
snapshot_time:         1377813072.97686 (secs.usecs)
read RPCs in flight:  0
write RPCs in flight: 0
pending write pages:  0
pending read pages:   0

                         read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                       0   0   0   |          0   0   0
2:                       0   0   0   |          0   0   0
4:                       0   0   0   |          0   0   0
8:                       0   0   0   |          0   0   0
16:                      0   0   0   |          0   0   0
32:                      0   0   0   |          0   0   0
64:                      0   0   0   |          0   0   0
128:                     0   0   0   |          0   0   0
256:                     0   0   0   |       1536 100 100

                         read                    write
rpcs in flight        rpcs   % cum % |       rpcs   % cum %
0:                       0   0   0   |          0   0   0
1:                       0   0   0   |       1536 100 100

                         read                    write
offset                rpcs   % cum % |       rpcs   % cum %
0:                       0   0   0   |         12   0   0
1:                       0   0   0   |          0   0   0
2:                       0   0   0   |          0   0   0
4:                       0   0   0   |          0   0   0
8:                       0   0   0   |          0   0   0
16:                      0   0   0   |          0   0   0
32:                      0   0   0   |          0   0   0
64:                      0   0   0   |          0   0   0
128:                     0   0   0   |          0   0   0
256:                     0   0   0   |         12   0   1
512:                     0   0   0   |         24   1   3
1024:                    0   0   0   |         48   3   6
2048:                    0   0   0   |         96   6  12
4096:                    0   0   0   |        192  12  25
8192:                    0   0   0   |        384  25  50
16384:                   0   0   0   |        768  50 100




my setup:
- 3 servers running v2_4_0-1dkms-1-PRISTINE-2.6.32-358.14.1.el6.x86_64
(1 mgs/mdt, 2 oss with 4 ost on each, all using zfs)
- 1st client 2.4.0-RC2-gd3f91c4-PRISTINE-2.6.32-358.6.2.el6.x86_64
- 2nd client 1.8.9 jenkins-wc1--PRISTINE-2.6.32-358.6.2.el6.x86_64

all 5 nodes (Sun x4270+x6575 with 24GBram) are running CentOS 6.4, 
interconnected with IB-QDR

I'm using a stripe count of 8 and a stripe sizeof 1M
and I set the following settings on the clients:
max_pages_per_rpc=256
max_rpcs_in_flight=32
max_dirty_mb=32

I didn't tune the servers yet, I would assume that the performance 
should be similar between 1.8.9 and 2.4.0 clients, right?

Does anyone have any ideas/tips/hints that would help me?

Oh, and btw, where is gone the brw_stats file in 2.4.0?

Thanks!
JS