[Lustre-discuss] read fragmentation

[Frank Riley]

I'm trying to diagnose why we are getting bad read performance with a brand new Lustre 2.1.1 install. It looks like fragmentation may be contributing to the problem. Looking at rpc_stats on the client, I can see that 9% of the rpcs are 1 page in size. The remaining 91% are 256 pages as expected. I have already applied the patch from LU-983, and this took care of the fragments I was seeing that were > 1 and < 256 pages in size. I'm not sure what is causing the 1 page fragments though. Any ideas what would be causing this?

I have tested on the OSS with obdfilter_survey and there is no fragmentation there and the read speeds are as expected. I have tested the network with lnet selftest and get 3GB/s so that is also good. The fragmentation is somewhere on the client. The setup I am testing is as follows:

1 MGS/MDS/MDT
1 OSS with 3 OSTs, each OST is 4+2 RAID6 with 128K stripe size (512K stripe width), each OST is 11TB
The journals for the OST file systems are on a different disk.
1 client
All connected with IB 40Gb.

I am testing on the client with the following iozone command:

iozone -w -M -t 3 -s 20G -r 1m -i0 -i1

I have verified that the 3 files that iozone writes here end up on different OSTs. This is backed up by the fact that the write speed maxes out. The read, however, comes in at less than 50% of where I would expect.