[Lustre-discuss] Performance problems with Lustre 1.6.1

[Andreas Dilger]

On Oct 09, 2007  17:13 -0700, Juan Piernas Canovas wrote:
> Andreas Dilger wrote:
> >On Oct 01, 2007  15:56 -0700, Juan Piernas Canovas wrote:
> >>The problem that I have is that, when the stripe size is 1MB (what means 
> >>that there are 1024 chunks in total, or 128 chunks per OST), it takes 
> >>more than 400 seconds to read the file, and the network traffic is very 
> >>high. However, if the stripe size is 128 MB (8 chunks altogether, one 
> >>per OST), it takes only around 100 seconds to read the file, and the 
> >>network traffic is 1/10th the previous one. Note that, in both cases, 
> >>the data I/O operations are local and that the processes read the same 
> >>amount of data.
> >
> >It sounds like the readahead is reading the "unused" parts of the file
> >on the other OSTs.  Are you also reading data from disk in 1MB chunks,
> >or in smaller chunks?  You should read at the stripe size for best
> >performance in this test.
> 
> Thank you. You are right. One problem was that the readahead made a 
> process on an OST read chunks from other OSTs. That explains the network 
> traffic. The other problem was the size of the I/O requests, which was 
> too "small" (16 KB). The interesting point is that the (small) request 
> size was the same in both configurations (1 MB and 128 MB stripe sizes) 
> but, even then, the times were very different.

In the 128MB stripe case, the readahead is mostly reading within the
"local" stripe, so the overhead is minimal (some overflow into the next
stripe but not so much).  In the 1MB stripe case, the majority of the
readahead will be in irrelevant parts of the filesystem and will slow
down the overall performance.

If you read with read size == stripe size you will get "random" read
heuristics (i.e. no readahead) and performance should be good.  Of
course, Lustre _should_ implement smarter strided readahead, but it
doesn't yet (patches welcome :-).

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.