[Lustre-discuss] Performance problems with Lustre 1.6.1

Juan Piernas Canovas juan.piernascanovas at pnl.gov
Tue Oct 9 17:13:53 PDT 2007


Andreas Dilger wrote:
> On Oct 01, 2007  15:56 -0700, Juan Piernas Canovas wrote:
>   
>> I have set up a small Lustre file system with 1 MDS and 8 OSS/OST. The 
>> particularity of our system is that every OSS is also a client of the 
>> file system (there are 8 clients altogether).
>>
>> The file system has a 1 GB file striped across all the OSTs. On every 
>> OST, there is a process which reads the file chunks stored locally, 
>> e.g., in its own OST (since the processes have the striping information 
>> of the file, each one knows which portions of the file are stored in its 
>> OST).
>>
>> The problem that I have is that, when the stripe size is 1MB (what means 
>> that there are 1024 chunks in total, or 128 chunks per OST), it takes 
>> more than 400 seconds to read the file, and the network traffic is very 
>> high. However, if the stripe size is 128 MB (8 chunks altogether, one 
>> per OST), it takes only around 100 seconds to read the file, and the 
>> network traffic is 1/10th the previous one. Note that, in both cases, 
>> the data I/O operations are local and that the processes read the same 
>> amount of data.
>>     
>
> It sounds like the readahead is reading the "unused" parts of the file
> on the other OSTs.  Are you also reading data from disk in 1MB chunks,
> or in smaller chunks?  You should read at the stripe size for best
> performance in this test.
>
>   
Hi Andreas,

Thank you. You are right. One problem was that the readahead made a 
process on an OST read chunks from other OSTs. That explains the network 
traffic. The other problem was the size of the I/O requests, which was 
too "small" (16 KB). The interesting point is that the (small) request 
size was the same in both configurations (1 MB and 128 MB stripe sizes) 
but, even then, the times were very different.

Regards,

    Juan.
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Software Engineer
> Cluster File Systems, Inc.
>   




More information about the lustre-discuss mailing list