[Lustre-discuss] OST I/O problems

[Andreas Dilger]

On 2009-12-04, at 04:59, Heiko Schröter wrote:
> we do see those messages (see below) on our OSTs when under heavy  
> _read_ load (or when 60+ Jobs are trying to read data at approx the  
> same time).
> The OSTs freezes and even console output is down to a few bytes the  
> minute.
> After some time the OSTs do revocer.
>
> But together with the 3ware support we are pretty sure to have  
> replaced all snipish disks and data transfer looks ok when not used  
> by lustre (i.e. verifying up to 30-90MB/s/disk throughput).
>
> OSTs with 2GB RAM on 16port controller, 4GB RAM on 24port controller
> Raid6 (write once, read often archive system)
> lustre-1.6.6
> vanilla-kernel 2.6.22.19
> 3ware 9650se (16 and 24port) latest 9.5.3 Version
> Seagate 31000340NS disks, HITACHI 1TB disks

We've had problems in the past with 3ware controllers at other sites  
in the past - the performance is not as good as expected, since they  
rely heavily on readahead to get good performance.

That said:

> Dec  4 12:42:56 sadosrd24 LustreError: 4744:0:(ost_handler.c: 
> 882:ost_brw_read()) @@@ timeout on bulk PUT after 100+0s   
> req at ffff81007efa7e00 x7869690/t0 o3->eb2e7e64-c1d9- 
> d1f6-8f9d-1ba9629ff4c0 at NET_0x20000c0a8106f_UUID:0/0 lens 384/336 e 0  
> to 0 dl 1259926976 ref 1 fl Interpret:/0/0 rc 0/0

This means that the IO didn't complete before the timeout.  This could  
be because the OST IO is so slow that no RPC can complete before the  
timeout, or because there is packet loss.

Some things to try:
- reduce the number of OSS threads via module parameter:
   option ost oss_num_threads=N
- increase the lustre timeout (details in the manual)

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.