[Lustre-discuss] page allocation failure

[Wang lu]

Dear Andreas, 
   After I update OSS servers to 64bit, the memory usage become normal( there
are 3GMemory used now). It is a big progress. However, there are still a lot of
logs like slow *** when I am doing heavy load test. 


testjob.sh
#!/bin/bash
c=0
while true
do
        let c=c+1
        filename="/besfs/wanglu/$HOSTNAME-$$-$RANDOM"
        dd if=/dev/zero of="$filename" bs=1M count=1000
        sleep 1
        dd if="$filename" of="/dev/null" bs=1M count=1000
        sleep 1
        rm "$filename"
        date
        echo "$c GB W+R  finished!"
        echo
done

 


Nov 28 16:12:50 boss01 kernel: LustreError: 2129:0:(filter_io_26.c:779:
filter_commitrw_write()) besfs-OST0005: slow direct_io 53s
Nov 28 16:12:50 boss01 kernel: LustreError: 871:0:(lustre_fsfilt.h:318:
fsfilt_commit_wait()) besfs-OST0005: slow journal start 53s
Nov 28 16:12:50 boss01 kernel: LustreError: 871:0:(filter_io_26.c:792:
filter_commitrw_write()) besfs-OST0005: slow commitrw commit 53s
Nov 28 16:13:24 boss01 kernel: LustreError: 838:0:(filter_io_26.c:714:
filter_commitrw_write()) besfs-OST0002: slow i_mutex 123s
Nov 28 16:13:24 boss01 kernel: LustreError: 838:0:(filter_io_26.c:714:
filter_commitrw_write()) Skipped 84 previous similar messages
Nov 28 16:13:24 boss01 kernel: LustreError: 2107:0:(lustre_fsfilt.h:262:
fsfilt_brw_start_log()) besfs-OST0002: slow journal start 123s
Nov 28 16:13:24 boss01 kernel: LustreError: 2107:0:(lustre_fsfilt.h:262:
fsfilt_brw_start_log()) Skipped 10 previous similar messages
Nov 28 16:13:24 boss01 kernel: LustreError: 2107:0:(filter_io_26.c:727:
filter_commitrw_write()) besfs-OST0002: slow brw_start 123s
Nov 28 16:13:24 boss01 kernel: LustreError: 2107:0:(filter_io_26.c:727:
filter_commitrw_write()) Skipped 11 previous similar messages
Nov 28 16:13:24 boss01 kernel: LustreError: 2093:0:(lustre_fsfilt.h:227:
fsfilt_start_log()) besfs-OST0002: slow journal start 122s
Nov 28 16:13:24 boss01 kernel: LustreError: 2175:0:(lustre_fsfilt.h:318:
fsfilt_commit_wait()) besfs-OST0002: 




Andreas Dilger 写:

> On Nov 28, 2008  10:16 +0800, Wang lu wrote:
>>   Thank you very much for your suggestion.   I have one more question. We 
>> have more then 100 client nodes running 32bit
>> Linux, when we switch the OSS kernel to 64bit, is there any special
>> configuration we should do?
> 
> No, there is no requirement for the kernels on the OSS and clients to match.
> 
>> Andreas Dilger 写:
>>
>>> On Nov 26, 2008  19:04 +0800, Wang lu wrote:
>>>> The %util of memory on OSS was always around 10% ,even when OSS was going
to
>>>> die.  The OSS kernel is:  2.6.9-67.0.7.EL_lustre.1.6.5smp(32bit)
>>>>
>>>> Lustre version is 1.6.5.1
>>>>
>>>> We have 8GB physical memory and 16GB(never been used) swap total. 
>>>>
>>>> Is there a problem with memory management?
>>>
>>> The problem is with the 32-bit kernel.  Linux doesn't allow a 32-bit
>>> kernel to use more than 900MB of memory on a 32-bit system, no matter
>>> how much RAM is installed.  900MB/8192MB ~= 10% of RAM.  Swap is not
>>> useful for the kernel.
>>>
>>>> Nov 20 19:40:45 boss02 kernel: Normal: 640*4kB 109*8kB 127*16kB 60*32kB 0*
>> 64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7384kB
>>>> Nov 20 19:40:45 boss02 kernel: HighMem: 376*4kB 1162*8kB 815*16kB 299*32kB
>> 160*64kB 61*128kB 34*256kB 25*512kB 8*1024kB 1*2048kB 1786*4096kB =
7398656kB
>>>
>>> As you can see, all of the memory is available in "highmem" and not in
>>> the "normal" memory region that the kernel uses.
>>>
>>>> Nov 21 05:48:44 boss02 kernel: ll_ost_io_114: page allocation 
>>>> failure. 
>> order:4, mode:0x50
>>>
>>> These are "order 4" allocations (64kB), which the kernel is bad at handling
>>> under memory pressure in any case.  You can see in the "Normal" zone above
>>> that all memory chunks 64kB and larger have no free memory to allocate.
>>>
>>>> Nov 20 19:40:46 boss02 kernel:  [<c02b162a>] tcp_v4_do_rcv+0x1b/0xe9
>>>> Nov 20 19:40:46 boss02 kernel:  [<fb18fd06>] ost_handle+0xe56/0x5790 
>>>
>>> This appears that the memory allocation problems are due to the TCP
>>> stack.  I would suspect that you are using TCP with jumbo packets.
>>>
>>> The easiest solution is to run a 64-bit kernel, which I suspect should
>>> be possible given that hardly any 32-bit machines allow more than 4GB
>>> of RAM.  Next it would be possible to use regular ethernet frames, which
>>> may help somewhat but it won't let you use the other 7GB of RAM in the
>>> system.
>>>
>>> Cheers, Andreas
>>> --
>>> Andreas Dilger
>>> Sr. Staff Engineer, Lustre Group
>>> Sun Microsystems of Canada, Inc.
>>>
>>
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>