[Lustre-discuss] 2.0-alpha2 MDS out of memory problem
Isaac Huang
He.Huang at Sun.COM
Wed Jun 10 09:47:12 PDT 2009
On Wed, Jun 10, 2009 at 11:30:39AM +0200, Arne Wiebalck wrote:
>>> ......
>>> Any comments welcome, I am also happy to provide more details.
>>
>> Please show us:
>> /proc/meminfo /proc/slabinfo
>> /proc/sys/lnet/memused /proc/sys/lustre/memused* /proc/sys/lustre/pagesused*
>>
>> Preferably at around the OOM.
>>
>> It'd also be helpful to get a debug dump of memory allocations:
>> 1. echo malloc > /proc/sys/lnet/debug
>> 2. at around the OOM, lctl dk > malloc.dk
>>
>> How many clients were there? How were they connected to the MDS?
>>
>> Isaac
>
> Issac,
>
> Please find the requested info attached.
>From meminfo, it looked like 'slab' consumed most of the memory:
MemTotal: 2058932 kB
MemFree: 7640 kB
Slab: 1976324 kB
>From slabinfo, the biggest offenders are:
ldiskfs_inode_cache 994608 994608 944 4 1 : tunables 54 27 8 : slabdata 248652 248652 0
size-256 2988210 2988210 256 15 1 : tunables 120 60 8 : slabdata 199214 199214 0
About 900M and 800M respectively.
LNet seemed innocent, less than 1M:
/proc/sys/lnet/memused
453172
Lustre accounted around 600M at most:
/proc/sys/lustre/memused*
604295420
485604764
The ldiskfs_inode_cache slab looked fishy to me, but it's above my head. The
malloc dk dump at around the OOM is attached.
Thanks,
Isaac
-------------- next part --------------
A non-text attachment was scrubbed...
Name: malloc-1244620938.dk.gz
Type: application/octet-stream
Size: 278024 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090610/88a8ff2e/attachment.obj>
More information about the lustre-discuss
mailing list