[Lustre-discuss] lustre ram usage (contd)

[Balagopal Pillai]

Hi Andreas,

          Thanks. The two OSS  also export two ext3 volumes each via NFS 
that are used
to backup the 4 smaller Lustre volumes. One possibility as you 
mentioned, is that the memory consumption is not Lustre related, but
ext3 related, as the destination ext3 volumes are also coming from the 
same OSS servers, but mounted over NFS on the Lustre client that does 
the rsync.

         I upgraded the ram today morning on both OSS from 4 G to 8 G 
and hope the ram on both OSS is enough for both Lustre operations and 
the rsync backup.

Regards
Balagopal

Andreas Dilger wrote:
> On Dec 23, 2007  18:01 -0400, Balagopal Pillai wrote:
>   
>>            The cluster is made idle on the weekend to look at the Lustre 
>> ram consumpton issue. The ram used during yesterday's rsync is still not 
>> freed up. Here is the output from free 
>>
>>              total       used       free     shared    buffers     cached
>> Mem:       4041880    3958744      83136          0     876132     144276
>> -/+ buffers/cache:    2938336    1103544
>> Swap:      4096564        240    4096324
>>     
>
> Note that this is normal behaviour for Linux.  Ram that is unused provides
> no value, so all available RAM is used for cache until something else is
> needing to use this memory.
>
>   
>>           Looking at vmstat -m, there is something odd. Seems like 
>> ext3_inode_cache and dentry_cache seems to be the biggest occupants of 
>> ram. ldiskfs_inode_cache comparatively smaller.   
>> -
>>
>> Cache                       Num  Total   Size  Pages
>> ldiskfs_inode_cache      430199 440044    920      4
>> ldlm_locks                10509  12005    512      7
>> ldlm_resources            10291  11325    256     15
>> buffer_head              230970 393300     88     45
>>     
>
>   
>> ext3_inode_cache         1636505 1636556    856      4
>> dentry_cache             1349923 1361216    240     16
>>     
>
> This is odd, because Lustre doesn't use ext3 at all.  It uses ldiskfs
> (which is ext3 renamed + patches), so it is some non-Lustre filesystem
> usage which is consuming most of your memory.
>
>   
>>              Is there anything in proc as explained in 
>> http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/ref-guide/s1-proc-directories.html
>> that can force the kernel to flush out the dentry_cache and 
>> ext3_inode_cache when the rsync is over and cache is not needed anymore? 
>> Thanks very much.
>>     
>
> Only to unmount and remount the filesystem, on the server.  On Lustre
> clients there is a mechanism to flush Lustre cache, but that doesn't
> help you here.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>