<font size=2 face="sans-serif">Hi there,</font>
<br>
<br><font size=2 face="sans-serif">I'm experiencing some issues with Lustre
and WebSphere Portal Server 6.1 (WPS) co-existing on the Lustre client
application server. WPS likes to use a lot of memory. The server
was originally allocated 16 GB of RAM. These servers are XEN virtualized
on RHEL 5.3 running Lustre 1.6.7 patched 16895 (but not fully 1.6.7.1).</font>
<br>
<br><font size=2 face="sans-serif">What I'm seeing is that WPS eventually
takes all 15.5 GB of available memory (or tries to) and then my server
will hang and show an out of memory error on the console:</font>
<br>
<br><tt><font size=2>Call Trace:<br>
[<ffffffff802bc998>] out_of_memory+0x8b/0x203<br>
[<ffffffff8020f657>] __alloc_pages+0x245/0x2ce<br>
[<ffffffff8021336e>] __do_page_cache_readahead+0xd0/0x21c<br>
[<ffffffff802639f9>] _spin_lock_irqsave+0x9/0x14<br>
[<ffffffff8023efff>] lock_timer_base+0x1b/0x3c<br>
[<ffffffff88081d4d>] :dm_mod:dm_any_congested+0x38/0x3f<br>
[<ffffffff80213c47>] filemap_nopage+0x148/0x322<br>
[<ffffffff80208db9>] __handle_mm_fault+0x440/0x11f6<br>
[<ffffffff802666ef>] do_page_fault+0xf7b/0x12e0<br>
[<ffffffff80207141>] kmem_cache_free+0x80/0xd3<br>
[<ffffffff8025f82b>] error_exit+0x0/0x6e<br>
<br>
DMA per-cpu:<br>
cpu 0 hot: high 186, batch 31 used:73<br>
cpu 0 cold: high 62, batch 15 used:61<br>
cpu 1 hot: high 186, batch 31 used:164<br>
cpu 1 cold: high 62, batch 15 used:59<br>
DMA32 per-cpu: empty<br>
Normal per-cpu: empty<br>
HighMem per-cpu: empty<br>
Free pages: 6040kB (0kB HighMem)<br>
Active:2087085 inactive:1984455 dirty:0 writeback:0 unstable:0
free:1510 slab:9371 mapped-file:992 mapped-anon:4054270 pagetables:11073<br>
DMA free:6040kB min:16384kB low:20480kB high:24576kB active:8348340kB
inactive:7937820kB present:16785408kB pages_scanned:375028407 all_unreclaimable?
yes<br>
lowmem_reserve[]: 0 0 0 0<br>
DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no<br>
lowmem_reserve[]: 0 0 0 0<br>
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no<br>
lowmem_reserve[]: 0 0 0 0<br>
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB
inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no<br>
lowmem_reserve[]: 0 0 0 0<br>
DMA: 0*4kB 3*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB
1*1024kB 0*2048kB 1*4096kB = 6040kB<br>
DMA32: empty<br>
Normal: empty<br>
HighMem: empty<br>
17533 pagecache pages<br>
Swap cache: add 6497852, delete 6497594, find 1492942/1923882,
race 4+83<br>
Free swap = 0kB<br>
Total swap = 4194296kB<br>
uptimeagent invoked oom-killer: gfp_mask=0x201d2, order=0,
oomkilladj=0</font></tt>
<br>
<br>
<br><font size=2 face="sans-serif">I have 6 filesystems of varying size
(1.5 TB - 4 GB), and we use Lustre to share them amongst the WebSphere
cluster; our use of Lustre is commercial in nature, non-HPC and uses legacy
filesystem structures (still uses Linux HA though).</font>
<br>
<br><font size=2 face="sans-serif">If we stop WPS as it begins chewing
through RAM, we still see a lot of memory in use (Lustre client cache).
As I unmount each Lustre filesystem, I gain back a significant portion
of memory (about 7 GB back total). For grins, we ripple stopped each
WPS server, adjusted the maxmem XEN value and gave each server an additional
6 GB of RAM for a total of 22 GB. I'd like to now limit the Lustre
clients to the following, but I'm not sure if doing so will mess things
up:</font>
<br>
<br><font size=2 face="sans-serif">lctl set_param /proc/fs/lustre/llite/lusfs01*/max_cached_mb
2048 # Lustre Default is 12288</font>
<br><font size=2 face="sans-serif">lctl set_param /proc/fs/lustre/llite/lusfs02*/max_cached_mb
2048 # Lustre Default is 12288</font>
<br><font size=2 face="sans-serif">lctl set_param /proc/fs/lustre/llite/lusfs03*/max_cached_mb
1048 # Lustre Default is 12288</font>
<br><font size=2 face="sans-serif">lctl set_param /proc/fs/lustre/llite/lusfs04*/max_cached_mb
1048 # Lustre Default is 12288</font>
<br><font size=2 face="sans-serif">lctl set_param /proc/fs/lustre/llite/lusfs05*/max_cached_mb
1048 # Lustre Default is 12288</font>
<br><font size=2 face="sans-serif">lctl set_param /proc/fs/lustre/llite/lusfs06*/max_cached_mb
512 # Lustre Default is 12288</font>
<br>
<br><font size=2 face="sans-serif">So, here are my questions. Why
is 75% the default for max_cached_mb? What will happen if I "lctl
set_param /proc/fs/lustre/llite/lusfs01*/max_cached_mb 2048" instead
of 12288 where it is today for each of those filesystems mentioned above?
How am I affecting the performance of the client by making that change?
Is this a bad thing to do or no big deal? Some filesystems
are more heavily used than others, should I give them more memory? Some
filesystems have large files that I'm sure end up sitting up in memory,
should I give them more memory? I know the lmld lru_size can be used
to flush cache, but I don't think that's a wise thing to do, people might
lose...locks (true/false?) on files they're downloading or something, right?
Is there another cache tunable where I can flush cached things that
are two hours or more old, but leave the newer stuff (a max_cache_time
parameter perhaps)?</font>
<br>
<br><font size=2 face="sans-serif">Cheers,<br>
<br>
Ms. Andrea D. Rucks<br>
Sr. Unix Systems Administrator,<br>
Lawson ITS Unix Server Team<br>
_____________________________<br>
<br>
Lawson<br>
380 St. Peter Street<br>
St. Paul, MN 55102<br>
Tel: 651-767-6252<br>
</font><a href=http://www.lawson.com/><font size=2 face="sans-serif">http://www.lawson.com</font></a>