[lustre-discuss] lustre OSC and system cache

[John Bauer]

I'm observing some undesirable caching of OSC data in the system 
buffers.  This is a single node, single process application. There are 2 
files of interest, *SCRATCH *and *SCR300*, both are scratch files with 
stripeCount=4.  The system has 128GB of memory.  Lustre maxes out at 
about 59GB of memory used for caching.

*SCRATCH*,  About 22GB is written/read during the first 300 seconds of 
the run.  No further activity to the file ( but remains open ) until 
about 18,700 seconds into the run when another 22GB is written/read.  
Illustrated in the top frame of the first plot below.  In the bottom 
frame of the first plot is the amount of system cache used by each of 
the 4 OSC's associated with the file over the course of the run ( nearly 
identical, as would be expected ).  Note that each the OSC's retains its 
5.5GB of memory even though nothing is happening to the file.

*SCR300*,  A 110GB file, written and repeatedly read between the times 
of the above SCRATCH file's I/O.

What is of interest it that while SCR300 is doing all its I/O, and its 
associated OSC's are fighting each other for caching memory, the 4 OSC's 
for the inactive file(SCRATCH) retain their 22GB of memory.  Why are the 
4 OSC's for the inactive file exempt from giving up their memory?  It is 
very reproducible.

The application is MSC.Nastran, which has the capability to put the data 
for SCR300 inside of SCRATCH(increasing its size to 132GB).  If run in 
this mode, the caching behavior is much better behaved and the job runs 
in 11,500 seconds, versus 19,000. Illustrated in 3rd plot below.  While 
this is a solution for this case, it is not a general solution.

Thanks

John

Plots for *SCRATCH*



Plots for *SCR300*




Plots for *SCR300 *inside of *SCRATCH*


-- 
I/O Doctors, LLC
507-766-0378
bauerj at iodoctors.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20161212/9fe5bfd4/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bfoimgfaenjmgmii.png
Type: image/png
Size: 8780 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20161212/9fe5bfd4/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mncccijbfkiekmmn.png
Type: image/png
Size: 16468 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20161212/9fe5bfd4/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: adnondhpelpohhjf.png
Type: image/png
Size: 14165 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20161212/9fe5bfd4/attachment-0005.png>