[lustre-discuss] Lustre caching behavior

[John Bauer]

Patrick,

I am running a simple simulator program that writes sim.dat and then 
reads sim.dat.  The program then writes sim.dat.B and then reads 
sim.dat.B forwards and backwards.

Each file is striped 3x16M.  Each file is striped on 3 different OST.


Without doing the stat()s from the front-end node the file activity 
versus time plot looks like this.


https://www.dropbox.com/scl/fi/31oyih092digsfqcmyrzs/3440.fpa.png?rlkey=77lybqpyp4t7fu11ndhqe78qu&st=a8nf16tt&dl=0



This next image is the amount of cached memory used by each of the 3 
OSC, on the compute node, versus time for the first file. It looks like 
only one line, but all 3 OSC have the same memory profile, so they are 
on top of each other.


https://www.dropbox.com/scl/fi/1n6q1fljndax0rene8dhb/3440.OSC_cached.png?rlkey=33rydhcbt0drl9to51rlmizi3&st=0qwx82bm&dl=0


Next image is the amount of cached memory used by each OSC, on the 
compute node, for the second file.


https://www.dropbox.com/scl/fi/xnmeoi0oni4x7zyy7ezfj/3440.B.OSC_cached.png?rlkey=557c55qdj60woelbyb3hf250b&st=x49bosvd&dl=0


The following 3 plots are the same as for above, but are from the second 
run of this job.  This time I am doing a stat() of the first file every 
10 seconds, from a login node while the job is running.


https://www.dropbox.com/scl/fi/v63gm040hn1ljwzx70u2h/3442.fpa.png?rlkey=bbr7ltxfemdqsq5jt04967t99&st=74e9aqts&dl=0


https://www.dropbox.com/scl/fi/m80b3o2og8fy527ayizff/3442.OSC_cached.png?rlkey=b9wcrx86dpl7qolalu77q8yey&st=748an788&dl=0


https://www.dropbox.com/scl/fi/lxr5ya7ihk7kx6ez182xl/3442.B.OSC_cached.png?rlkey=5utkp5gdywocafvt8ohh5fpc4&st=7yhkai9i&dl=0


On 3/24/2026 10:40 AM, Patrick Farrell wrote:
> Not as far as I know - can you be more specific about the behavior change?
> ------------------------------------------------------------------------
> *From:* lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on 
> behalf of John Bauer via lustre-discuss <lustre-discuss at lists.lustre.org>
> *Sent:* Tuesday, March 24, 2026 10:35 AM
> *To:* lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
> *Subject:* [lustre-discuss] Lustre caching behavior
>
> All,
>
> Is it a known effect that a stat() of a file on a given node will 
> change the caching behavior on a second node.  I have noticed that the 
> caching behavior of Lustre on a dedicated compute node will change 
> significantly if I do intermittent stats of the file of interest from 
> the front-end node.
>
> Thanks,
>
> John
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260324/49aae60f/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 3440.fpa.png
Type: image/png
Size: 11737 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260324/49aae60f/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 3440.OSC_cached.png
Type: image/png
Size: 8532 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260324/49aae60f/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 3440.B.OSC_cached.png
Type: image/png
Size: 13431 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260324/49aae60f/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 3442.fpa.png
Type: image/png
Size: 11653 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260324/49aae60f/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 3442.OSC_cached.png
Type: image/png
Size: 10060 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260324/49aae60f/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 3442.B.OSC_cached.png
Type: image/png
Size: 9535 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260324/49aae60f/attachment-0011.png>