[lustre-discuss] Lustre caching, chapter 2

[John Bauer]

All,

An update on the previous lustre-discuss post.  The closing of the first 
file after writing, dumping from cache, and reopening with O_RDONLY, 
does make it immune to the stat()s from another client node.

Now to the real question I have concerning the Lustre caching for the 
reproducer program.  I have changed the reproducer to :

1: write the first file, sim.dat.A.

2: force a memory dump of the OSCs ofA by doing O_DIRECT reads of the 
first stripe block on each OSC.  ( a trick I recently learned )

3: close the fd for A and reopen with only O_RDONLY.

4: read A.

5: write the larger second file, sim.dat.B

6: using the same fd, without closing B, read B forwards and backwards.

7: close both files.

This can all be see in this first image.

Why do the OSCs forA retain their memory even though the data is 
used/touched only once, while the OSCs for B are fighting for caching 
memory and their data is used/touched 3 times?

https://www.dropbox.com/scl/fi/flpvi9dnvftbtpjquleef/3461_fpa.png?rlkey=ux7zko8igh6loosrfs05ryb1c&st=mtpixhyy&dl=0

Another plot of interest relating to this run is the amount of memory 
for filePages on each node(socket) of the system.  This happens to be an 
8 node system. When A is read back in it is placed on nodes 1,2, and 4.  
While B is later fighting for caching memory, it also has filePages on 
node 4.  How can A hang on to the memory when its pages's LRU is 
certainly less than or equal to those for B.  The memory behavior can't 
be a node thing either as both files are on node 4 for much of the run, 
but A never gets pushed out of node 4.  I should also point out that the 
aggregate of the filePages on the 8 nodes maps directly to the cached 
memory from meminfo.  Not sure why it was decided to call the node thing 
filePages and the meminfo thing cached in /proc.

https://www.dropbox.com/scl/fi/gcpbct21e0knx5d420njo/3461_nodemem.png?rlkey=c47nl06a5hoqswh62ivx9uxdh&st=za8ec97b&dl=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260326/b0bc2328/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 3461_fpa.png
Type: image/png
Size: 69821 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260326/b0bc2328/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 3461_nodemem.png
Type: image/png
Size: 37903 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260326/b0bc2328/attachment-0003.png>