[lustre-discuss] Two fd's for a single file, 1 with O_DIRECT, 1 without O_DIRECT

[John L Bauer]

Andreas,

Thanks for the response.  I ran the job on my laptop on an XFS file 
system. I changed the file size from 100MB to 1000MB to get the 
buffering numbers out of the noise.  The write of the file behaves as 
one would expect.  The first 500MB being written non-direct increases 
the buffer cache usage by 500MB.  The writing of the second 500MB of the 
file using direct does not increase the buffer cache usage, and the 
first half that was written buffered does not get dumped.

The second image below is for the Lustre run, but replacing the total 
amount of meminfo buffer cache with the amount of buffer cache used by 
each OSC, where you can see each OSC's cache usage dropping as each OSC 
handles its first direct write.

John


On 1/9/25 14:34, Andreas Dilger wrote:
> Hi John,
> can you trace to determine if this cache flush is triggered by Lustre DLM lock management or if this is triggered by the kernel (i.e. does ext4 or xfs have the same behavior with two fds)?
>
> There was work done recently to be able to pin pages on the client so that they are not evicted by e.g. LDLM LRU aging:
> https://jira.whamcloud.com/browse/LU-17463
>
> but this hasn't finished landing yet.
>
> On Dec 19, 2024, at 15:53, John Bauer<bauerj at iodoctors.com>  wrote:
>> I have a file where I would like to keep part of the file in system cache and the other part not.  So I open the file twice, once with O_DIRECT, and once without O_DIRECT.
>> I have written a simple testcase where I write the first 50 MB of the file without DIRECT, and the second 50M with O_DIRECT.  The file is striped 4x1M.  It would appear that when the second 50M of the file is written O_DIRECT, the first 50MB of the file is dropped from system cache.   Both fd remain open during the entire process.  From what I can tell, it seems that once a given stripe is impacted by an O_DIRECT  write then the entire stripe is dropped from cache. Is this the expected behavior?
> Cheers, Andreas
> —
> Andreas Dilger
> Lustre Principal Architect
> Whamcloud/DDN
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250110/72352544/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FgWSoGPEfAD6Fv1W.png
Type: image/png
Size: 10418 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250110/72352544/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: AbVQ2ZbMCjn5Mmwe.png
Type: image/png
Size: 14339 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250110/72352544/attachment-0003.png>