[lustre-discuss] Lustre caching behavior

[John Bauer]

Patrick,

Please find attached my reproducer, *sim.c*, and my run script *sim.sh* 
My apologies for sloppy coding ( no error handling, documentation ).  
Pretty simple though.

I'm not sure it will matter, but it was run on a system with 512GB RAM.  
I tuned the number of requests and transfer size to trigger something 
that is not relevant for this problem, the caching behavior for the 
second file being read forwards and backwards.  We discussed that one at 
LUG 24.  That's the program I am simulating here.

John

John

On 3/24/2026 6:56 PM, Oleg Drokin wrote:
> On Tue, 2026-03-24 at 23:42 +0000, Patrick Farrell wrote:
>> John,
>>
>> Are you able to pin this down into more of a reproducer?  Even just a
>> more granular description.
>>
>> I’m curious to explore it - this is poor behavior, not desirable for
>> sure.  I’m curious in particular to see about the lock cancellation -
>> my understanding had been the glimpse request to read lock path was
>> entirely opportunistic (NONBLOCKING in ldlm speak) - and would never
>> cause a cancel (ie, my understanding doesn’t accord with Andreas’s).
>>   I was pretty sure about that.
> We definitely consider dropping unused the lock after glimpse:
> /**
>   * Callback handler for receiving incoming glimpse ASTs.
>   *
>   * This only can happen on client side.  After handling the glimpse AST
>   * we also consider dropping the lock here if it is unused locally for
> a
>   * long time.
>   */
> static void ldlm_handle_gl_callback(struct ptlrpc_request *req,
>                                      struct ldlm_namespace *ns,
>                                      struct ldlm_request *dlm_req,
>                                      struct ldlm_lock *lock)
>
>
> ...
>          if (lock->l_granted_mode == LCK_PW &&
>              !lock->l_readers && !lock->l_writers &&
>              ktime_after(ktime_get(),
>                          ktime_add(lock->l_last_used, ns-
>> ns_dirty_age_limit))) {
> ...
>                  if (ldlm_bl_to_thread_lock(ns, ld, lock))
>                          ldlm_handle_bl_callback(ns, ld, lock);
>
> And ns_dirty_age_limit is 10 seconds.
>
>
> John, what version are you testing on btw?
> There were some lru changes in master very recently and I wonder if it
> made any difference for this kind of workload.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20260324/ebf92cc5/attachment.htm>
-------------- next part --------------
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <fcntl.h>

int do_io( int fd, int rwMode, int fbMode, int nRec, char *buffer, int bSize ) ;

int
main(int argc, char **argv) {
int bSize = 4096 ;
int nRec=1000 ;

   if( argc < 2 ) exit(-1) ;

   char *fileName = argv[1] ;
   if( argc > 2 ) nRec = atoi( argv[2] ) ;
   if( argc > 3 ) bSize = atoi( argv[3] )*1048576 ;
   char *buffer = malloc(bSize) ;
   if( buffer==NULL ) exit(-1) ;

   int fdA = open( fileName, O_RDWR|O_CREAT|O_TRUNC, 0640 ) ;
   if(fdA<0) exit(-1);

   char nameB[1024] ;
   snprintf(nameB,1023,"%s.B",fileName);
   int fdB = open( nameB, O_RDWR|O_CREAT|O_TRUNC, 0640 ) ;
   if(fdB<0) exit(-1);

   do_io( fdA, 'w', 'f', 2*nRec, buffer, bSize ) ;
   do_io( fdA, 'r', 'f', 2*nRec, buffer, bSize ) ;

   do_io( fdB, 'w', 'f', 3*nRec, buffer, bSize ) ;
   do_io( fdB, 'r', 'f', 3*nRec, buffer, bSize ) ;
   do_io( fdB, 'r', 'r', 3*nRec, buffer, bSize ) ;

   close( fdA ) ;
   close( fdB ) ;

}

int
do_io( int fd, int rwMode, int fbMode, int nRec, char *buffer, int bSize ) {
int i ;

   for(i=0;i<nRec;i++){
      off_t pos = (off_t)bSize * (fbMode=='f'? i : (nRec-i-1)) ;

      if( rwMode=='w' || rwMode=='b' ) {
         ssize_t ret = pwrite( fd, buffer, bSize, pos);
      } 
      
      if( rwMode=='r' || rwMode=='b'  ) {
         ssize_t ret = pread( fd, buffer, bSize, pos);
      }
   }
   return 0 ;
}
-------------- next part --------------
#!/bin/bash

FILE=/hpelustre/${USER}/sim.dat
rm -f ${FILE}

./sim ${FILE} 5140 16

lfs getstripe ${FILE}
ls -lh ${FILE}

rm -f ${FILE} ${FILE}.B