[lustre-discuss] Lustre striping and MPI

[John Bauer]

All

I am running a4 rank MPI job where all the ranks do an open of the file, 
attempt to set the striping with ioctl() and then do a small write.  
Intermittently, I get errors on the write() and ioctl(). This is a 
synthetic test case, boiled down from a much larger real world job.  
Note that I set the stripe_count to rank+1 so I can tell which of the 
ranks actually set the striping.

I have determined that I only get the write failure when the ioctl also 
failed with "No data available".  It also strikes me that at most, only 
one rank reports "File exists".  With a 4 rank job, I would think that 
normal behavior would be 1 rank would work as expected ( no error ) and 
the other 3 would report file exists.

Is this expected behavior?

rank=1 doIO() -1=ioctl(fd=9) No data available
rank=1 doIO() -1=write(fd=9) Bad file descriptor
rank=3 doIO() -1=ioctl(fd=9) File exists

oflags = O_CREAT|O_TRUNC|O_RDWR

void
doIO(const char *fileName, int rank){
int status ;
    int fd=open(fileName, O_RDWR|O_TRUNC|O_CREAT|O_LOV_DELAY_CREATE, 
0640 ) ;
    if( fd < 0 ) return ;

    struct lov_user_md opts = {0};
    opts.lmm_magic = LOV_USER_MAGIC;
    opts.lmm_stripe_size    = 1048576;
    opts.lmm_stripe_offset  = -1 ;
    opts.lmm_stripe_count   = rank+1 ;
    opts.lmm_pattern        = 0 ;

    status = ioctl ( fd , LL_IOC_LOV_SETSTRIPE, &opts);
    if(status<0)fprintf(stderr,"rank=%d %s() %d=ioctl(fd=%d) 
%s\n",rank,__func__,status,fd,strerror(errno));

    char *string = "this is it\n" ;
    int nc = strlen(string) ;
    status = write( fd, string, nc ) ;
    if( status != nc ) fprintf(stderr,"rank=%d %s() %d=write(fd=%d) 
%s\n",rank,__func__,status,fd,status<0?strerror(errno):"");
    status = close(fd) ;
    if(status<0)fprintf(stderr,"rank=%d %s() %d=close(fd=%d) 
%s\n",rank,__func__,status,fd,strerror(errno));
}

-- 
I/O Doctors, LLC
507-766-0378
bauerj at iodoctors.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20161026/3e11f702/attachment.htm>