[Lustre-discuss] Shared pointers, file locking and MPI-IO

[Tom.Wang]

Andreas Dilger wrote:
> On Oct 23, 2008  16:06 +0200, S�bastien Buisson wrote:
>   
>> We are using MPI-IO on Lustre, so that we can achieve good performance 
>> when writing or reading the same file from multiple processes and clients.
>> In the shared pointers read/write routines, MPI-IO makes an intensive 
>> use of file locks (for instance, see the file 
>> src/mpi/romio/adio/common/ad_get_sh_fp.c from mpich2-1.0.7). The problem 
>> is that file locking support in Lustre is very limited today:
>> - there is the localflock option for the local mode. In this mode, locks 
>> are coherent on a node (a single-node flock), but not across all clients;
>> - there is the flock option for the consistent mode. In this mode, locks 
>> are coherent across all clients. But the Lustre Operations Manual 
>> includes a big warning about this mode, saying that it can affect 
>> performance and stability.
>>
>> That being said, does anybody in the Lustre community know about current 
>> alternatives to the MPI-IO shared pointers read/write routines, which do 
>> not use the file locking mechanism but provide the same functionality?
>>     
>
> Tom Wang is our MPI-IO expert, and has done a lot of work to improve the
> performance of MPI-IO with Lustre, so he can probably answer this question
> better than I can.
>   
Hello,

As I know, there are no other ways to implement this shared pointers,
in MPI-IO. You must rely on this flock currently, if you want to use the
shared file pointer. Considering each client will maintain the file offset
itself for each file, so if MPI want to achieve this shared file-pointer 
over
multi-clients (also atomic), it seems flock is the only way here.

I also enable flock in my own cluster sometimes, it works fine in most 
case.
But I only ran IOR or flash IO. But these benchmarks does not
touch shared file pointers.

Could you please describe how you use this shared file pointer in your
application?

Thanks
WangDi
>   
>> By the way, do we have an idea of the Lustre version that will provide 
>> the reliable version of the flock consistent mode?
>>     
>
> There are in fact a number of users that enable the flock support in
> conjunction with MPI-IO already.  The reason we haven't made it enabled
> for general usage is that there may still be some bad interactions with
> flock under less well-behaved applications.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>