[Lustre-discuss] Shared pointers, file locking and MPI-IO

[Andreas Dilger]

On Oct 23, 2008  16:06 +0200, S�bastien Buisson wrote:
> We are using MPI-IO on Lustre, so that we can achieve good performance 
> when writing or reading the same file from multiple processes and clients.
> In the shared pointers read/write routines, MPI-IO makes an intensive 
> use of file locks (for instance, see the file 
> src/mpi/romio/adio/common/ad_get_sh_fp.c from mpich2-1.0.7). The problem 
> is that file locking support in Lustre is very limited today:
> - there is the localflock option for the local mode. In this mode, locks 
> are coherent on a node (a single-node flock), but not across all clients;
> - there is the flock option for the consistent mode. In this mode, locks 
> are coherent across all clients. But the Lustre Operations Manual 
> includes a big warning about this mode, saying that it can affect 
> performance and stability.
> 
> That being said, does anybody in the Lustre community know about current 
> alternatives to the MPI-IO shared pointers read/write routines, which do 
> not use the file locking mechanism but provide the same functionality?

Tom Wang is our MPI-IO expert, and has done a lot of work to improve the
performance of MPI-IO with Lustre, so he can probably answer this question
better than I can.

> By the way, do we have an idea of the Lustre version that will provide 
> the reliable version of the flock consistent mode?

There are in fact a number of users that enable the flock support in
conjunction with MPI-IO already.  The reason we haven't made it enabled
for general usage is that there may still be some bad interactions with
flock under less well-behaved applications.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.