[Lustre-discuss] MPI-IO / ROMIO support for Lustre

[Rob Latham]

On Fri, Nov 05, 2010 at 05:11:47PM +0000, Mark Dixon wrote:
> Assuming I do not disable data sieving, which of the following
> options will most likely give me correct behaviour?
> 
> 1) Enable Lustre's cluster-wide coherent fcntl locks.
> 
I think you're going to have to go with this approach

> 2) Cheat, and enable Lustre's (cheaper) fcntl locks that are only
> coherent on an individual computer, on the assumption that Lustre's
> own internal locking mechanisms will "do the right thing".

Let's consider two process A and B who are doing non-contiguous
writes.  A writes to the even bytes and B writes to the odd bytes.
Data sieving means that both processes will read a big chunk of the
file, modify the local copy of the buffer and then write it.  OK, so
Lustre will ensure that either all of A's write or all of B's write
will show up, but that still means one process stomps all over the
other.

so you need something to serialize the read-modify-write.  

Note that if you use *collective* i/o then much of this no longer
becomes a problem.  Because it's collective the processes can use MPI
to coordinate and eliminate this false sharing problem.  

Note also that if you never ever use a shared file, then I guess you
can use local locks.  But if you are always doing one file per
processor instead of a few shared files,  you will end up with at the
very least a mess of files.  

==rob

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA