<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";
color:black;}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
pre
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor=white lang=EN-US link=blue vlink=blue>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Marty,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>If my understand is right, when multiple
clients issue non-collective I/O and if their data buffer is a vector of small non-overlapping
file regions, instead of performing ‘n’ seeks + read/write ROMIO
uses data sieving algorithm. For data-sieving write, first the extent of
request is read into big buffer and respective write vectors memcpy’d
into big buffer and then single BIG write is performed. Prior to performing
data-sieving write, ROMIO locks the portion of the file pertaining to
data-sieving buff-size, does seek + write, and then unlocks the file-range.
This ensures the file integrity. ROMIO relies on ADIO-FS specific locking (in
this case Lustre). So if the underlying file-system does not support fcntl()
lock, then you see errors when the extent of the non-collective writes from
multiple clients overlap.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>The easy solution, would be to replace
non-collective MPI-IO calls with collective I/O MPI-IO calls. The two phase
collective I/O algorithm should ensure file integrity and does not rely on
file-locking since each process writes to a big non-overlapping region during
the second phase.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Or if you have to use non-collective I/O,
may be implement ad_lustre fcntl exclusive lock using<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>i) <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>fcntl(EXCL_LOCK) </span></font><font
size=2 color=navy face=Wingdings><span style='font-size:10.0pt;font-family:
Wingdings;color:navy'>à</span></font><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'> open(lock_file, O_CREATE
| O_EXCL) + close<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>fcntl(UNLOCK) </span></font><font size=2
color=navy face=Wingdings><span style='font-size:10.0pt;font-family:Wingdings;
color:navy'>à</span></font><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'> unlink(lock_file)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>ii) <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>fcntl(EXCL_LOCK) </span></font><font
size=2 color=navy face=Wingdings><span style='font-size:10.0pt;font-family:
Wingdings;color:navy'>à</span></font><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'> MPI_Win_Lock()<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>fcntl(EXCL_LOCK) </span></font><font
size=2 color=navy face=Wingdings><span style='font-size:10.0pt;font-family:
Wingdings;color:navy'>à</span></font><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'> MPI_Win_Unlock()<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Ofcourse you need to create a one-sided
shared buffer in rank 0 when the file is opened MPI_File_Open + buffer
destroyed during MPI_File_close()<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>HTH,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>-Kums<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
color=black face="Times New Roman"><span style='font-size:12.0pt;color:windowtext'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 color=black face=Tahoma><span
style='font-size:10.0pt;font-family:Tahoma;color:windowtext;font-weight:bold'>From:</span></font></b><font
size=2 color=black face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma;
color:windowtext'> lustre-discuss-bounces@lists.lustre.org [mailto:lustre-discuss-bounces@lists.lustre.org]
<b><span style='font-weight:bold'>On Behalf Of </span></b>Marty Barnaby<br>
<b><span style='font-weight:bold'>Sent:</span></b> Thursday, May 08, 2008 12:35
PM<br>
<b><span style='font-weight:bold'>Cc:</span></b> lustre-discuss@clusterfs.com<br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [Lustre-discuss]
mpi-io support</span></font><font color=black><span style='color:windowtext'><o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt'>To return to this discussion, in recent testing, I
have found that writing to a Lustre FS via a higher level library, like
PNetCDF, fails because the default for value for romio_ds_write is not disable.
This is set in the mpich code in the file /src/mpi/romio/adio/common/ad_hints.c<br>
<br>
I believe it has something to do with locking issues. I'm not sure how best to
handle this, I'd prefer the data sieving default be disable, though I don't
know all the implications there. Maybe an ad_lustre_open should be a place
where the _ds_ hints are set to disable.<br>
<br>
Marty Barnaby<br>
<br>
<br>
Weikuan Yu wrote: <o:p></o:p></span></font></p>
<pre wrap=""><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>Andreas Dilger wrote:<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'> <o:p></o:p></span></font></pre>
<blockquote style='margin-top:5.0pt;margin-bottom:5.0pt' type=cite><pre wrap=""><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>On Mar 11, 2008 16:10 -0600, Marty Barnaby wrote:<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'> <o:p></o:p></span></font></pre>
<blockquote style='margin-top:5.0pt;margin-bottom:5.0pt' type=cite><pre wrap=""><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>I'm not actually sure what ROMIO abstract device the multiple CFS<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>deployments I utilize were defined with. Probably just UFS, or maybe NFS.<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Did you have a recommended option yourself.<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'> <o:p></o:p></span></font></pre></blockquote>
<pre wrap=""><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>The UFS driver is the one used for Lustre if no other one exists.<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'> <o:p></o:p></span></font></pre>
<blockquote style='margin-top:5.0pt;margin-bottom:5.0pt' type=cite><pre wrap=""><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Besides the fact that most of the adio that were created over the years are<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>completely obsolete and could be cleaned from ROMIO, what will the new one<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>for Lustre offer? Particularly with respect to controls via the lfs utility<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>that I can already get?<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'> <o:p></o:p></span></font></pre></blockquote>
<pre wrap=""><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'>There is improved collective IO that aligns the IO on Lustre stripe<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>boundaries. Also the hints given to the MPIIO layer (before open,<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>not after) result in lustre picking a better stripe count/size.<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'> <o:p></o:p></span></font></pre></blockquote>
<pre wrap=""><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>In addition, the one integrated into MPICH2-1.0.7 contains direct I/O<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>support. Lockless I/O support was purged out due into my lack of<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>confidence in low-level file system support. But it can be revived when<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>possible.<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>--<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'>Weikuan Yu <+> 1-865-574-7990<o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><a
href="http://ft.ornl.gov/%7Ewyu/">http://ft.ornl.gov/~wyu/</a><o:p></o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt'> <o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt'><o:p> </o:p></span></font></p>
</div>
</body>
</html>