<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi RobR,<br>
<br>
Rob Ross wrote:
<blockquote cite="mid:0011AEB7-9A96-4B01-A37A-1AE1896CC512@mcs.anl.gov"
type="cite">Hi LiuYing,
<br>
<br>
Unfortunately, the group here is committed to our interpretation of the
standard as being that the user passing a hint parameter that is
misleading to the implementation cannot cause *incorrect* behavior
(i.e. change the semantics of the call).
<br>
<br>
An option for determining contiguity is to pass messages during
file_set_view time; if the file view is contiguous, then the access is
contiguous. Since file_set_view is a collective call, you have an
opportunity to do this message passing.
<br>
<br>
I'm not sure how you're avoiding any communication, because the
application processes can still be performing I/O at arbitrary offsets.
Perhaps knowing that the access in file is contiguous, however, can be
used to reduce the overall communication at I/O time anyway? Can you
further explain how these hints worked? Maybe we can come up with an
alternative together.
<br>
</blockquote>
We don't mean to avoid any communication. We just do some optimization
only when the communication overhead has a big impact on the system
performance. <br>
When we tested the lustre adio driver, we found MPI_Alltoall cost much
time. That is why we try to avoid it.<br>
<br>
<meta http-equiv="CONTENT-TYPE" content="text/html; charset=utf-8">
<title></title>
<meta name="GENERATOR" content="OpenOffice.org 3.0 (Linux)">
There are two MPI_Alltoall calls in the original codes. One is called
by ADIOI_W_Exchange_data() and the other by ADIOI_Calc_others_req().<br>
<meta http-equiv="CONTENT-TYPE" content="text/html; charset=utf-8">
<title></title>
<meta name="GENERATOR" content="OpenOffice.org 3.0 (Linux)">
<style type="text/css">age { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<ul>
<li>ADIOI_W_Exchange_data():
In this function, <big><font color="#000000"><font size="2"><big><span
style="background: transparent none repeat scroll 0%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;">the
original ADIO driver uses MPI_Alltoall() to
exchange recv/send size among the processes. However, since Lustre ADIO
driver reorganizes the requests into stripe-contiguous I/O pattern,
recv/send size for each process can be calculated in the beginning of
ADIOI_LUSTRE_Exch_and_write(). So we don't need
MPI_Alltoall() to exchange offset and length any more.</span></big></font></font></big></li>
</ul>
<ul>
<li>
<meta http-equiv="CONTENT-TYPE" content="text/html; charset=utf-8">
<title></title>
<meta name="GENERATOR" content="OpenOffice.org 3.0 (Linux)">
<style type="text/css">@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<meta http-equiv="CONTENT-TYPE" content="text/html; charset=utf-8">
<title></title>
<meta name="GENERATOR" content="OpenOffice.org 3.0 (Linux)">
<meta http-equiv="CONTENT-TYPE" content="text/html; charset=utf-8">
<title></title>
<meta name="GENERATOR" content="OpenOffice.org 3.0 (Linux)">
ADIOI_Calc_others_req(): This is what we are discussing. In this
function,
<style type="text/css">age { margin: 0.79in }
P { margin-bottom: 0.08in }
--></style><font
color="#000000"><font size="3"><span
style="background: transparent none repeat scroll 0%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;">the
original ADIO driver uses </span></font></font><font color="#000000"><font
size="3"><span
style="background: transparent none repeat scroll 0% 50%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;">MPI_Alltoall()</span></font></font><font
color="#000000"><font size="3"><span
style="background: transparent none repeat scroll 0%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;">
</span></font></font><font color="#000000"><font size="3"><span
style="background: transparent none repeat scroll 0% 50%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;">to
exchange the access information among the processes</span></font></font><font
color="#000000"><font size="3"><span
style="background: transparent none repeat scroll 0%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;">.
It's necessary, but there is still a little space to improve. </span></font></font>
<style type="text/css"><!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
--></style>
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
--></style><big><font
color="#000000"><font size="2"><big><span
style="background: transparent none repeat scroll 0%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;"></span></big></font></font></big></li>
<ul>
<li><big><font color="#000000"><font size="2"><big><span
style="background: transparent none repeat scroll 0%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;">if
the request
data are contiguous, the access information (offset and length) can
be calculated by other simpler communication type(i.e. MPI_Allreduce). <br>
</span></big></font></font></big></li>
<li><big><font color="#000000"><font size="2"><big><span
style="background: transparent none repeat scroll 0%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;">Further,
if the
request size is same, the access information can be calculated
directly without any communication. </span></big></font></font></big></li>
</ul>
</ul>
As you said, if the hints settings are inconsistent with the data
given by the user, it will cause incorrect behavior and break the
semantics of the call. So I agree this problem must be fixed.<br>
<br>
I don't remember exactly how much the hints help us, but I think only
when the request data are contiguous and have same size, we can get the
real benefit.<br>
So, in other words, if checking contiguity+size will introduce new
overhead, I prefer to remove the hints and use MPI_Alltoall cleanly. <br>
<br>
Any idea ?<br>
<br>
Thanks,<br>
-LiuYing<br>
<big><big> </big></big>
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-</style>
<blockquote cite="mid:0011AEB7-9A96-4B01-A37A-1AE1896CC512@mcs.anl.gov"
type="cite"><br>
Thanks,
<br>
<br>
Rob
<br>
<br>
On Mar 19, 2009, at 1:27 AM, emoly.liu wrote:
<br>
<br>
<blockquote type="cite">Hi rob,
<br>
<br>
Robert Latham wrote:
<br>
<blockquote type="cite"><br>
On Mon, Mar 16, 2009 at 03:41:47PM +0800, emoly.liu wrote:
<br>
<blockquote type="cite">
<blockquote type="cite">romio_lustre_contig_data
<br>
romio_lustre_samesize
<br>
</blockquote>
They are two hints to tell the driver whether the request data are
<br>
contiguous and whether each request IO has the same size. If they
<br>
are both "yes", we can optimize ADIOI_LUSTRE_Calc_others_req() by
<br>
removing MPI_Alltoall(). Because each process can easily calculate
<br>
the pairs of offset and length for each request without collective
<br>
communication. BTW, currently only when they are both positive, the
<br>
optimization can work. In the future, probably some efforts will be
<br>
made to other conditions.
<br>
<br>
</blockquote>
OK, here's the one with the major problem. RobR reminds me that
<br>
MPI-IO requires hints to be optional and cannot cause incorrect
<br>
behavior. A user supplying these hints and then giving you data that
<br>
is noncontiguous or not of the same size would cause incorrect
<br>
behavior, so these aren't appropriate.
<br>
<br>
Is there a way you can check what the caller is doing? caller can lie
<br>
to you via hints, but ROMIO still has to give the right answer. RobR
<br>
thought maybe MPI_Allreduce or something along those lines before the
<br>
MPI_Alltoall would let you check.
<br>
<br>
</blockquote>
Hmm, it is indeed a problem, although we did get benefits from them in
our previous tests.
<br>
<br>
I will check it. But currently, is it possible to make mention of the
risk with some words, just like "Don't set these two hints, until you
know exactly what you are doing" ?
<br>
If it is still inappropriate, I will remove them in this version, then
submit another patch once I figure out how to check it with low
overhead.
<br>
</blockquote>
<br>
<br>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Best regards,
LiuYing
System Software Engineer, Lustre Group
Sun Microsystems ( China ) Co. Limited
</pre>
</body>
</html>